[hpsa] regular freezes with smartctl -d cciss, 0

Hi All,

I'm reporting a problem with the hpsa driver that's triggered by smartctl, thus
the two lists.  Please copy me in any responses.

I have a nagios check that runs smartctl -d cciss,$i looping around all drives,
and has been running successfully on a bunch of servers.  The newest server
(most recent install, if not hardware) uses the hpsa driver and SATA drives,
(/dev/sd?).  This server has a 2-drive RAID1, plus spare, and a 5-drive RAID5,
plus spare, so we're looping around cciss,0 through cciss,9, and cciss,11 says
"No such device" and the loop is terminated.

NOTE: I noticed that -dcciss,10 seems to work (but there aren't 11 disks); but,
when using -dsat+cciss,10, it has: this Read Device Identity failed: empty
IDENTIFY data.  In both cases, 11 has "No such device".

Here's the full output of one invocation:
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.16.0-28-generic] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/sda [cciss_disk_00] [SAT]: Device open changed type from 'sat,auto+cciss' to 'sat'
=== START OF READ SMART DATA SECTION ===
SMART Status not supported: Incomplete response, ATA output registers missing
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

Since enabling that check, the server has been freezing regularly.  Yesterday I
brought up a serial console, to see these messages:

[  361.453116] INFO: task smartctl:3746 blocked for more than 120 seconds.
[  361.460620]       Tainted: G        W I   3.16.0-28-generic #38-Ubuntu
[  361.468013] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Also happened on this kernel:

[ 8650.079319]       Tainted: G        W I   3.13.0-43-generic #72-Ubuntu
[ 8650.086635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 [ 8659.251188] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 12, t=15015 jiffies, g=275781, c=275780, q=0)                                  

with:
root     10756 10720  0 14:49 ?        00:00:00 smartctl -dcciss,4 -H /dev/sda

and many processes starting to hang a few minutes later:

root     10775 10766 99 14:52 ?        00:02:20 cciss_vol_status -V -s /dev/cciss/c*d0 /dev/sg0 /dev/sg1 /dev/sg2 /dev/sg3
root     10837 10836 99 14:52 ?        00:02:04 lvs
root     10893 10885  0 14:53 ?        00:00:00 dumpe2fs -h /dev/sda1

Note the "taint", which appears to be due to this:
[    0.256948] This system BIOS has enabled interrupt remapping
[    0.256948] on a chipset that contains an erratum making that
[    0.256948] feature unstable.  To maintain system stability
[    0.256948] interrupt remapping is being disabled.  Please
[    0.256948] contact your BIOS vendor for an update

This is running Xeon E5620 I've installed the microcode package, but no updates
were loaded.

>From hpsa:

[    8.321717] HP HPSA Driver (v 3.4.4-1)
[    8.326187] hpsa 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
[    8.334605] hpsa 0000:04:00.0: MSIX
[    8.338733] hpsa 0000:04:00.0: irq 66 for MSI/MSI-X
[    8.338741] hpsa 0000:04:00.0: irq 67 for MSI/MSI-X
[    8.338748] hpsa 0000:04:00.0: irq 68 for MSI/MSI-X
[    8.338755] hpsa 0000:04:00.0: irq 69 for MSI/MSI-X
[    8.338763] hpsa 0000:04:00.0: irq 70 for MSI/MSI-X
[    8.338768] hpsa 0000:04:00.0: irq 71 for MSI/MSI-X
[    8.338773] hpsa 0000:04:00.0: irq 72 for MSI/MSI-X
[    8.338780] hpsa 0000:04:00.0: irq 73 for MSI/MSI-X
[    8.338785] hpsa 0000:04:00.0: irq 74 for MSI/MSI-X
[    8.338790] hpsa 0000:04:00.0: irq 75 for MSI/MSI-X
[    8.338795] hpsa 0000:04:00.0: irq 76 for MSI/MSI-X
[    8.338800] hpsa 0000:04:00.0: irq 77 for MSI/MSI-X
[    8.338805] hpsa 0000:04:00.0: irq 78 for MSI/MSI-X
[    8.338812] hpsa 0000:04:00.0: irq 79 for MSI/MSI-X
[    8.338816] hpsa 0000:04:00.0: irq 80 for MSI/MSI-X
[    8.338821] hpsa 0000:04:00.0: irq 81 for MSI/MSI-X
[    8.338878] hpsa 0000:04:00.0: Logical aborts not supported
[    8.390154] hpsa 0000:04:00.0: hpsa0: <0x323a> at IRQ 66 using DAC
[    8.410709] scsi6 : hpsa
[    8.416074] hpsa 0000:04:00.0: RAID              device c6b3t0l0 added.
[    8.416078] hpsa 0000:04:00.0: Direct-Access     device c6b0t0l0 added.
[    8.416080] hpsa 0000:04:00.0: Direct-Access     device c6b0t0l1 added.

Some other notable kernel boot messages:

[    1.131923] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    1.314604] pci 0000:00:1f.0: can't claim BAR 13 [io  0x0800-0x087f]: address conflict with ACPI CPU throttle [io  0x0810-0x0815]
[    1.396232] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    3.153246] ERST: Failed to get Error Log Address Range.
[    3.117767] [Firmware Warn]: GHES: Poll interval is 0 for generic hardware er
ror source: 1, disabled.
[    3.128268] GHES: APEI firmware first mode is enabled by WHEA _OSC.
[    5.991621] i8042: Failed to disable AUX port, but continuing anyway... Is th
is a SiS?
[    6.000596] i8042: If AUX port is really absent please use the 'i8042.noaux' option

I just ran it in a loop, using the latest smartmon version (6.3), and triggered
badness pretty quickly (<5min), with the addition of IO errors .. however I
haven't been able to reproduce those, so may have been "real" or some spurious
side effect.

[  304.990869] end_request: I/O error, dev sda, sector 256149504
[  305.195133] end_request: I/O error, dev sda, sector 256149504
[  309.128312] end_request: I/O error, dev sda, sector 256149504
[  309.272488] end_request: I/O error, dev sda, sector 138677936
[  309.279068] Aborting journal on device sda1-8.
[  309.825224] end_request: I/O error, dev sda, sector 138676224
[  309.831717] Buffer I/O error on device sda1, logical block 17334272
[  309.838872] JBD2: Error -5 detected when updating journal superblock for sda1-8.
[  310.590236] end_request: I/O error, dev sda, sector 2048
[  310.596238] Buffer I/O error on device sda1, logical block 0
[  310.602719] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal
[  310.612538] EXT4-fs (sda1): Remounting filesystem read-only
[  310.618830] EXT4-fs (sda1): previous I/O error to superblock detected
[  311.419315] end_request: I/O error, dev sda, sector 2048
[  311.425318] Buffer I/O error on device sda1, logical block 0
[  340.093063] end_request: I/O error, dev sda, sector 230757056
[  340.099624] EXT4-fs error (device sda1): ext4_find_entry:1312: inode #7212241: comm kerneloops: reading directory lblock 0
[  340.112070] EXT4-fs (sda1): previous I/O error to superblock detected
[  341.210480] end_request: I/O error, dev sda, sector 2048
[  341.216510] Buffer I/O error on device sda1, logical block 0
[  361.453116] INFO: task smartctl:3746 blocked for more than 120 seconds.
[  361.460620]       Tainted: G        W I   3.16.0-28-generic #38-Ubuntu
[  361.468013] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  404.706013] end_request: I/O error, dev sda, sector 0
[  405.278753] end_request: I/O error, dev sda, sector 312512256
[  405.450981] end_request: I/O error, dev sda, sector 312512424
[  406.412259] end_request: I/O error, dev sda, sector 0
[  407.437599] end_request: I/O error, dev sda, sector 8
[  408.350794] end_request: I/O error, dev sda, sector 0
[  409.284030] end_request: I/O error, dev sdb, sector 2147484032
[  409.404178] end_request: I/O error, dev sdb, sector 2147484144
[  410.129138] end_request: I/O error, dev sdb, sector 512
[  411.130442] end_request: I/O error, dev sdb, sector 520
[  411.923503] end_request: I/O error, dev sdb, sector 512
[  413.073001] end_request: I/O error, dev sda, sector 278976384
[  413.305307] end_request: I/O error, dev sda, sector 278976496
[  413.421464] end_request: I/O error, dev sda, sector 2048
[  414.134401] end_request: I/O error, dev sda, sector 2056
[  416.024908] end_request: I/O error, dev sda, sector 2048
[  417.090238] end_request: I/O error, dev sdb, sector 2181038464
[  417.342625] end_request: I/O error, dev sdb, sector 2181038576
[  417.643037] end_request: I/O error, dev sdb, sector 2147484160
[  418.363966] end_request: I/O error, dev sdb, sector 2147484168
[  419.357273] end_request: I/O error, dev sdb, sector 2147484160
[  421.175150] end_request: I/O error, dev sda, sector 312510336
[  421.371310] end_request: I/O error, dev sda, sector 312510448
[  422.147928] end_request: I/O error, dev sda, sector 278978560
[  423.148761] end_request: I/O error, dev sda, sector 278978568
[  424.581910] end_request: I/O error, dev sda, sector 278978560
[  425.282476] end_request: I/O error, dev sdb, sector 0
[  426.511531] end_request: I/O error, dev sdb, sector 7813837056
[  427.552337] end_request: I/O error, dev sdb, sector 7813837216
[  428.417064] end_request: I/O error, dev sdb, sector 0
[  429.293762] end_request: I/O error, dev sdb, sector 8
[  430.154463] end_request: I/O error, dev sdb, sector 0

Thanks for any help you can provide.

Justin

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk