Re: FW: [hpsa] regular freezes with smartctl -d cciss, 0
Justin Pryzby <[email protected]>
| Newsgroups | gmane.linux.utilities.smartmontools |
|---|---|
| Message-ID | <[email protected]> |
[Re-adding smartmontools] Just noticed: it looks like the freeze happened when my nagios check queried SMART on the same drive as my shell loop. On Thu, Feb 12, 2015 at 11:45:10AM -0600, Justin Pryzby wrote: > Hi All, > > Thanks for your response, and sorry for the delay; the server was inaccessible > for the last 45 days (wouldn't boot, perhaps due to storage issues; and, IPMI > ended up with ARP disabled, somehow..). > > You're right: there are 9 drives, which I take to mean a 2 drive RAID1 (160GB), > plus spare, and 6 drive RAID5 (3.7TB), plus spare. > > All the kernel messages I gave before are also visible on 3.13.0-45-generic > (ubuntu). > > I found the following using commands like this: > sudo smartctl -i -d cciss,10 /dev/sg0 > > cciss,10: SRC 8x6G > cciss,9: DL18xG6BP > cciss,8: Hitachi Ultrastar A7K2000 (HUA722010CLA330) > cciss,7: GB1000EAMYC > cciss,6: GB1000EAMYC > cciss,5: Hitachi Ultrastar A7K2000 (HUA722010CLA330) > cciss,4: Hitachi Ultrastar A7K2000 (HUA722010CLA330) > cciss,3: Hitachi Ultrastar A7K2000 (HUA722010CLA330) > cciss,2: GB0160EAFJE > cciss,1: GB0160EAPRR > cciss,0: GB0160EAFJE > > I don't know why I didn't include this at first: > pryzbyj@TelsasoftBackupSvr:~$ sudo cciss_vol_status /dev/sg0 -V > Controller: Smart Array P410 > Board ID: 0x3243103c > Logical drives: 2 > Running firmware: 6.60 > ROM firmware: 6.60 > /dev/sda: (Smart Array P410) RAID 1 Volume 0 status: OK. At least one spare drive designated. At least one spare drive remains available. > /dev/sdb: (Smart Array P410) RAID 5 Volume 1 status: OK. At least one spare drive designated. At least one spare drive remains available. > Physical drives: 9 > connector 1I box 1 bay 1 ATA GB0160EAFJE 9RXFFDWX HPG0 OK > connector 1I box 1 bay 2 ATA GB0160EAPRR WCAT25858718 HPG2 OK > connector 1I box 1 bay 3 ATA GB0160EAFJE 9RXFFE4K HPG0 OK > connector 1I box 1 bay 4 ATA Hitachi HUA72201 JPW9K0N01RUE4L JP4OA3MA OK > connector 1I box 1 bay 5 ATA Hitachi HUA72201 JPW9K0N01L5K2L JP4OA3MA OK > connector 1I box 1 bay 6 ATA Hitachi HUA72201 JPW9K0N01NT0GL JP4OA3MA OK > connector 1I box 1 bay 7 ATA GB1000EAMYC WMATV0894358 HPG3 OK > connector 1I box 1 bay 8 ATA GB1000EAMYC WMATV1143562 HPG3 OK > connector 1I box 1 bay 9 ATA Hitachi HUA72201 JPW9J0N000WTUV JP4OA3MA OK > /dev/sg0: (Smart Array P410) Enclosure DL18xG6BP (S/N: ) on Bus 0, Physical Port 1I status: OK. > /dev/sg0(Smart Array P410:0): Non-Volatile Cache status: > Cache configured: Yes > Read cache memory: 100 MiB > Write cache memory: 300 MiB > Write cache enabled: Yes > Flash backed cache present > > Confirming there are 9 drives. > > Note that some of the smart inquiries were very slow: > pryzbyj@TelsasoftBackupSvr:~$ time sudo smartctl -i -d cciss,8 /dev/sg0 > smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-45-generic] (local build) > Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org > > /dev/sg0 [cciss_disk_08] [SAT]: Device open changed type from 'sat,auto+cciss' to 'sat' > === START OF INFORMATION SECTION === > Model Family: Hitachi Ultrastar A7K2000 > Device Model: Hitachi HUA722010CLA330 > Serial Number: JPW9J0N000WTUV > LU WWN Device Id: 5 000cca 39ac0685c > Firmware Version: JP4OA3MA > User Capacity: 1,000,204,886,016 bytes [1.00 TB] > Sector Size: 512 bytes logical/physical > Rotation Rate: 7200 rpm > Device is: In smartctl database [for details use: -P show] > ATA Version is: ATA8-ACS T13/1699-D revision 4 > SATA Version is: SATA 2.6, 3.0 Gb/s > Local Time is: Thu Feb 12 11:34:03 2015 EST > SMART support is: Available - device has SMART capability. > SMART support is: Enabled > > real 0m2.940s > user 0m0.031s > sys 0m0.002s > > I don't know why I'd set that one particular server to query smart on /dev/sda, > instead of /dev/sg0; most of the other servers are using /dev/cciss/*d0, and > one server is using /dev/sg0 (It doesn't have /dev/cciss, I believe since it's > using: HP HPSA Driver (v 2.0.2-3)). > > I was still able to break the machine in minutes by looping around: > smartctl -H -d cciss,0 /dev/sg0 > > So changing the nagios test to check smartctl -i first is unlikely to help. > > Justin > > On Fri, Dec 19, 2014 at 09:07:47PM +0000, Scott Benesh wrote: > > Hi Justin, > > > > My group does the hpsa driver and we have some thoughts about what you are seeing. > > > > From what I can tell you have you 9 drives in the system; 2-drive RAID1, plus spare (3), and a 5-drive RAID5, plus spare (6), so shouldn't you be looking at drives cciss,0 - cciss,8? > > > > There appears to be a bug in which smartctl will try to report status on other devices that are attached to the Smart Array controller. These devices include expanders, JBODS and the controller itself. > > > > For example, in my setup with 3 disks if I try to look at the fourth one I get this error: > > > > [root@dl380g81 ~]# smartctl -i -d cciss,3 /dev/sg2 > > smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.10.0-123.el7.x86_64] (local build) > > Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org > > > > /dev/sg2 [cciss_disk_03] [SCSI]: Device open changed type from 'sat,auto+cciss' to 'cciss' > > === START OF INFORMATION SECTION === > > Vendor: PMCSIERA > > Product: SRCve8x12G > > Revision: RevB > > >> Terminate command early due to bad response to IEC mode page > > A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. > > > > But if I use a -H it reports ok. > > > > [root@dl380g81 bin]# smartctl -H -d cciss,3 /dev/sg2 > > smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.10.0-123.el7.x86_64] (local build) > > Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org > > > > /dev/sg2 [cciss_disk_03] [SCSI]: Device open changed type from 'sat,auto+cciss' to 'cciss' > > === START OF READ SMART DATA SECTION === > > SMART Health Status: OK > > > > ----------------------------------- > > > > You need to make sure you are using the sg node for the controller and not the drive device node. I noticed this command, > > root 10756 10720 0 14:49 ? 00:00:00 smartctl -dcciss,4 -H /dev/sda > > > > which isn't exactly right (although it works). It should be > > > > smartctl -d cciss,4 -H /dev/sgX > > > > where /dev/sgX is the controller sg node. > > > > I'm guessing that the device at index 0 is not a disk drive. You could check and see what it the device is by running smartctl -i -d cciss,0 /dev/sgX > > > > cciss_vol_status -V /dev/sgX may also show all the other devices connected to the Smart Array. > > > > You will probably have to adjust your Nagios setup to only look at disk drives. > > > > --------------------------------------- > > > > I suspect the I/O errors you have seen are related to a bad drive. Do you have access to hpssacli? There is a diag mode in it to output the Smart Array controller diagnostics information. You could use this to see if any drives are bad. > > > > Hope this helps, > > Scott > > > > -----Original Message----- > > From: Justin Pryzby [mailto:[email protected]] > > Sent: Thursday, 18 December, 2014 5:12 PM > > To: ISS StorageDev > > Cc: [email protected] > > Subject: [hpsa] regular freezes with smartctl -d cciss,0 > > > > Hi All, > > > > I'm reporting a problem with the hpsa driver that's triggered by smartctl, thus the two lists. Please copy me in any responses. > > > > I have a nagios check that runs smartctl -d cciss,$i looping around all drives, and has been running successfully on a bunch of servers. The newest server (most recent install, if not hardware) uses the hpsa driver and SATA drives, (/dev/sd?). This server has a 2-drive RAID1, plus spare, and a 5-drive RAID5, plus spare, so we're looping around cciss,0 through cciss,9, and cciss,11 says "No such device" and the loop is terminated. > > > > NOTE: I noticed that -dcciss,10 seems to work (but there aren't 11 disks); but, when using -dsat+cciss,10, it has: this Read Device Identity failed: empty IDENTIFY data. In both cases, 11 has "No such device". > > > > Here's the full output of one invocation: > > smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.16.0-28-generic] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org > > > > /dev/sda [cciss_disk_00] [SAT]: Device open changed type from 'sat,auto+cciss' to 'sat' > > === START OF READ SMART DATA SECTION === SMART Status not supported: Incomplete response, ATA output registers missing SMART overall-health self-assessment test result: PASSED > > Warning: This result is based on an Attribute check. > > > > Since enabling that check, the server has been freezing regularly. Yesterday I brought up a serial console, to see these messages: > > > > [ 361.453116] INFO: task smartctl:3746 blocked for more than 120 seconds. > > [ 361.460620] Tainted: G W I 3.16.0-28-generic #38-Ubuntu > > [ 361.468013] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > > Also happened on this kernel: > > > > [ 8650.079319] Tainted: G W I 3.13.0-43-generic #72-Ubuntu > > [ 8650.086635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > [ 8659.251188] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 12, t=15015 jiffies, g=275781, c=275780, q=0) > > > > with: > > root 10756 10720 0 14:49 ? 00:00:00 smartctl -dcciss,4 -H /dev/sda > > > > and many processes starting to hang a few minutes later: > > > > root 10775 10766 99 14:52 ? 00:02:20 cciss_vol_status -V -s /dev/cciss/c*d0 /dev/sg0 /dev/sg1 /dev/sg2 /dev/sg3 > > root 10837 10836 99 14:52 ? 00:02:04 lvs > > root 10893 10885 0 14:53 ? 00:00:00 dumpe2fs -h /dev/sda1 > > > > Note the "taint", which appears to be due to this: > > [ 0.256948] This system BIOS has enabled interrupt remapping > > [ 0.256948] on a chipset that contains an erratum making that > > [ 0.256948] feature unstable. To maintain system stability > > [ 0.256948] interrupt remapping is being disabled. Please > > [ 0.256948] contact your BIOS vendor for an update > > > > This is running Xeon E5620 I've installed the microcode package, but no updates were loaded. > > > > From hpsa: > > > > [ 8.321717] HP HPSA Driver (v 3.4.4-1) > > [ 8.326187] hpsa 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control > > [ 8.334605] hpsa 0000:04:00.0: MSIX > > [ 8.338733] hpsa 0000:04:00.0: irq 66 for MSI/MSI-X > > [ 8.338741] hpsa 0000:04:00.0: irq 67 for MSI/MSI-X > > [ 8.338748] hpsa 0000:04:00.0: irq 68 for MSI/MSI-X > > [ 8.338755] hpsa 0000:04:00.0: irq 69 for MSI/MSI-X > > [ 8.338763] hpsa 0000:04:00.0: irq 70 for MSI/MSI-X > > [ 8.338768] hpsa 0000:04:00.0: irq 71 for MSI/MSI-X > > [ 8.338773] hpsa 0000:04:00.0: irq 72 for MSI/MSI-X > > [ 8.338780] hpsa 0000:04:00.0: irq 73 for MSI/MSI-X > > [ 8.338785] hpsa 0000:04:00.0: irq 74 for MSI/MSI-X > > [ 8.338790] hpsa 0000:04:00.0: irq 75 for MSI/MSI-X > > [ 8.338795] hpsa 0000:04:00.0: irq 76 for MSI/MSI-X > > [ 8.338800] hpsa 0000:04:00.0: irq 77 for MSI/MSI-X > > [ 8.338805] hpsa 0000:04:00.0: irq 78 for MSI/MSI-X > > [ 8.338812] hpsa 0000:04:00.0: irq 79 for MSI/MSI-X > > [ 8.338816] hpsa 0000:04:00.0: irq 80 for MSI/MSI-X > > [ 8.338821] hpsa 0000:04:00.0: irq 81 for MSI/MSI-X > > [ 8.338878] hpsa 0000:04:00.0: Logical aborts not supported > > [ 8.390154] hpsa 0000:04:00.0: hpsa0: <0x323a> at IRQ 66 using DAC > > [ 8.410709] scsi6 : hpsa > > [ 8.416074] hpsa 0000:04:00.0: RAID device c6b3t0l0 added. > > [ 8.416078] hpsa 0000:04:00.0: Direct-Access device c6b0t0l0 added. > > [ 8.416080] hpsa 0000:04:00.0: Direct-Access device c6b0t0l1 added. > > > > Some other notable kernel boot messages: > > > > [ 1.131923] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug > > [ 1.314604] pci 0000:00:1f.0: can't claim BAR 13 [io 0x0800-0x087f]: address conflict with ACPI CPU throttle [io 0x0810-0x0815] > > [ 1.396232] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' > > [ 3.153246] ERST: Failed to get Error Log Address Range. > > [ 3.117767] [Firmware Warn]: GHES: Poll interval is 0 for generic hardware er > > ror source: 1, disabled. > > [ 3.128268] GHES: APEI firmware first mode is enabled by WHEA _OSC. > > [ 5.991621] i8042: Failed to disable AUX port, but continuing anyway... Is th > > is a SiS? > > [ 6.000596] i8042: If AUX port is really absent please use the 'i8042.noaux' option > > > > I just ran it in a loop, using the latest smartmon version (6.3), and triggered badness pretty quickly (<5min), with the addition of IO errors .. however I haven't been able to reproduce those, so may have been "real" or some spurious side effect. > > > > [ 304.990869] end_request: I/O error, dev sda, sector 256149504 [ 305.195133] end_request: I/O error, dev sda, sector 256149504 [ 309.128312] end_request: I/O error, dev sda, sector 256149504 [ 309.272488] end_request: I/O error, dev sda, sector 138677936 [ 309.279068] Aborting journal on device sda1-8. > > [ 309.825224] end_request: I/O error, dev sda, sector 138676224 [ 309.831717] Buffer I/O error on device sda1, logical block 17334272 [ 309.838872] JBD2: Error -5 detected when updating journal superblock for sda1-8. > > [ 310.590236] end_request: I/O error, dev sda, sector 2048 [ 310.596238] Buffer I/O error on device sda1, logical block 0 [ 310.602719] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal [ 310.612538] EXT4-fs (sda1): Remounting filesystem read-only [ 310.618830] EXT4-fs (sda1): previous I/O error to superblock detected [ 311.419315] end_request: I/O error, dev sda, sector 2048 [ 311.425318] Buffer I/O error on device sda1, logical block 0 [ 340.093063] end_request: I/O error, dev sda, sector 230757056 [ 340.099624] EXT4-fs error (device sda1): ext4_find_entry:1312: inode #7212241: comm kerneloops: reading directory lblock 0 [ 340.112070] EXT4-fs (sda1): previous I/O error to superblock detected [ 341.210480] end_request: I/O error, dev sda, se ctor 2048 [ 341.216510] Buffer I/O error on device sda1, logical block 0 [ 361.453116] INFO: task smartctl:3746 blocked for more than 120 seconds. > > [ 361.460620] Tainted: G W I 3.16.0-28-generic #38-Ubuntu > > [ 361.468013] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > [ 404.706013] end_request: I/O error, dev sda, sector 0 [ 405.278753] end_request: I/O error, dev sda, sector 312512256 [ 405.450981] end_request: I/O error, dev sda, sector 312512424 [ 406.412259] end_request: I/O error, dev sda, sector 0 [ 407.437599] end_request: I/O error, dev sda, sector 8 [ 408.350794] end_request: I/O error, dev sda, sector 0 [ 409.284030] end_request: I/O error, dev sdb, sector 2147484032 [ 409.404178] end_request: I/O error, dev sdb, sector 2147484144 [ 410.129138] end_request: I/O error, dev sdb, sector 512 [ 411.130442] end_request: I/O error, dev sdb, sector 520 [ 411.923503] end_request: I/O error, dev sdb, sector 512 [ 413.073001] end_request: I/O error, dev sda, sector 278976384 [ 413.305307] end_request: I/O error, dev sda, sector 278976496 [ 413.421464] end_request: I/O error, dev sda, sector 2048 [ 414.134401] end_request: I/O error, dev sda, sector 2056 [ 416.024908] end_request: I/O error, dev sda, sector 2048 [ 417.090238] end_request: I/O error, dev sdb, sector 2181038464 [ 417.342625] end_request: I/O error, dev sdb, sector 2181038576 [ 417.643037] end_request: I/O error, dev sdb, sector 2147484160 [ 418.363966] end_request: I/O error, dev sdb, sector 2147484168 [ 419.357273] end_request: I/O error, dev sdb, sector 2147484160 [ 421.175150] end_request: I/O error, dev sda, sector 312510336 [ 421.371310] end_request: I/O error, dev sda, sector 312510448 [ 422.147928] end_request: I/O error, dev sda, sector 278978560 [ 423.148761] end_request: I/O error, dev sda, sector 278978568 [ 424.581910] end_request: I/O error, dev sda, sector 278978560 [ 425.282476] end_request: I/O error, dev sdb, sector 0 [ 426.511531] end_request: I/O error, dev sdb, sector 7813837056 [ 427.552337] end_request: I/O error, dev sdb, sector 7813837216 [ 428.417064] end_request: I/O error, dev sdb, sector 0 [ 429.293762] end_request: I/O error, dev sdb, sector 8 [ 430.154463] end_request: I/O error, dev sdb, sector 0 > > > > Thanks for any help you can provide. > > > > Justin ------------------------------------------------------------------------------ Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/