RE: PERC 3/Di Drive failure...

(Copying linux-aacraid-devel, this is more relevant there)

> Nov 11 04:19:19 foo kernel: aacraid:ID(1:02:0); Error Event 
> [command:0x28]
> Nov 11 04:19:19 foo kernel: aacraid:ID(1:02:0); Medium Error 
> [k:0x3,c:0x11,q:0x1]
> Nov 11 04:19:19 foo kernel: aacraid:ID(1:02:0); Read Retries Exhausted

Ouch, you definitely lost a disk, channel 1, id 2.

> Nov 11 05:52:51 foo kernel: aacraid:ID(1:02:0) Medium Error, 
> LBN Range 56084398:56084479
> Nov 11 05:52:52 foo kernel: aacraid:ID(1:02:0) Starting BBR sequence

and it tried to remap the bad blocks (BBR is Bad Block Remap) onto good
spare blocks on the disk, but that took some time.  The controller took
longer to try to remap the blocks than Linux was preparted to wait.

> Nov 11 12:24:57 foo kernel: aacraid:ID(1:02:0) Starting BBR sequence
> Nov 11 12:24:57 foo kernel: aacraid:RAID5 Container 0 Drive 
> 1:2:0 Failure
> Nov 11 12:24:57 foo kernel: aacraid:RAID5 Failover Container 
> 0 No Failover Assigned

But the blocks couldn't be remapped, and the controller noted the drive
failure

> followed by these a few seconds later:
> 
> Nov 11 12:25:18 foo kernel: aacraid:Drive 1:2:0 returning error
> <Drive pulled from system here>
> Nov 11 12:26:33 foo kernel: aacraid:ID(1:02:0) - Drive spindown failed
> Nov 11 12:26:33 foo kernel: aacraid:Drive 1:2:0 offline on 
> container 0:

90 seconds or so elapsed between when the OS issued the request, and when
the controller was ready to report complete failure to the OS.


> So, 2 questions:  1) What caused the failure

Disks go bad.

> 2) Why did ext3 explode? 

This is the better question.  The SCSI mid-layer's timeout for I/O requests
is 60 seconds.  After 60 seconds, it invokes the controller's error handling
code to try to abort the command.  But, the aacraid error handling code (at
least in 2.4.20-20.9 kernel) is a NOOP, so the SCSI mid-layer just hung,
hence ext3 hung.  That's both a shortcoming in the mid-layer (the driver
told it that it couldn't abort, mid-layer can't handle that answer), and of
the driver (it should try to abort somehow.)

Good news is recent aacraid versions in test (v1.1.4) do implement an abort
mechanism, which should reduce these types of problems, if not eliminate
them completely.  In this case the firmware on the controller was still
running, so the driver should just wait until all its previous commands
complete, then start accepting new commands.  I don't believe the 2.4.x SCSI
mid-layer error handling is yet prepared to handle being told "no, I can't
abort" by the driver; that work was happening in 2.5.x I believe.

Thanks,
Matt

--
Matt Domsch
Sr. Software Engineer, Lead Engineer
Dell Linux Solutions www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com

_______________________________________________
Linux-aacraid-devel mailing list
[email protected]
http://lists.us.dell.com/mailman/listinfo/linux-aacraid-devel
Please read the FAQ at http://lists.us.dell.com/faq or search the list archives at http://lists.us.dell.com/htdig/