RE: PE 1650 with Perc 3/Di

On Fri, 2004-03-05 at 17:00, Salyzyn, Mark wrote: 
> Thank You Thank You Thank You ....

No, Mark, Thank you!  Given the scsi errors, I was completely focused on
the disks, the scsci controller, or the aacraid driver.  The only other
dell machine in our shop with a perc controller is also having stability
problems (turns out it is the Seagate disk firmware issue), so I
followed the red herring to a very frustrated conclusion.

> NMI_SECONDARY_ATU_ERROR is a catastrophic failure on the I960 internal
> PCI bus. Being generically catastrophic, the Firmware `just stops' if no
> corrective action is indicated.
> 
> External sources of this fault can be PCI bus reset, and *possibly* a
> sizing request to the BARs (I know this is the primary side, but it
> still causes a transfer error). Make sure you do not have any utilities
> that are doing a pci trace of any form. Other PCI targets on the machine
> could be faulty, triggering a PCI bus reset. Since the machine turns to
> mud, you have no idea what else has been damaged, the Adapter is just
> the highest activity source and thus feels the pain. You could have bad
> memory (a stretch, I admit) for example ...

Upon the advice of a collegue, I ran the Dell 32bit diag tools to test
the memory and all of the hardware, even though the controller bios
tests and afacli dumps were all kosher.  It turns out that the
controller for the dual power supplies was flakey.

A new mainboard, pci riser, p/s controller, and plug adapter has made
the IMPI errors disappear, and I have yet to really do any stress tests.

> Internal sources can be internal hardware failure of Firmware bug.
> 
> Although Dell may have to make a determination if there is a hardware
> problem on your machine, I feel that this *may* be a good machine to
> analyze to see if there is a recoverable condition here. The Firmware
> Engineer is probably going to have to instrument up to dump the entire
> I960's PCI configuration register set at point of failure and print it
> out the UART (a serial debug connector added to the adapter) to trace
> this down. They may be able to just work on your Firmware if you are
> willing ...

Since the hardware replacement has, at first glance, helped, would
anyone still be interested in doing any testing?  If any engineer or
programmer wants to try to add better logging or recovery handling for
pci glitches like this, I would gladly help any way I could.  If this is
the case, let me know by the end of the week, otherwise I will send the
old parts back to Dell.

Sorry for the false alarm, and thanks to all who replied.

Matt

> Please contact Dell Technical Support to make sure their cookbook does
> not point to a bad motherboard or card on your system. If they find this
> is pandemic and can be reproduced within Dell, or if not and endemic a
> post-mortem on *your* machine is in order.
> 
> Sincerely -- Mark Salyzyn
> 
> --Original Message--
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Matt Soccio
> Sent: Friday, March 05, 2004 2:12 PM
> To: [email protected]
> Subject: PE 1650 with Perc 3/Di
> 
> I have a Poweredge 1650 with the Perc 3/Di raid controller, and have not
> been able to keep it up an running for more than a week or so at a
> time.  The mainboard was recently replaced on a pre-emptive basis for an
> unrelated problem, but the problem existed before the replacement as
> well.  After replacement, I updated all bios and esm firmware files that
> were available on dell's website.
> 
> I was running a 2.4.22 kernel with the stock aacraid driver and it would
> crash about every 8-12 days with errors like this:
> 
> Feb 29 18:43:55 phoenix kernel: aacraid:        NMI ISR:
> NMI_SECONDARY_ATU_ERROR
> Feb 29 18:47:13 phoenix kernel: scsi: device set offline - command error
> recover failed: host 0 channel 0 id 0 lun 0
> Feb 29 18:47:13 phoenix kernel: SCSI disk error : host 0 channel 0 id 0
> lun 0 return code = 6000000
> Feb 29 18:47:13 phoenix kernel:  I/O error: dev 08:04, sector 15466552
> Feb 29 18:47:13 phoenix kernel:  I/O error: dev 08:02, sector 3211272
> 
> Based on info on this list, I moved to a 2.4.25 kernel but kept the
> 2.4.22 aacraid driver.  It crashed the same way.
> 
> I have since got the latest (1.1.5) version of the driver from Adaptec's
> site and it crashed in less than 48 hours with different errors, but I
> could not grab them as it locked up harder than usual.
> 
> The errors flooding the console were about scsi hang? and scsi bus not
> responding.
> 
> The really strange part of this is that I can't reproduce the problem,
> it just happens randomly.  If I run bonnie++ for a couple hours with a
> few gigs of I/O it chugs right along, but then it'll be sitting there
> doing nothing the next day and just stop responding.  I just can't find
> the trigger for the failure.
> 
> Is there any way to run linux with this hardware?  Is there some way to
> bypass this hardware and run a software raid, or would I have to buy a
> new scsi controller?  With the disk backplane the way it is in the
> server, would I even be able to put a new controller in there?
> 
> Any suggestions or pointers are appreciated.
> 
> Matt
> 
> _______________________________________________
> Linux-aacraid-devel mailing list
> [email protected]
> http://lists.us.dell.com/mailman/listinfo/linux-aacraid-devel
> Please read the FAQ at http://lists.us.dell.com/faq or search the list
> archives at http://lists.us.dell.com/htdig/
> 
> _______________________________________________
> Linux-aacraid-devel mailing list
> [email protected]
> http://lists.us.dell.com/mailman/listinfo/linux-aacraid-devel
> Please read the FAQ at http://lists.us.dell.com/faq or search the list archives at http://lists.us.dell.com/htdig/
> 

_______________________________________________
Linux-aacraid-devel mailing list
[email protected]
http://lists.us.dell.com/mailman/listinfo/linux-aacraid-devel
Please read the FAQ at http://lists.us.dell.com/faq or search the list archives at http://lists.us.dell.com/htdig/