RE: kernel: aacraid: Host adapter reset request. SCSI hang ?

I was `quick' to release this info because I believe it is `low risk', a 2,
but you are right in deferring to someone else to experiment. The only
problem is that there is never a pristine bug ... ;-/

Enjoy the weekend, we look forward to whatever experimental results happen.

Sincerely -- Mark Salyzyn

-----Original Message-----
From: Ivan Kluzak [mailto:[email protected]]
Sent: Friday, June 13, 2003 5:47 PM
To: Salyzyn, Mark; [email protected]
Subject: RE: kernel: aacraid: Host adapter reset request. SCSI hang ?



Gotcha.  Well our server is currently in production but
on the same token if it already has issues it can't get much 
worse right?  I take that back, we havn't lost anything terrible
yet as we are running a journaling file system and the nature of 
the errors havn't caused any real data loss.

On scale of 1 - 10 with 10 being the most risky how risky do you 
think this patch is.  I do backup this server nightly also, but 
just so I have a vague idea of what to expect.

It's my policy not to make major changes on a friday so i'm not 
going to install it tonight, but I am likely to schedule a maintinence
window early next week and could put it in then.

Does anyone else on this list have a non-production server that they
could try this patch on to see the results?  Just curious.

It'd be nice to see if this could confirm or disprove the timeout theory.

-Ivan



-----Original Message-----
From: Salyzyn, Mark [mailto:[email protected]]
Sent: Friday, June 13, 2003 11:22 AM
To: 'Ivan Kluzak'; [email protected]
Subject: RE: kernel: aacraid: Host adapter reset request. SCSI hang ?


No, this is not `in there' because it is purely an experiment. Frankly, I
would be *disturbed* at the long timeouts (5 minutes to first attempt at
recovery) on most systems, but if you have a problem with panics, you chose
the lesser of two evils. Should patch to almost any kernel variant once you
dig into drivers/scsi/aacraid. It's small enough you could add it manually
and check it out.

This is *not* a solution, just a workaround based on your problem for now
until the real cause of the fire and damnation that results when the cards
and the SCSI subsystem error recovery collide. If the following does not
solve the problem, and simply delays it, then we will need to explore
elsewhere. There is the possibility that a card fully loaded with commands
can take more than one minute to complete a specific request (either through
mild starvation or other reason), so if this is the case, then it may be a
solution to try 90 seconds instead of 300.

Sincerely -- Mark Salyzyn

-----Original Message-----
From: Ivan Kluzak [mailto:[email protected]]
Sent: Friday, June 13, 2003 11:56 AM
To: Salyzyn, Mark; [email protected]
Subject: RE: kernel: aacraid: Host adapter reset request. SCSI hang ?



Hi Mark,

Which kernel source version is this patch against?   I just saw also that
kernel 2.4.21 was released this morning.  Is it already in there?

-Ivan


-----Original Message-----
From: Salyzyn, Mark [mailto:[email protected]]
Sent: Friday, June 13, 2003 8:50 AM
To: 'Ivan Kluzak'; [email protected]
Subject: RE: kernel: aacraid: Host adapter reset request. SCSI hang ?


Hmmm.

I am currently, on a separate investigation, changing the timeouts from the
default 60 to 300 on an experiment.

Anyone willing to try the following on drivers/scsi/aacraid/aachba.c:

--- aachba.c.old	Fri Jun 13 09:12:33 2003
+++ aachba.c	Wed Jun 11 09:32:39 2003
@@ -1107,6 +1107,12 @@
 			 *	corresponds to a container. Needed to
convert
 			 *	containers to /dev/sd device names
 			 */
+#define AAC_EXTENDED_TIMEOUT 300
+#if (defined(AAC_EXTENDED_TIMEOUT))
+			if(scsicmd->eh_state != SCSI_STATE_QUEUED) {
+				mod_timer(&scsicmd->eh_timeout, jiffies +
(AAC_EXTENDED_TIMEOUT * HZ));
+			}
+#endif
 			spin_unlock_irq(&io_request_lock);
 			fsa_dev_ptr->devno[cid] =
DEVICE_NR(scsicmd->request.rq_dev);
 			ret = aac_read(scsicmd, cid);
@@ -1115,6 +1120,11 @@
 
 		case SS_WRITE:
 		case SM_WRITE:
+#if (defined(AAC_EXTENDED_TIMEOUT))
+			if(scsicmd->eh_state != SCSI_STATE_QUEUED) {
+				mod_timer(&scsicmd->eh_timeout, jiffies +
(AAC_EXTENDED_TIMEOUT * HZ));
+			}
+#endif
 			spin_unlock_irq(&io_request_lock);
 			ret = aac_write(scsicmd, cid);
 			spin_lock_irq(&io_request_lock);

Sincerely -- Mark Salyzyn

-----Original Message-----
From: Ivan Kluzak [mailto:[email protected]]
Sent: Friday, June 13, 2003 9:21 AM
To: [email protected]
Subject: RE: kernel: aacraid: Host adapter reset request. SCSI hang ?



I didn't see any comments to the effect as this is primarily a Dell list
is it not?  But this isn't a Dell specific problem.  It is an aacraid 
problem to be sure.  I have a generic server Dual Xeon with an Adaptec
2200S raid controller and get these exact same errors for no apparent 
reason.  I thought for a while that heavy disk access could cause them but
now i'm not even certain of that.

There was talk for a while that the driver just wasn't waiting long enough
for the RAID card to respond and when it times out prematurly the driver
then
takes the volume offline which on our mail server causes a fair bit of a 
catastrophy :)  Here is a log excerpt if anyone is interested

May 27 14:13:19 Clear kernel: aacraid:ID(1:10:0) Abort Time-out. Resetting
bus.
May 27 14:13:19 Clear kernel: aacraid:ID(1:10:0) Abort Time-out. Resetting
bus.
May 27 14:13:20 Clear kernel: aacraid: Host adapter reset request. SCSI hang
?
May 27 14:13:20 Clear last message repeated 2 times
May 27 14:13:20 Clear kernel: SCSI disk error : host 0 channel 0 id 1 lun 0
return code = 6000000
May 27 14:13:20 Clear kernel:  I/O error: dev 08:11, sector 3624
May 27 14:13:20 Clear kernel:  I/O error: dev 08:11, sector 3632
May 27 14:13:20 Clear kernel: SCSI disk error : host 0 channel 0 id 1 lun 0
return code = 6000000
May 27 14:13:20 Clear kernel:  I/O error: dev 08:11, sector 109799384
May 27 14:13:20 Clear kernel:  I/O error: dev 08:11, sector 109799392
May 27 14:13:20 Clear kernel: SCSI disk error : host 0 channel 0 id 1 lun 0
return code = 6000000
May 27 14:13:20 Clear kernel:  I/O error: dev 08:11, sector 3368
May 27 14:13:20 Clear kernel:  I/O error: dev 08:11, sector 109799384
May 27 14:13:20 Clear kernel: journal-601, buffer write failed
May 27 14:13:20 Clear kernel:  I/O error: dev 08:11, sector 3376
May 27 14:13:20 Clear kernel: kernel BUG at prints.c:334!
May 27 14:13:20 Clear kernel: invalid operand: 0000
May 27 14:13:20 Clear kernel: CPU:    1
May 27 14:13:20 Clear kernel: EIP:    0010:[<c01b22b8>]    Not tainted
May 27 14:13:20 Clear kernel: EFLAGS: 00010282
May 27 14:13:20 Clear kernel: eax: 00000024   ebx: f5ccc800   ecx: c03639d8
edx: 00000001
May 27 14:13:20 Clear kernel: esi: 00000020   edi: 00000047   ebp: f5ccc800
esp: f7badec0
May 27 14:13:20 Clear kernel: ds: 0018   es: 0018   ss: 0018
May 27 14:13:20 Clear kernel: Process kupdated (pid: 9, stackpage=f7bad000)
May 27 14:13:20 Clear kernel: Stack: c03120f5 c040c920 00000047 f89f5ebc
c01bdfea f5ccc800 c031fc80 00001000 
May 27 14:13:20 Clear kernel:        f56e7e00 0000004a 00000048 00000000
eb4bd800 00000000 00000029 f56e5000 
May 27 14:13:20 Clear kernel:        00000004 c01c2251 f5ccc800 f89f5ebc
00000001 00000006 f89fe590 00000004 
May 27 14:13:20 Clear kernel: Call Trace:    [<c01bdfea>] [<c01c2251>]
[<c01c1464>] [<c01af000>] [<c01472da>]
May 27 14:13:21 Clear kernel:   [<c0146354>] [<c01466cd>] [<c0105000>]
[<c0105000>] [<c01058ce>] [<c01465d0>]
May 27 14:13:21 Clear kernel: 
May 27 14:13:21 Clear kernel: Code: 0f 0b 4e 01 7e 52 31 c0 85 db 74 0e 0f
b7 43 08 89 04 24 e8 
May 27 14:13:57 Clear kernel:   I/O error: dev 08:11, sector 112096
May 27 14:13:57 Clear kernel: zam-7001: io error in reiserfs_find_entry
May 27 14:14:07 Clear kernel:  I/O error: dev 08:11, sector 112096
May 27 14:14:07 Clear kernel: zam-7001: io error in reiserfs_find_entry
May 27 14:18:42 Clear kernel:  I/O error: dev 08:11, sector 112658760
May 27 14:18:42 Clear kernel:  I/O error: dev 08:11, sector 112659640


I believe the driver timeout is still the issue but I am not aware of any
patches
that for sure solve the problem yet.  Alan Cox said that this was an issue
that he
was working on with someone from Adaptec so i'm hoping to see this resolved
in an
upcoming kernel or kernel patch but have no estimates of when that could be.

-Ivan

------------------------------------------------------------------------
Ivan Kluzak
Senior Systems Administrator - Mankato
CP Internet / Prairie Lakes Internet



-----Original Message-----
From: Andy De Petter [mailto:[email protected]]
Sent: Friday, June 13, 2003 6:29 AM
To: [email protected]
Subject: Re: kernel: aacraid: Host adapter reset request. SCSI hang ?
(Andy De Petter)


Adam Newby wrote:

>
>  
>
>>Anyone has come up with any solutions and/or explications on this
behaviour yet? ;)
>>    
>>
>
>I *thought* we had, in that we thought it was a combination of faulty
>disks + faulty firmware. Given your information, this seems not to be
>correct now.
>
>
>  
>

Yup, sucky problem, which apparantly affects more and more people.  I 
have a 2650 2x2.4Ghz in production for months, with exactly the same 
hardware, as the 4 new 2650 servers we acquired.  Only difference is the 
cpu clockspeed (faulty ones are 2x2Ghz).  We also acquired a new 2650 
with 2x2.4Ghz, and it works smoothly too...

I have tried with/without hyperthreads, without any luck: problem keeps 
appearing.  I have just installed RH9 now, with 2.4.20-8smp kernel, so 
gonna check whether Debian <-> RedHat makes a difference.

-Andy

-- 
Andy De Petter - Expert  IT Analyst - [email protected]
Belgacom ANS/EIS/ISA - Carlistraat  2 - 1140 Brussels (Belgium)
Head office: Koning Albert II Laan 27 - 1030 Brussels (Belgium)
Tel +32 (0)2 7061170  -  Fax +32 (0)2 7061150  -  ICQ  #1548957

*** DISCLAIMER ***
This e-mail and any attachments thereto may contain information, which
is confidential and/or protected by intellectual property rights and
are intended for the sole use of the recipient(s) named above. Any use
of the information contained herein (including, but not limited to,
total or partial reproduction, communication or distribution in any
form) by persons other than the designated recipient(s) is prohibited.
If you have received this e-mail in error, please notify the sender
either by telephone or by e-mail and delete the material from any
computer. Thank you for your cooperation.


_______________________________________________
Linux-aacraid-devel mailing list
[email protected]
http://lists.us.dell.com/mailman/listinfo/linux-aacraid-devel
Please read the FAQ at http://lists.us.dell.com/faq or search the list
archives at http://lists.us.dell.com/htdig/

_______________________________________________
Linux-aacraid-devel mailing list
[email protected]
http://lists.us.dell.com/mailman/listinfo/linux-aacraid-devel
Please read the FAQ at http://lists.us.dell.com/faq or search the list
archives at http://lists.us.dell.com/htdig/

_______________________________________________
Linux-aacraid-devel mailing list
[email protected]
http://lists.us.dell.com/mailman/listinfo/linux-aacraid-devel
Please read the FAQ at http://lists.us.dell.com/faq or search the list archives at http://lists.us.dell.com/htdig/