Re: random system crashes

----- Original Message -----
From: "Alex Kramarov" <[email protected]>
To: <[email protected]>
Sent: Wednesday, October 16, 2002 11:32 AM
Subject: Re: random system crashes


> ok people, now we are getting somewhere.
>
> yesterday, i have done the following on my machine (that was running
steady
> for 2 months !)
>
> echo 1 > /proc/sys/net/tux/logging
> echo 1 > /proc/sys/net/tux/referrer_logging
>
> and in crontab :
>
> */5 * * * * root rm -f /var/log/tux
>
> after 5 hours the machine crashed. good thing i have an automated failover
> to the backup machine - and that my isp people took about 4 hours to find
> out that the machine was down, so that gave me some more time to sleep
today
> ;)
>
> so, this seems to confirm the logging-related theory .
> now, after a restart, i am trying the same setup without referrer_logging
> ...

So, has it crashed yet? ;-)

I've remembered two possibly related items of interest:

On one of the final hangs before reverting to thttpd, I fired up a browser
(IE6 on Win2k, as it happens) at the homepage of the website, and then
alt-tabbed elsewhere whilst it loaded.  After it had failed to load after 10
seconds or so, I started making checks to find out whether the network was
down, etc.  I then left absentmindedly left the stalled browser window
sitting in the foreground whilst trying to contact the ISP to get the box
powercycled.

And after about 3 minutes of being completely stalled, the first 1500 bytes
or so of HTML off the page actually loaded in the browser.

It's quite likely that this is a red herring, however - the machine was
powercycled before much more investigation could be performed, and it's
quite possible that other processes were running pathalogically wild on the
machine.  Alternatively, there could have been network issues that I failed
to pick up on - and also I was behind a transparent web proxy at the time,
so the incomplete page could well have been purely generated by that.  Or is
this in fact symptomatic of a variant of the infamous 99% CPU bug?  Is that
so bad that a dual 1.26GHz PIII server would be this badly crippled?  SSH
access was completely unresponsive and refused to reconnect.


Secondly:
If referrer logging appears to be to blame, presumably a vaguely reasonable
diagnostic procedure would be to take a sample of the Apache/thttpd logs off
a live and busy site for a period of a few days, cull out all the GET lines,
and then feed them at speed into a debugging TUX test-server.

Obviously this would neglect any other malformed request details which may
be to blame (let alone subtleties about non-GET operations, and the original
network-level details) - but might provide a pretty good soak test dataset
with which to confirm a reproducible crash scenario which could then be
debugged if the problem did indeed lie solely in the referrer string part of
the query.


M.