Re: random system crashes

----- Original Message -----
From: "Alex Kramarov" <[email protected]>
To: <[email protected]>
Sent: Tuesday, October 15, 2002 12:14 AM
Subject: Re: random system crashes


>
> >Given how similar Marco and Alex's hardware setup appears to be, perhaps
a
> hint at the root of the problem might be isolated by a straightforward
> detailed comparison of the hardware, drivers, distribution, loaded
> modules, or even the actual TUX configuration itself...?
>
> I have no problem with that, but we need to have a substantial base for
> comparacent.
>
> everyone who is successfully running tux and willing to provide that info,
> please reply to the list.
> i believe we need at least 10 ppl to get started ...

Here's my slightly more detailed contribution of my abortive tux setup
details: this server completely froze hung roughly every 2 to 7 days,
sometimes responding to icmp ping packets (with good response times(!)) on
one or more of the NIC interfaces.  TUX was serving around 30 hits a second.

Hardware:

Dell PowerEdge 1650
2 * PIII 1.26GHz processors, 512K Cache.
4*1Gb 133MHz SDRAM DIMMs
PERC3-DI SCSI RAID controller
3*18Gb 15Krpm Ultra 160 SCSI drives (RAID-5)
Dual Intel(R) PRO/1000 Gigabit ethernet NICs.


Drivers:

SCSI subsystem driver Revision: 1.00
Red Hat/Adaptec aacraid driver, May 14 2002
scsi0 : percraid
  Vendor: DELL      Model: PERCRAID RAID5    Rev: 0001
  Type:   Direct-Access                      ANSI SCSI revision: 02

Intel(R) PRO/1000 Network Driver - version 4.2.17


Software (at the time of TUX experimentation back in late May):

Debian Woody, with 2.4.18 kernel, patched with tux2-full-2.4.18-final-A3
(statically).
Reiserfs on both the web root and logging partition (version 3.6.25).
TUX userspace gubbins version 2.2.7.
Apache/1.3.24 with PHP/4.1.2 mod_perl/1.26 mod_gzip/1.3.19.1a


Configuration:

Well, originally Apache served about 15 domains listening on one IP address,
and thttpd listening on another IP was used for serving images and binaries
for another 5 domains (linked to from the HTML that Apache was spewing).
One NIC was connected through to the external router; the other used for a
very low traffic private LAN segment.  No console messages were recorded on
the serial console, nor anything whatsoever abnormal written in the system's
logs.

On changing to TUX, Apache was shifted to listen to all IPs on port 8080,
and TUX was set up as:

/proc/net/tux/0/listen/0   http://0.0.0.0:80
/proc/net/tux/1/listen/0   http://0.0.0.0:80

/proc/sys/net/tux/404_page          404.html
/proc/sys/net/tux/ack_pingpong             1
/proc/sys/net/tux/all_userspace            0
/proc/sys/net/tux/cgi_cpu_mask            -1
/proc/sys/net/tux/cgi_gid                 -1
/proc/sys/net/tux/cgi_inherit_cpu          0
/proc/sys/net/tux/cgi_uid                 -1
/proc/sys/net/tux/cgiroot           /www01/
/proc/sys/net/tux/clientport            8080
/proc/sys/net/tux/compression              0
/proc/sys/net/tux/cpu_offset               0
/proc/sys/net/tux/default_vhost     default   *
/proc/sys/net/tux/defer_accept             1
/proc/sys/net/tux/documentroot      /www01/
/proc/sys/net/tux/Dprintk                  0
/proc/sys/net/tux/extra_html_header
/proc/sys/net/tux/extra_html_header_       0
/proc/sys/net/tux/ftp_log_retr_only        0
/proc/sys/net/tux/ftp_login_message        0
/proc/sys/net/tux/ftp_subdocroot
/proc/sys/net/tux/ftp_virtual_server       0
/proc/sys/net/tux/ftp_wait_close           1
/proc/sys/net/tux/generate_cache_con       1
/proc/sys/net/tux/generate_etags           1
/proc/sys/net/tux/generate_last_mod        1
/proc/sys/net/tux/hide_unreadable          1
/proc/sys/net/tux/http_dir_indexing        0
/proc/sys/net/tux/http_subdocroot
/proc/sys/net/tux/ip_logging               1
/proc/sys/net/tux/keepalive_timeout        0
/proc/sys/net/tux/log_incomplete           0
/proc/sys/net/tux/logentry_align_ord       5
/proc/sys/net/tux/logfile           /var/log/tux
/proc/sys/net/tux/logging                  1
/proc/sys/net/tux/mass_hosting_hash        0
/proc/sys/net/tux/max_backlog           2048
/proc/sys/net/tux/max_connect          10000
/proc/sys/net/tux/max_free_requests     1000
/proc/sys/net/tux/max_header_len        3000
/proc/sys/net/tux/max_keepalives       10000
/proc/sys/net/tux/max_object_size          0
/proc/sys/net/tux/max_output_bandwid       0
/proc/sys/net/tux/mode_allowed             4
/proc/sys/net/tux/mode_forbidden           0
/proc/sys/net/tux/noid                     0
/proc/sys/net/tux/push_all                 0
/proc/sys/net/tux/redirect_logging         0
/proc/sys/net/tux/referer_logging          1
/proc/sys/net/tux/strip_host_tail          0
/proc/sys/net/tux/TDprintk                 0
/proc/sys/net/tux/threads                  2
/proc/sys/net/tux/virtual_server           1
/proc/sys/net/tux/zerocopy_header          1
/proc/sys/net/tux/zerocopy_parse           1
/proc/sys/net/tux/zerocopy_sendfile        1

* Whilst recreating this setup in preparation for a new attempt of
persuading it to work, i've found that I've had to specify a 'sensible'
setting for the undocumented default_vhost in order to get vhosting to work
at all (for the purposes of experimentation now running TUX listening on all
port 8080's, connecting to respective live Apaches and thttpds on clientport
80.)

I believe the above kernel settings to largely be the default options, but
with the following /etc/sysctl.tux applied:

net/tux/clientport=8080
net/tux/virtual_server=1
net/tux/referer_logging=1
net/tux/redirect_logging=0

The TUX user-space utilities itself was invoked using a debianised version
of the Red Hat /etc/init.d script, the key line being:

/usr/sbin/tux -d -t $TUXTHREADS -r $DOCROOT $EXTRAOPTS $TUXMODULES

with the proviso of references to /var/lock/subsys/tux being commented out,
due to the apparent lack of a /var/lock/subsys tree on Debian.


We have to keep our logs and maintain referrer logs in order to keep tabs on
traffic, especially people direct-linking to the images server from
off-site.

Off the top of my head, this is the kind of problem-solving strategies that
were adopted back in May/June before reverting to Apache + thttpd:

Hypothesis: Assumption that it's a problem with logging, due to logs being
rotated mid-write - or due to SNMP utilities dipping into the logfile to
measure hits-per-second.

Test: Write various elaborate 'dances' for log rotation, e.g.:

echo 0 > /proc/sys/net/tux/logging
cat /var/log/tux > /var/log/apache/archive/tuxlogs/tuxbin.tmp
sleep 5
cat /dev/null > /var/log/tux
echo 1 > /proc/sys/net/tux/logging

...and disabled SNMP utilities

Result: No detectable change in behaviour.

Hypothesis: Assumption that it's due to files being altered in userspace in
the webroot whilst being served from kernelspace (many static HTML files
that TUX was serving are automatically pre-generated by cron jobs at regular
intervals to reduce processing load on the server).

Test: Look for patterns in crashing times with the 15-minutely cron jobs
regenerating index files etc.

Result: No pattern detected in the crashing times (other than a slight
preference for 2am -> 7am).

And that is as far as we got before ditching TUX and beating a hasty retreat
back towards thttpd.

Untested Hypotheses in decreasing order of plausability (imo):
* Caused by running 2 (>1) TUX threads?
* Caused by referrer logging being enabled?
* Clashing with some unexpected module/kernel feature?
* An issue with the reiserfs implementation.
* Caused by zerocopy problems with Intel NICs or RAID?
* A problem with >1GHz PIIIs?
* Malformed TCP/IP packets somehow affecting the system?

The pattern of hangs was vaguely similar to the pattern of a rare client or
user perhaps hitting the server every few days, so I'm quite taken with the
idea of referrer logging possibly being buggy.  I simply cannot afford to
test it in production, however - and having a debug cycle of a few days is
tedious at best.  My next best bet is to shove Debian on a dual Athlon 1800+
MP Tyan Tiger MP based machine (but one lacking the Gbit NICs and SCSI RAID)
and hammer it as best I can - but sacrificing my main workstation for such a
potentially longterm project isn't really practically viable.

I'd also certainly be interested to see if TUX debugging had anything
interesting to say.

Anyway, I hope that this helps the cause and I haven't missed anything too
obvious out - if Ingo ever returns in from the cold, I'd be more than happy
to send him a full gettuxconfig dump.

M.