Re: Random system crashes

Quoting Michael Eager ([email protected]):

> Perhaps someone has a suggestion on how to address a problem
> which has me pulling my hair.

Maybe, but:

> Subject: Re: [svlug] Random system crashes

Intermittent system problems, ones that correlate with no pattern, are
by definition ones you have no means to solve.  You've defined the
problem as this in your initial subject header.  Are you _sure_ it's
random?  ;->  Seems to me, maybe you mean that you'd like other people's
help finding what causes the problem (or at least to make the symptom go
away), which would entail it _not_ being random.

> I have a server which periodically freezes and becomes unresponsive.

'Periodically'?  ;->   Perhaps you actually mean aperiodically.  If the
system failed every 42 minutes, for example (an example of
'periodically'), then that would be a huge help in solving the problem.

I see further down that you clarify that the freezes 'appear to be
random', which I take to mean merely that you didn't spot a pattern
(and sure also implies that it's not periodic).  

You do, commendably, provide additional diagnostic detail, e.g., you're
pretty sure there's not heat buildup, and freeze-ups happen both with
light and heavy loading.  (I once had an embarrassing problem with a
homebuilt AMD K6 system where it became unstable once up to full normal
operating heat but never before.  Turned out, I'd mounted the CPU heat
sink 180 degrees rotated such that it was in contact with only about 15%
of the CPU top surface.  The pattern of what had to happen before
spontaneous reboot was the key clue.)

By the way, it would have been incredibly useful if you had included a
guesstimate of _median time to freeze-up_ -- but you said nothing about
time spans.  For all we know, you might mean on average twice a day, or
you might mean on average once a month.  It makes quite a difference,
between those two examples, about what testing approach is practical.

> I've run memtest86 overnight, but there are no errors.  (Memory
> failures would normally show up as a crash in the syslog.)

Sometimes not at all, in my experience.  Also, I've found to my shock
that memtest86 sometimes utterly fails to find bad RAM.  In 2007, I had
a pair of sticks of 512 MB of ECC SDRAM sticks in my ECC-supporting PIII
server that passed at least most of a day of continuous memtest86 testing,
and yet memtest86 passed them with flying colours.  OTOH, this iterative
Linux kernel compile exposed the problem:

  setterm -blank 0
  while : ; do make clean && make -j NN ; done

...with integer NN set high enough to drive the machine very slightly
into swap as shown by vmstat, so that all the memory was exercised by
the parallel compilation.

Once an overnight freeze-up showed pretty conclusively that there was a
probable RAM problem, moving sticks around and removing some of them but
not others isolated the problem to, not one, in this case, but two bad
ECC SDRAM sticks.  All details here:

http://linuxmafia.com/pipermail/conspire/2006-December/002662.html
http://linuxmafia.com/pipermail/conspire/2006-December/002668.html
http://linuxmafia.com/pipermail/conspire/2007-January/002743.html



> Any ideas on how to track down this problem?

Above is an example of going with a specific hypothesis and pursuing it
exhaustively (a plausible hypothesis because 'random' freeze-up are
often caused by bad RAM -- probably the most common cause, with the next
most common being a dodgy/weak PSU, discussed below), but hardly the
only way.  A different diagnostic approach is to divide all possible
causes into mutually exclusive groups and see if any of them can be
eliminated logically.  For example, you could start with 'It's almost
certainly caused by either a hardware problem or a software one, but not
both.'  Can you think of a way two eliminate one of those all at one go?

I can.  It's called a live-CD Linux distribution.  Boot up your
favourite such distribution _rather than_ your loaded software system.
Personally, I would use the current console-only ('no X') release of
Siduction.
http://ftp.spline.de/mirrors/siduction/iso/patience/nox/amd64_2018-03-07_23-56/siduction-18.2.0-patience-nox-amd64-201803072356.iso

Run whatever suits your fancy on the live-CD non-installed runtime
system.  (You could do worse than doing that parallel kernel compile in
a loop, again.)  Run it 2-3 times as long as the median time to freezeup 
with your installed system.  If it remains running, then to a high
degree of confidence you've isoated the problem to installed software.
If not, hardware -- which is what I suspect you'll find.  

That wouldn't tell you _which_ hardware (nor, if the system remains
running, which software), but you'd already be ahead of the game,
knowing more than you do now, because half of all possible causes can
now for all practical purposes be ruled out.

If it's hardware and screwing around with SDRAM sticks and doing the
kernel-compilation stress-testing doesn't find a culprit, then that's
too bad but (again) you're zeroing in on the hardare at fault by ruling
out hardware that isn't.  As I mention in the three links above (which, 
please do read), don't forget that worn or tarnished or defective
sockets leading to intermittent connections can cause freezeups.  I mean
not only RAM sockets but all the other ones, and every other example of
an electical connection.  Cleaning connectors can't hurt (being really
careful about static).  Using canned air (or whatever) to blow out dust
can't hurt.  Unplugging and replugging connectors can't hurt.

Are there any non-essential hardware devices in the box, e.g., in
expansion slots?  Not so common anymore with servers, but I thought I'd
ask.  Reason I ask is:  Sometimes either a fault with that expansion
card or simply the additional strain on the PSU can tip the system over
into borderline stability.  If you have that non-essential-component
situation, it can be enlightening to power down, remove the thing, then
do a long test run without it.

Of the essential components, e.g., the HD(s) or SSD(s), how about you
temporarily removing and setting aside those for diagnostic purposes 
only, and booting and doing a test run from live-CD media?

Do you have a second, roughly similar system where you could temporarily
move some of the suspect system's components in order to see whether the
freeze-ups move with the parts in question?  Could be worth your time.

Last, there's those damned PSUs.  Turns out, there are lots of really
bad ones out there, and apparently always have been.  Personally, I
think it's worth the money to keep a known-good, really robust one
sitting around on the shelf.  Substitute it in for testing, do a long
test run.  Problem vanished?  That'd tell you the cause.