Re: System Panic Makes My Life Easier

Quoting Ivan Sergio Borgonovo ([email protected]):

> That's like to ask an elephant to drive your car and then check if the 
> air conditioning worked. You ain't even testing the suspensions.

I actually need to walk back and apologise for part of what I said,
because without entirely meaning to, I claimed that iterative kernel
compiles is a _general_ tester for RAM.  As you point out, it's
probably not.  And memtest is.  But what I did _is_ a truly excellent
tool for (at least) many specific situations to diagnose a possible RAM
problem where memtest cannot.

It's been so many years (ten) since the situation arose when I wrote
those posts that I forgot the specifics, but I've refreshed my memory
and can clarify:

I had set up in my dining room a spare VA Linux Systems model 2230 2U
rackmount unit and was preparing to migrate my Internet server to it.
But something kept worrying me:  occasional spontaneous reboots -- and
also at one point a 'NMI: Dazed and confused but struggling to continue' 
kernel message that correlates strongly with either bad RAM or a bad RAM
socket.  This was four ECC sticks, _and_ ECC was enabled in the BIOS,
_and_ all four sticks had passed at least a day of memtest86.  

Yet, it turned out half the RAM _was_ the cause of the reboots, and
memtest did not find it -- as I shall explain.

I didn't believe it could be a software problem.  (Perhaps I eliminated
the likelihood of that by running a live CD and observing that it still
occasionally had spontaneous reboots, but I no longer remember.)  So, it
had to be hardware.  Nothing in the logs in any way suggested any
problem with mass storage, so by process of elimination it pretty much
had to be motherboard/CPU or bad RAM.  I really didn't want to buy
replacement parts (what can I say?  I'm cheap), but I _especially_
didn't want to replace part of my system and then find I also had to
replace another part.  In other words, I _greatly_ wanted to isolate
which component.

Thus _if_ I could make the system completely cease to display the
symptom _under heavy load of the RAM, CPU, and motherboard_ with a
subset of all its parts, then all non-removed parts became known-good.  
Add back the removed parts one at a time.  The moment the symptom
returned, you know where the problem lay.

So, (skipping over some fumbling around) I stripped out all but one
stick of RAM at a time.  With each stick as the sole stick, I put the
machine under heavy stress with parallel (make -j N with N cranked up
suitably) kernel compiles heavy enough to exercise all RAM and sent the
system slightly into swap.  Gradually, this pointed suspicion at not one
but two SDRAM sticks, because the problem persisted when they were
present and went away when they were gone.  The main point of the
iterative and parallelised kernel compiles was to reduce the test cycle
time (time to spontaneous reboot if it was going to happen) to as short
a time as possible -- minutes or hours rather than days.  I had a hunch
that the reboots were happening only when instructions were being
written to and read from a rarely-used stretch of bad RAM, so my theory
of testing was to make it more likely for that RAM to be used sooner and
more often by making _all_ RAM be heavily used.

The two good sticks of RAM could be in any RAM socket and the machine
never spontaneously rebooted or got a 'NMI: Dazed and confused but
struggling to continue' message.  The two bad sticks of RAM could be
any RAM sockets and the system _unfailingly and quickly_ reached
spontaneous reboot.  So, by the end of testing, I had extremely high
confidence that two sticks of RAM were bad, and had absolved the two
other sticks and motherboard (and its RAM sockets) from suspicion.  

I terminated the bad sticks with extreme prejudice (ah, threw them away)
and bought good ones from S.A. Technologies in Santa Clara -- who I'm
pleased to see are still around:  http://www.satech.com/

Anyway:  Parallel kernel compiles and a bit of logic found the bad.
memtest couldn't.  In any similar situation, therefore, I know which
toolset I'll try first:  parallel kernel compiles and thinking a bit.


> CTCS and its successor most probably were better engineered than
> something that does some computation on CPU and send it to the disk
> but... supposing your computer hang during a kernel compilation, which
> part of your PC failed?  And what if sometimes the kernel get compiled
> but in a wrong way? Are you going to compare the images?

You didn't read the story at my links, did you?  Well, I've now recapped
it above, so I hope my rationale and thinking are clearer.



> Most of the time is not economical even to find it out.

Quite so.  Always worthwhile considering what your time's worth -- not
to mention the troubling possibility that you won't end up finding out
what's wrong even after a lot of trouble.

But I would have been really irritated at myself if I'd not bothered to
isolate the cause of _that_ problem and had just thrown away the system 
-- because I'm stubborn and because I'm cheap.  And I sort of liked the
challenge. 

And I also liked the fact that a bit of sodding around and _thinking_
gave my definitive results in a situation where the recommended tool for
the specific symptoms (spontaneous reboots and ''NMI: Dazed and confused
but struggling to continue' on a just-installed, quiescent Debian
server that also replicated IIRC with a live CD distro), memtest, hadn't
found any problem.

> Diagnosing hardware is something that is better to leave to the
> vendor.

Maybe, but I have had really good luck doing in _many_ problem scenarios
-- and my main tool has always just been logical thinking, really.
Maybe I'm just better at it than you?  ;->

(Also, being cheap, I almost never have an in-force warranty to fall
back on.  If I did, I'd certainly rejoice and use it -- assuming I knew
which thing to RMA.)