Re: System Panic Makes My Life Easier

Quoting Joseph Brenner ([email protected]):

> But anyway, just to review/clarify: yes, the main symptom
> was a total lock-up requiring a hard reboot, roughly once
> or twice a day.  It seemed to me it was more likely to
> happen if there was more stuff running (e.g. if I was
> logged in with firefox running with many tabs open).  When
> it started throwing system panics that was the clue-by-four
> that I was almost certainly looking at decaying hardware.

I concur.  In my experience, that's either bad RAM, or a flaky
CPU/motherboard, or some similar hardware glitch (weak PSU, something
like that).  Since you mentioned that this host had gone through an
overheating episode, that's automatically a strong suspicion, as
overheating damages electronics and often makes it unreliable even
before the usual early death that often ensues.

> Just by coincidence this all started happening right after
> I did one of my major software upgrades, so I was feeling
> paranoid about the software...

It's a natural suspicion, sure.

> You see, the kind of paranoid scenarios I was nursing was
> things like "Uh oh, what if there's some subtle
> incompatibility between systemd and older AMD chips? 

Eh, doubt that.  If that were the case, we'd be hearing a lot more 
reports.  Also, it'd be incredibly rare for there to be an
incompatibility between a particular class of x86 CPU and anything other
than the kernel and libc.  (It's possible to compile software with
support for certain CPUs' instruction extensions.  We probably don't
want to get into that.)

Right around the year 1999 or 2000, my wife Deirdre had a puzzling
problem with Linux on an AMD-based system of that day:  The 'man'
command segfaulted, as did 'ls' if you used certain options to that
command, but not otherwise.  This was practically a black swan event,
but someone on a Linux mailing list managed to identify the root cause:
The CPU's floating point circuitry was wonky, with the result that
/usr/bin/man's calculations that it uses for formatting broke, and the
command segfaulted.  I don't know what the exact function in 'ls' was
that had the same problem, but it was something on that approximate
order of things.

> It occurred to me recently that I should've tried booting to
> a lower runlevel, and leaving it parked without network or
> X11 running.

Sure.  Simple way to do that is to enter GRUB's command line editing and
put the word 'single' at the end of the kernel line.
http://www.cyberciti.biz/faq/grub-boot-into-single-user-mode/

> Anyway, yes, the early symptom was a completely frozen
> system, usually, but not always, occurring sometime
> overnight.  The fact that it was frozen complicated looking
> for more information.  Like, what did it say in
> /var/log/messages? I have no idea.

And when you looked later, nothing stood out.  Sounds _very_ like dodgy
hardware is the cause.  Because that's what you often get.

> That's an excellent question.  My memory is that I've seen
> both behaviors, but the live-mouse-pointer-that-isn't-useful
> was more common.

Yeah, honestly, on reflection, I'm not totally sure what distinguishing
between nothing-responds and mouse-pointer-moves-but-nothing-else buys
you.  I -think- the latter tends to absolve the CPU, but that's just a
hunch.

> Okay, but one would hope that you wouldn't see the problem
> on multiple machines... my impression is that this is a
> near universal problem at this point.

Well, that _would_ be depressing.  You say you saw it on multiple
ThinkPads.  Maybe they were all being driven into thermal overload?
That could happen with, say, using vlc for video.  Much though I love
vlc.

Dunno, insufficient data in the hands of yr. humble servant.

-- 
Cheers,             Grossman's Law:  "In time of crisis, people do not rise to
Rick Moen           the occasion.  They fall to the level of their training."
rick-IyCrq+X4Fdq2oZ/[email protected]          http://linuxmafia.com/~rick/lexicon.html#grossman
McQ! (4x80)