Re: System Panic Makes My Life Easier

Well, I fear that I've let this thread get away from me,
but anyway, here's the response I've been planning to write:

> > But, I've been wondering about how one would narrow down one's
> > suspicions about flaky software, and I thought I would ask how one
> > would go about it, even though it's just academic now.  Look through
> > system logs?  Play with dtrace?
>
> I kind of like the problem scenario you've just posed, because one could
> spend a lifetime on it.  ;->

And all to save a box that probably should be replaced
anyway.  Sanity is not a requirement for computer use.

But anyway, just to review/clarify: yes, the main symptom
was a total lock-up requiring a hard reboot, roughly once
or twice a day.  It seemed to me it was more likely to
happen if there was more stuff running (e.g. if I was
logged in with firefox running with many tabs open).  When
it started throwing system panics that was the clue-by-four
that I was almost certainly looking at decaying hardware.
Just by coincidence this all started happening right after
I did one of my major software upgrades, so I was feeling
paranoid about the software...

(And to answer someone else's question: yes, it's now
throwing panics before I get through grub, though at
least once it let me through the login screen before panicing.)

> Well, actually, let's get serious.  First thing would be to attempt to
> bifurcate the problem space:  Test to determine - might it be software,
> or can that possibility be eliminated?
>
> Get yourself your favourite live-CD distro.

Yes, that sounds like an excellent idea to try to narrow
down whether it's the hardware or the software... I
could've left it running over night off of a live disk and
if it was still getting wedged by morning, that'd be a good
sign it was hardware.  Unless of course, the software issue
was shared with the live disk and my hard drive
installation...

You see, the kind of paranoid scenarios I was nursing was
things like "Uh oh, what if there's some subtle
incompatibility between systemd and older AMD chips? A lot
of open source projects are still pretty weak on suites of
regression tests...".  If I'd grabbed a recent Debian Live
disk it might not have been different enough from the
new Debian installation I was worried about.

But it'd be easy enough to track down a live install that
*is* pretty different, so yeah, a good, simple idea to get
more data...  easier than, say, doing a complete reinstall
rebuidling the box around, say, gentoo, which is the kind
of thing I was contemplating

It occurred to me recently that I should've tried booting to
a lower runlevel, and leaving it parked without network or
X11 running.  (Yes: this is a desktop workstation I'm
talking about, so yes, it's normally running X11 in some
form.  Even if I'm logged out there's still gdm (unless
it's kdm or something...  I think of these things as the
"dungeon master" and pay as little attention to them as I
can. I've played around with ripping it out and going back
to the old "startx" approach, but doing away with the
*dm these days turns out to be difficult).

> With a _little_ effort, the VA Linux Systems Cerberus Test
> Control System (va-ctcs, or CTCS, or Cerberus) can be made to run from a
> system running on a RAMdisk.  CTCS normally expects the ability to write
> to local disk, so that's pretty much the only gotcha you have to work
> around.  CTCS _seriously_ torture-tests any hardware set, doing a great
> many simultaneous tasks including iterative kernel compiles.  Stuff at:
> 'Cerberus FAQ' at http://linuxmafia.com/kb/Hardware/ .

Sounds good.

> After a few days, then, to quote an old boss's expression, 'What do we
> know?'  We know either that the system fell over, in which case it
> definitely wasn't your software and is pretty definitively your
> hardware, or that the system didn't fall over, in which case it's almost
> certainly your software.

> I am curious about the exact symptom.  You say in different parts of the
> narrative 'hard lock-up', 'totally lock-up', and 'system panics'.  Not a
> complaint, but do you have any more-specific indicators or description
> you could give?

Yes, I could've been more precise about it, but my
experience is that if you provide loads of detail it puts
people to sleep and they end up ignoring you.  It's a better
strategy to go light on detail and just let 'em complain
later that you don't know how to write a bug report.

Anyway, yes, the early symptom was a completely frozen
system, usually, but not always, occurring sometime
overnight.  The fact that it was frozen complicated looking
for more information.  Like, what did it say in
/var/log/messages? I have no idea.

(I was toying with writing my own monitoring tools...  it'd
be easy enough for me to, say, run top remotely and capture
the information on another box so I could look for
things like memory leaks right before it hung.)

> That could be any of several system-not-responding
> scenarios.  Here are some tips to narrow that down:

> Imagine a system that normally runs X11 (which based on mention of
> Firefox seems likely the case, here).

Correct.

> Let's say you are doing that and you either do or do not
> normally run a screensaver / X11 locking application.

I don't bother with them on a home-based system.  (If it
were in a shared office, that'd be different.)

> Is there no longer even the ability to move the mouse
> pointer? Or does the mouse pointer move around but you
> cannot get it to activate menus and do things to screen
> objects?

That's an excellent question.  My memory is that I've seen
both behaviors, but the live-mouse-pointer-that-isn't-useful
was more common.

> Does Ctrl-Alt-F1 no longer transport you to the text
> console on console #1?

No, it wouldn't  And trying to ssh in from another box never worked.

> Knowing the answer to those questions
> helps determine how much of your system is non-responding.

Right.  And normally I brag about the detail of my notes,
but it looks like they're pretty sketchy this time.
(A boiled-frog problem, I think... the trouble was getting
steadily worse, and I was bored with it before it was clear
I needed to pay attention...).

> Short of that, looking in logfiles is always a really good place to
> start looking for trouble symptoms, as proceses in trouble have a strong
> tendency to mutter in logfiles.

Yes, I did look through them, but nothing leapt out at me...
and I think that's because it was a hardware issue, and it
was just coincidence that made me think it was the recent
software upgrade that was causing the problem.

> > In the old days, my first guess actually would've been that linux
> > itself is rock solid, but I'm afraid linux has seemed increasingly
> > flaky of late.  I've seen this hard lock-up symptom on a number of
> > thinkpads, particularly when running a media player like vlc or totem.
>
> Might be bad RAM, you know.

Okay, but one would hope that you wouldn't see the problem
on multiple machines... my impression is that this is a
near universal problem at this point.  If you use linux as
a video player, you'll eventually see it freeze-up
(usually playing a small snippet of sound over-and-over
again in a loop).


On Thu, Jul 28, 2016 at 1:42 AM, Rick Moen <rick-IyCrq+X4Fdq2oZ/[email protected]> wrote:
> Quoting Joseph Brenner ([email protected]):
>
>> I've been puzzling over a sick linux box for a little while lately.
>>
>> It's a dual-Opteron box (over ten years old now? wow...) that I've been
>> upgrading off-and-on (bigger disks, a new video card...),
>> but after a recent round of software upgrades it had been
>> acting incredibly flaky, with uptimes of only a few days.  It would
>> totally lock-up and require a hard reboot... couldn't even ssh into it.
>>
>> I was trying to get an idea of what software change could've caused
>> this problem-- the list of possibilities was long-- but of late the
>> problem has gotten far worse, and it throws system panics and won't
>> boot at all, so it's almost certainly a hardware problem. The cpu fan
>> has been in bad shape for some time... it's got some cooling now, but
>> I can easily imagine it's lifespan was shortened by overheating in the
>> past.
>>
>> But, I've been wondering about how one would narrow down one's
>> suspicions about flaky software, and I thought I would ask how one
>> would go about it, even though it's just academic now.  Look through
>> system logs?  Play with dtrace?
>
> I kind of like the problem scenario you've just posed, because one could
> spend a lifetime on it.  ;->
>
> Well, actually, let's get serious.  First thing would be to attempt to
> bifurcate the problem space:  Test to determine - might it be software,
> or can that possibility be eliminated?
>
>
> Get yourself your favourite live-CD distro.  Put it onto your choice of
> CD/DVD/Blu-Ray or a USB thumb drive.  Boot it.  I personally would use
> one of the (several) DE flavours of Siduction for this purpose.  I'd
> probably go for the Fluxbox flavour, but you can indulge what you like:
> http://distrowatch.com/table.php?distribution=siduction
>
> Booting that, you are now running a full-fledged Linux distribution
> entirely from the boot media and a big RAMdisk.  You are -not- using any
> of the code on the installed system.  Now... let's see...  You said you
> were getting lockups and/or 'system panics' (kernel panics?) every few
> days.  So, _maybe_ just having it sitting there running is enough.  Or
> maybe not.  Maybe you need to start a few services, load up some RAM.
> If you want to torture-test the hardware while running the live-CD load,
> you have to get a little creative and fire up the software brass band.
>
> With a _little_ effort, the VA Linux Systems Cerberus Test
> Control System (va-ctcs, or CTCS, or Cerberus) can be made to run from a
> system running on a RAMdisk.  CTCS normally expects the ability to write
> to local disk, so that's pretty much the only gotcha you have to work
> around.  CTCS _seriously_ torture-tests any hardware set, doing a great
> many simultaneous tasks including iterative kernel compiles.  Stuff at:
>
> 'Cerberus FAQ' at http://linuxmafia.com/kb/Hardware/ .
> Link is to the same SourceForge repo where CTCS is semi-maintained.
> (Looks like the last code check-in was 11 years ago, but one person's
> 'unmaintained' is sometimes another person's 'feature-complete and
> working'.)
>
> After a few days, then, to quote an old boss's expression, 'What do we
> know?'  We know either that the system fell over, in which case it
> definitely wasn't your software and is pretty definitively your
> hardware, or that the system didn't fall over, in which case it's almost
> certainly your software.
>
>
> I am curious about the exact symptom.  You say in different parts of the
> narrative 'hard lock-up', 'totally lock-up', and 'system panics'.  Not a
> complaint, but do you have any more-specific indicators or description
> you could give?  That could be any of several system-not-responding
> scenarios.  Here are some tips to narrow that down:
>
>
> Imagine a system that normally runs X11 (which based on mention of
> Firefox seems likely the case, here).  Let's say you are doing
> that and you either do or do not normally run a screensaver / X11
> locking application.
>
> If the latter, you would probably want to disable your screensaver /
> screenlock-thingie so as to be able to gather data better.  You're
> running that for several days and then... what specifically?  Is there
> no longer even the ability to move the mouse pointer?  Or does the mouse
> pointer move around but you cannot get it to activate menus and do
> things to screen objects?  Does Ctrl-Alt-F1 no longer transport you to
> the text console on console #1?  Knowing the answer to those questions
> helps determine how much of your system is non-responding.
>
> Or imagine a server-type system or some other system that normally does
> _not_ run X11.  You connect a monitor to it (so you have a functional
> local console)  You login and leave yourself logged in.  Also, you do
> this at a bash prompt:
>
>    setterm -blank 0
>
> This disables the console screen blanker so that you can see what most
> recently happened on the console (if anything) if/when the system
> freezes, or whatever the heck it does.  Again, you wait a few days.
> After whatever-it-is occurs, you look and see if there are clues.  You
> also of course try to type some new shell commands; see if anything's
> alive.
>
> Short of that, looking in logfiles is always a really good place to
> start looking for trouble symptoms, as proceses in trouble have a strong
> tendency to mutter in logfiles.
>
>
>> In the old days, my first guess actually would've been that linux
>> itself is rock solid, but I'm afraid linux has seemed increasingly
>> flaky of late.  I've seen this hard lock-up symptom on a number of
>> thinkpads, particularly when running a media player like vlc or totem.
>
> Might be bad RAM, you know.  About that:
>
> http://linuxmafia.com/pipermail/conspire/2006-December/002662.html
> http://linuxmafia.com/pipermail/conspire/2006-December/002668.html
> http://linuxmafia.com/pipermail/conspire/2007-January/002743.html
> That's my tutorial of how to _thoroughly_ test RAM.
>
>
> 'Hope that helps!  Good meaty problem, thanks!
>
> --
> Cheers,                                "Why struggle to open a door between us,
> Rick Moen                              when the whole wall is an illusion?"
> rick-IyCrq+X4Fdq2oZ/[email protected]                                                     -- Rumi
> McQ! (4x80)
>
> _______________________________________________
> svlug mailing list
> [email protected]
> http://lists.svlug.org/lists/listinfo/svlug
>
> --
> You received this message because you are subscribed to the Google Groups "Vanity Alias - Joe Brenner" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to doom+unsubscribe-vb6xrSscvWjRrdkEUGVx5d3JkUJRut2i@public.gmane.org
>