Re: System Panic Makes My Life Easier

On Thu, 28 Jul 2016 01:42:33 -0700
Rick Moen <rick-IyCrq+X4Fdq2oZ/[email protected]> wrote:

> Quoting Joseph Brenner ([email protected]):
> 
> > I've been puzzling over a sick linux box for a little while lately.
> > 
> > It's a dual-Opteron box (over ten years old now? wow...) that I've
> > been upgrading off-and-on (bigger disks, a new video card...),
> > but after a recent round of software upgrades it had been
> > acting incredibly flaky, with uptimes of only a few days.  It would
> > totally lock-up and require a hard reboot... couldn't even ssh into
> > it.
> > 
> > I was trying to get an idea of what software change could've caused
> > this problem-- the list of possibilities was long-- but of late the
> > problem has gotten far worse, and it throws system panics and won't
> > boot at all, so it's almost certainly a hardware problem. The cpu
> > fan has been in bad shape for some time... it's got some cooling
> > now, but I can easily imagine it's lifespan was shortened by
> > overheating in the past.
> > 
> > But, I've been wondering about how one would narrow down one's
> > suspicions about flaky software, and I thought I would ask how one
> > would go about it, even though it's just academic now.  Look through
> > system logs?  Play with dtrace?  
> 
> I kind of like the problem scenario you've just posed, because one
> could spend a lifetime on it.  ;->
> 
> Well, actually, let's get serious.  First thing would be to attempt to
> bifurcate the problem space:  Test to determine - might it be
> software, or can that possibility be eliminated?
> 
> 
> Get yourself your favourite live-CD distro.  Put it onto your choice
> of CD/DVD/Blu-Ray or a USB thumb drive.  Boot it.  I personally would
> use one of the (several) DE flavours of Siduction for this purpose.
> I'd probably go for the Fluxbox flavour, but you can indulge what you
> like: http://distrowatch.com/table.php?distribution=siduction
> 
> Booting that, you are now running a full-fledged Linux distribution
> entirely from the boot media and a big RAMdisk.  You are -not- using
> any of the code on the installed system.  Now... let's see...  You
> said you were getting lockups and/or 'system panics' (kernel panics?)
> every few days.  So, _maybe_ just having it sitting there running is
> enough.  Or maybe not.  Maybe you need to start a few services, load
> up some RAM. If you want to torture-test the hardware while running
> the live-CD load, you have to get a little creative and fire up the
> software brass band.  

My prime number generator hits the processor and RAM hard, and also
exercises the hard disk. It's pretty easy to compile and run it on a
live CD:

http://www.troubleshooters.com/codecorn/primenumbers/primenumbers.htm#_Implementation:_Scalability_Through_Paging

If you run my prime number generator, watch your processor temperature
like a hawk.

By the way, before doing all of this, I'd take a very close look at the
mobo's capacitors. If the little cans' tops are bulging, then whether
or not capacitors are causing your *current* problem, you need to
either replace them or toss the mobo, because you know that right now
they're in the process of moving toward your machine's intermittence,
even if they're not the current cause. Running an intermittent machine
with bulging caps is like running a car that can't go over 45MPH and
coughs and sputters on hills, and hasn't had the fuel filter replaced
in 5 years.


SteveT

Steve Litt 
July 2016 featured book: Troubleshooting Techniques
     of the Successful Technologist
http://www.troubleshooters.com/techniques