Re: System Panic Makes My Life Easier

On 07/26/2016 10:56 AM, Joseph Brenner wrote:
> I've been puzzling over a sick linux box for a little while lately.
>
> It's a dual-Opteron box (over ten years old now? wow...) that I've been
> upgrading off-and-on (bigger disks, a new video card...),
> but after a recent round of software upgrades it had been
> acting incredibly flaky, with uptimes of only a few days.  It would
> totally lock-up and require a hard reboot... couldn't even ssh into it.
>
> I was trying to get an idea of what software change could've caused
> this problem-- the list of possibilities was long-- but of late the
> problem has gotten far worse, and it throws system panics and won't
> boot at all, so it's almost certainly a hardware problem. The cpu fan
> has been in bad shape for some time... it's got some cooling now, but
> I can easily imagine it's lifespan was shortened by overheating in the
> past.

Personally, I rely heavily on logging serial consoles.   The panic, if 
it isn't obvious to me, which it often is, is usually something I can 
take to srn or someone else who understands more than I do, and they can 
then often give me an exact cause.   Often, searching for choice bits of 
the backtrace takes me to kernel bugs.

I think being able to send the actual backtrace of the actual problem to 
someone who knows what they are doing is worth many, many hours of 
trying to replicate the problem in burn-in, personally, and a logging 
serial console makes that pretty easy.

Back when prgmr had just a couple of servers, I'd set them up in a 
'buddy system' so that each computer had a USB to serial that null 
modem'd to another server's console port.   At first, I logged with 
screen in logging mode.