Re: debugging hang on headless server

Hi

So access to the server is required, if there are other servers in the netwo=
rk, then netconsole will send to the remote box over same syslog idea logs t=
hat get printed to the console

syslog for logs that go to the normal log files

and kdump will be able to generate a memory dump and some logs if there is a=
ny crash

with that you should be able to have ll the information outside this server w=
hen there is a hang

othertools that can be user, are sar/sysstat you can copy the files out for i=
nspection and get an idea about performance and what was doing what before a=
 hang

good luck
alvaro

> On 5 Jul 2017, at 03:54, Alexandr <[email protected]> wrote:
>=20
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>=20
> =D0=92 =D0=92=D1=82, 04/07/2017 =D0=B2 07:48 -0500, Billy Crook =D0=BF=D0=B8=
=D1=88=D0=B5=D1=82:
>> You can set up syslog to forward of to an external system.  That
>> helps but
>> sometimes it can miss things if the crash is severe enough.
>>=20
>> Serial console can capture much more if there's another server near
>> by that
>> you can connect and leave minicom logging.  Even better if you have a
>> nice
>> physical "serial console server" like an OpenGear.
>>=20
>> Most server motherboards these days contain some sort of management
>> chip
>> implementing ipmi and additional remote access features which often
>> include
>> remote KVM over IP.
>>=20
>> Even the most basic ipmi controllers will include serial over LAN
>> which
>> let's do you use serial console without a server nearby to connect a
>> physical serial cable.
>>=20
>> If the hang is caused by a hardware problem you may find indication
>> of it
>> in the ipmi system event list.  'ipmitool sel list'
>>=20
>> There's also a kernel module, netconsole, that forwards kernel
>> messages off
>> host in a syslog compatible format. It implements IP and UDP and its
>> own
>> code so it works independent of the system's TCP stack which is
>> important
>> during some crashes which might otherwise break networking before the
>> messages get transmitted.
>>=20
>> That's just a few ways.  There's probably others. Hope it helps
>>=20
>>> On Jul 4, 2017 07:32, "Alexandr" <[email protected]> wrote:
>>>=20
>>> Good day to all.
>>> please suggest methods of system hang debugging on headless server
>>> ?
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-
>>> admin" in
>>> the body of a message to [email protected]
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>=20
>=20
> thx for reply.
> it's self built server on regular desktop hardware, so have not ipmi.
> it's possible to have phisical access to it, so i guess serial console
> is my solution.
> i am also have few questions about serial console setup.
> as i understand simplest way to achieve it is default serial com port,
> i am also have read something about serial via usb (looks like it
> require some sort of special adaptor)
>=20
> also netconsole looks interesting too, i will setup it right now.
> -----BEGIN PGP SIGNATURE-----
>=20
> iQIzBAEBCgAdFiEEl/sA7WQg6czXWI/dsEApXVthX7wFAllcRvEACgkQsEApXVth
> X7w5DA//VnlRsWDxy0RifqMrmBtiL1/fxdl3Wz/PhH/Fdu+AVgM2XdaITYHfI0uU
> 7l6bYPPnMnXlmiqtJ0DkYEV+Xgq7Gkuw3JK8eppLHehgMfaIEmnXWPMoxVVZalmn
> u1HVGBdEGfRtiLisrNvZvGgeCwXdGIfMeZbnBFwFbG3vbxzHzaGd7mPTO9CvhPyj
> FArTHF+i3DtVpteXbww78JS0qEKLTh83cW1FcCqAWjZBvrnz35i7Egp4grajhhzQ
> ki8wFUqwwVWY+JVg6UcpGr5CH4b11b4qv3Vol/ideooiTHWPv1xKXMjT3+DIggzc
> gh0g6ZwK3IDcR2rtJ2lGD/UOi4jJNti8A6O/48sXIsgruePuDIPNJOIbBbRR5EDv
> cDlb6vVHA0QO6XilUcTKFPEjg9yG++JIyYHmGrGxJGo28KpF+GbStA1xgm98Mbqj
> AJP/mZIdLCoKuRGEGkPCvHAX4BfMm4zdPmBcj3XAiw9xGJCFuts0xowX9uSqnzT6
> ITlVA9VHWlm030TFYp0h2HYjuzpVrKNw06OQ0AqHy7E932DlPr3us2y7ugWGXUgE
> Sxewc2+QXR2RGEHXVWfbymGr6uKtqwmwuyl8q7QQ/oqYGFK99WblRMT+6+K0kJI6
> f/YzGyAqHRtN2gBZvRjX2BYVPdPKcGbNKqp27B9wOkz58NNRpF8=3D
> =3DPmkQ
> -----END PGP SIGNATURE-----
>=20
> --
> To unsubscribe from this list: send the line "unsubscribe linux-admin" in
> the body of a message to [email protected]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html