RE: Apache looping?

Stephen,

	I am the only admin of this particular box, and I am quite
certain nothing has changed. I'm seeing nothing in the system logs
indicating an error. A root-kit is possible, but this server is intranet
accessible only, so that would be a very remote possibility.

I forgot to mention that a couple things:

1) the parent httpd process is currently eating about 50% of one CPU
2) an strace on the httpd startup appears fine, until it hits the
"times()" call

I looked at the 'man 2 times' and see this:

RETURN VALUE
       The  function times returns the number of clock ticks that have
elapsed
       since an arbitrary point in the past.  For  Linux  this  point
is  the
       moment  the system was booted.  This return value may overflow
the pos-
       sible range of type clock_t.  On error, (clock_t) -1 is
returned,  and
       errno is set appropriately.

Since it is returning a large negative number, and my system has been up
255 days, I suspect this is the culprit. 

I'm planning to schedule a reboot of the system, but in the meantime any
other suggestions are welcome.

Thanks,

Kevin

-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Stephen John
Smoogen
Sent: Wednesday, July 12, 2006 11:05 AM
To: Discussion of Red Hat Enterprise Linux 3 (Taroon)
Subject: Re: Apache looping?

On 7/12/06, Collins, Kevin [MindWorks] <[email protected]> wrote:
>
>
>
> Hi folks,
>
>         I've discovered this morning that one of my web servers was
> unresponsive. I check with a 'ps -ef' and discovered I had only one
httpd
> process (the parent, running as root). So, I restarted Apache with
'service
> httpd restart' and see the same behavior - parent is started with no
> children. The access.log and error.log appear normal.
>

My first look would be to see what has changed in /etc and /var in the
last 8-10 days (or however long ago it was working.) A lock file that
can't be removed for some odd reason etc. It can be something silly
like Asst Admin C forgot to tell Admin A that he put out a new
nsswitch.conf and ldap which was for a test network, and httpd is
waiting to find some UID:GID entry in the ldap server.

If Nothing has changed.. I would first suspect hardware problems
(memory,disk) and then root-kit issues.

I would first do a check of the system logs for any memory issues
being reported or hardware crapping out. Then I would boot with a
livecd and run a memory checker and then a chkrootkit.

If those come up clear, I would go for an strace of httpd being
started up to see where it starts trying to time out. Look in top or
ps to see if the httpd process is in device wait... waiting for
something to become free (an NFS mount lets say).


> Nothing has changed on this server in months and I can confirm via the
> previous access logs that it was responsive a couple days ago.
>
> If I use 'strace' to attach to the running parent process I see this:
>

-- 
Stephen J Smoogen.
CSIRT/Linux System Administrator

--
Taroon-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/taroon-list



--
Taroon-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/taroon-list