Re: Runaway Java Processes

On Jul 27, 2006, at 11:52 AM, Sacha Michel Mallais wrote:

> On Jul 27, 2006, at 8:33 AM, Ken Schweigert wrote:
>
>> We are having a problem with one of our WO applications.  I'm  
>> hoping someone will have some guidance as to what to do.  I have a  
>> lot of symptoms, but no real idea where to go next.
>>
>> The machine is running on MacOSX-10.3.5 with WO-5.2.  We run 5  
>> instances of it and about 3 or 4 weeks the application becomes  
>> unresponsive.  We also host a few other applications that never  
>> seem to have this problem.  When is 'ssh' in to the machine and do  
>> 'top' I see:
>
> I was just talking about this in another thread: it is normal to  
> regularly schedule your apps to restart or a regular basis.  You  
> might start with that.

I've tried scheduling the instances to restart, but usually get a  
case where something in the instances "hangs" and the instance stays  
in a state of "refuse new instances" and then all of them are like  
that and then the application is unreachable.  I would schedule each  
instance a couple of hours apart hoping that by the time next next  
instance restarts, the previous one would've closed down and restarted.

>
>
>> While reading the 'top' documentation I see that when the '#TH'  
>> column has a '>>' that it has more than 99 threads and when  
>> '#PRTS' has a '>>>' that there are more than 999 ports.  I find  
>> this a little excessive seeing that none of the other applications  
>> use this many resources.
>
> It is excessive.  You should only see that under two conditions:
> 1) your app is being hit that many times at the same time
> 2) worker threads are getting locked up so that WO has to spawn  
> more of them
>
> Assuming its #2, one way to determine where the deadlock is  
> occurring is to send the process a QUIT signal, as in "kill -QUIT  
> <pid>".  This will tell java to spew out a stack trace.   
> Unfortunately, this stack trace is sent to /dev/null by default, so  
> you'll have to edit the startup script SpawnOfWotaskd as described  
> under "Where's my stderr" on this page: http://en.wikibooks.org/ 
> wiki/Programming:WebObjects/Web_Applications/Deployment/ 
> Common_Pitfalls_and_Troubleshooting.

Great tip!  I just modified that startup script and am in the process  
of restarting the application's instances.  I'm sure I'll be able to  
'kill' and instance soon.

>
>
>> I can also verify that each of the java processes with a '>>>' is  
>> a process of the same application.  I used 'lsof -i -n -P | grep  
>> java | less' and can see that associated tcp port with the one set  
>> in the application.  Also scattered throughout the runaway  
>> process's output I see quite a few of these lines:
>>
>> COMMAND     PID   USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
>> <snip>
>> java      19994   root  181u  IPv6                 0t0  TCP can't  
>> read in6pcb at 0x00000000
>> java      19994   root  182u  IPv6                 0t0  TCP can't  
>> read in6pcb at 0x00000000
>>
>> Another thing that also concerns me is the 'VSIZE' column having  
>> one instances with 1.12GB and another with 942MB.
>
> Probably related to the number of worker threads.

Just killed an instance and noticed that the free memory shot way up.


--
Ken Schweigert, Network Administrator
Byte Productions, LLC