Re: Runaway Java Processes

On Jul 27, 2006, at 8:33 AM, Ken Schweigert wrote:

> We are having a problem with one of our WO applications.  I'm  
> hoping someone will have some guidance as to what to do.  I have a  
> lot of symptoms, but no real idea where to go next.
>
> The machine is running on MacOSX-10.3.5 with WO-5.2.  We run 5  
> instances of it and about 3 or 4 weeks the application becomes  
> unresponsive.  We also host a few other applications that never  
> seem to have this problem.  When is 'ssh' in to the machine and do  
> 'top' I see:

I was just talking about this in another thread: it is normal to  
regularly schedule your apps to restart or a regular basis.  You  
might start with that.


> While reading the 'top' documentation I see that when the '#TH'  
> column has a '>>' that it has more than 99 threads and when '#PRTS'  
> has a '>>>' that there are more than 999 ports.  I find this a  
> little excessive seeing that none of the other applications use  
> this many resources.

It is excessive.  You should only see that under two conditions:
1) your app is being hit that many times at the same time
2) worker threads are getting locked up so that WO has to spawn more  
of them

Assuming its #2, one way to determine where the deadlock is occurring  
is to send the process a QUIT signal, as in "kill -QUIT <pid>".  This  
will tell java to spew out a stack trace.  Unfortunately, this stack  
trace is sent to /dev/null by default, so you'll have to edit the  
startup script SpawnOfWotaskd as described under "Where's my stderr"  
on this page: http://en.wikibooks.org/wiki/Programming:WebObjects/ 
Web_Applications/Deployment/Common_Pitfalls_and_Troubleshooting.


> I can also verify that each of the java processes with a '>>>' is a  
> process of the same application.  I used 'lsof -i -n -P | grep java  
> | less' and can see that associated tcp port with the one set in  
> the application.  Also scattered throughout the runaway process's  
> output I see quite a few of these lines:
>
> COMMAND     PID   USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
> <snip>
> java      19994   root  181u  IPv6                 0t0  TCP can't  
> read in6pcb at 0x00000000
> java      19994   root  182u  IPv6                 0t0  TCP can't  
> read in6pcb at 0x00000000
>
> Another thing that also concerns me is the 'VSIZE' column having  
> one instances with 1.12GB and another with 942MB.

Probably related to the number of worker threads.


sacha


-- 
Sacha Michel Mallais             Senior Developer / President
Global Village Consulting Inc.   http://www.global-village.net/
PGP Key ID: 7D757B65             AIM: smallais
"I resist change even as I call for it."  -- Mason Cooley