Runaway Java Processes

Ken Schweigert <[email protected]> Thu, 27 Jul 2006 11:33:39 -0400
Newsgroups gmane.comp.web.webobjects.admin
Message-ID <[email protected]>
We are having a problem with one of our WO applications.  I'm hoping  
someone will have some guidance as to what to do.  I have a lot of  
symptoms, but no real idea where to go next.

The machine is running on MacOSX-10.3.5 with WO-5.2.  We run 5  
instances of it and about 3 or 4 weeks the application becomes  
unresponsive.  We also host a few other applications that never seem  
to have this problem.  When is 'ssh' in to the machine and do 'top' I  
see:

--------------------------
Processes:  58 total, 4 running, 54 sleeping... 1355  
threads           11:45:42
Load Avg:  0.46, 0.61, 0.43     CPU usage:  11.2% user, 10.8% sys,  
78.1% idle
SharedLibs: num =   89, resident = 7.17M code,  860K data, 2.69M  
LinkEdit
MemRegions: num = 14137, resident = 1.24G + 3.60M private, 24.1M shared
PhysMem:   148M wired,  420M active,  900M inactive, 1.43G used,  
67.1M free
VM: 9.15G + 57.3M   416495(0) pageins, 942650(0) pageouts

   PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD   
RSIZE  VSIZE
27600 java         0.0% 67:18.84  29   558   285  80.8M  15.0M   
81.4M   307M
13570 top         19.2%  0:08.34   1    17    26   592K   352K    
968K  27.1M
13569 bash         0.0%  0:00.03   1    12    15   180K   796K    
792K  18.2M
13568 sshd         0.0%  0:00.01   1     9    41   112K  1.30M    
448K  30.0M
13566 sshd         0.0%  0:00.21   1    15    41   116K  1.30M   
1.45M  30.0M
13441 java         0.0% 28:00.67  30   510   248  89.7M  11.0M   
98.7M   317M
13439 java        18.4% 20:07.25  30   535   257  90.7M+ 11.0M  99.8M 
+  317M
13363 java         0.8% 14:48.68  30   531   248  93.0M  11.0M    
102M   317M
13339 java         0.0% 32:54.35  30   522   251  91.0M  11.0M    
100M   317M
13335 java         0.0% 30:38.76  30   546   262   101M  11.0M    
110M   325M
13004 httpd        0.0%  0:00.78   1    10   103   232K  4.41M   
2.20M  36.6M
8771 httpd        0.0%  0:00.76   1    10   106   356K  4.41M  2.27M   
36.7M
6043 httpd        0.0%  0:00.63   1    10   106   352K  4.41M  2.28M   
36.7M
2941 java         0.8% 99:44.73  >>   >>>   983   199M  10.3M    
110M   942M
2906 java         0.0% 53:56.68  >>   >>>  1080  13.5M  10.3M   
6.41M   576M
2871 java         0.0% 49:29.13  >>   >>>  1060  26.1M  10.3M   
15.9M   543M
2836 java         0.0%  2:10:53  >>   >>>  1213   238M  10.3M   116M   
1.12G
2781 java         0.0% 47:16.36  78   836   482  64.2M  10.3M   
58.8M   414M
   477 java         0.0% 82:49.47  28   524   280  50.7M  15.8M   
48.8M   307M
--------------------------

While reading the 'top' documentation I see that when the '#TH'  
column has a '>>' that it has more than 99 threads and when '#PRTS'  
has a '>>>' that there are more than 999 ports.  I find this a little  
excessive seeing that none of the other applications use this many  
resources.

I can also verify that each of the java processes with a '>>>' is a  
process of the same application.  I used 'lsof -i -n -P | grep java |  
less' and can see that associated tcp port with the one set in the  
application.  Also scattered throughout the runaway process's output  
I see quite a few of these lines:

COMMAND     PID   USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
<snip>
java      19994   root  181u  IPv6                 0t0  TCP can't  
read in6pcb at 0x00000000
java      19994   root  182u  IPv6                 0t0  TCP can't  
read in6pcb at 0x00000000

Another thing that also concerns me is the 'VSIZE' column having one  
instances with 1.12GB and another with 942MB.

So, as you can see, there are lot of symptoms and I'm not sure if  
they are all pointing to the same problem.  I should also mention  
that our WO developer has since left so I'm in the position where I  
need to get this under control.

Thank you for any help you can offer.
--
Ken Schweigert, Network Administrator
Byte Productions, LLC