Re: CLOSE_WAIT issues

On Jul 25, 2006, at 10:03 AM, John Pollard wrote:

> We have had similar problems in the past, but this seems to be  
> happening more often now:
>
> Web site performance becomes erratic (some instances not responding)
> Viewing JavaMonitor one or two instances show 0 in the Transactions  
> column, even though you know that is not the case (there will have  
> been hundreds or thousands)
>
> Our WO app instances are all within the port range of 2000 - 2050:
>
> Running:
> /usr/sbin/lsof -i tcp:2001-2050 -P
> gives some ok LISTEN lines + some problem lines like this:
>
> java     553 root   20u  IPv6 0x04bbcd68      0t0  TCP plug:2025- 
> >plug:64477 (CLOSE_WAIT)
> java     553 root   21u  IPv6 0x0646b2c8      0t0  TCP plug:2025- 
> >plug:64483 (CLOSE_WAIT)
>
> When I send a kill -9 to the problem process, 533 these are of  
> course cleared.
>
> Casting my lsof net a bit wider:
> /usr/sbin/lsof -i tcp:2001-2050 -P
> gives about 10 lines like this:
>
> java     281  root    9u  IPv6 0x04bba470      0t0  TCP [:: 
> 127.0.0.1]:49228->[::127.0.0.1]:3306 (CLOSE_WAIT)
> java     569  root    9u  IPv6 0x0616dd70      0t0  TCP [:: 
> 127.0.0.1]:49247->[::127.0.0.1]:3306 (CLOSE_WAIT)
>
> These all seem to be wotaskd processes. Are there meant to be 10  
> wotaskd processes running?

You definitely should only have 1 wotaskd running.  However, its  
unlikely that these are both wotaskd: on startup, wotaskd attaches to  
port 1085 and if there's a previous wotaskd attached, it fails and  
quits.


> I tried sending a kill -QUIT signal to one of my hung WO instances  
> to force a stack trace, but nothing came out in the log file. I  
> have now read the posting about editing SpawnOfWotaskd.sh to get  
> the log output, so will do this for next time.

This is helpful if your apps are deadlocked, but the infamous  
CLOSE_WAIT problem can occur without a deadlock.


> As an aside, we try to use lsof every night in a script to detect  
> these problems and reboot, but lsof sometimes returns absolutely  
> nothing or "unable to read process table" or something close to  
> that. I have read this is a bug with 10.4.6. Not sure if 10.4.7  
> helps. Perhaps related to also having > 2G of RAM.

I've had similar problems in the distant past (you can probably find  
my posts on the related MacOSX Admin list), but the only solution I  
found was to re-install.


> These CLOSE_WAIT problems seem to appear with a release of WO (I  
> forget which) and never went away, though I am still using the WO  
> version one back from the latest (where WO is not bundled in XCode).
>
> Is the root of this problem likely to be a deadlock in our  
> instances? I will report back if I can get some stack trace info  
> next time.

Unlikely a deadlock, especially if you're still using WO 5.1 or 5.2.1  
or (maybe?) 5.2.2.  These versions all have known problems with  
CLOSE_WAIT states.  Rebooting will work, but so will killing wotaskd  
and your apps, which should be faster to recover from.  Of course,  
your best bet is to upgrade to the lastest 5.2 or better yet, 5.3.


sacha


-- 
Sacha Michel Mallais             Senior Developer / President
Global Village Consulting Inc.   http://www.global-village.net/
PGP Key ID: 7D757B65             AIM: smallais
ObAd: read "Practical WebObjects" <fnord>
http://www.global-village.net/products/practical_webobjects