Re: CLOSE_WAIT issues

Thanks Sacha,
I believe we are on WO5.2.4, though not sure how to confirm that 100% these 
days. We do need to upgrade to 5.3, so will try to bring that process 
forwards in case it helps.
You are right, I now think I was mistaken about the multiple wotaskd 
processes. These are actually our java instances talking to mysqld (at 
3306). Not sure if these being in a CLOSE_WAIT state is a problem or not? I 
suspect not.
In our nightly script we do call:
/sbin/SystemStarter restart 'WebObjects Services'
but the CLOSE_WAITS do seem to remain afterwards, even with a 5 min sleep 
after the wo restart, so then we reboot
John

At 18:20 25/07/2006, Sacha Michel Mallais wrote:

>On Jul 25, 2006, at 10:03 AM, John Pollard wrote:
>
>>We have had similar problems in the past, but this seems to be
>>happening more often now:
>>
>>Web site performance becomes erratic (some instances not responding)
>>Viewing JavaMonitor one or two instances show 0 in the Transactions
>>column, even though you know that is not the case (there will have
>>been hundreds or thousands)
>>
>>Our WO app instances are all within the port range of 2000 - 2050:
>>
>>Running:
>>/usr/sbin/lsof -i tcp:2001-2050 -P
>>gives some ok LISTEN lines + some problem lines like this:
>>
>>java     553 root   20u  IPv6 0x04bbcd68      0t0  TCP 
>>plug:2025- >plug:64477 (CLOSE_WAIT)
>>java     553 root   21u  IPv6 0x0646b2c8      0t0  TCP 
>>plug:2025- >plug:64483 (CLOSE_WAIT)
>>
>>When I send a kill -9 to the problem process, 533 these are of
>>course cleared.
>>
>>Casting my lsof net a bit wider:
>>/usr/sbin/lsof -i tcp:2001-2050 -P
>>gives about 10 lines like this:
>>
>>java     281  root    9u  IPv6 0x04bba470      0t0  TCP [:: 
>>127.0.0.1]:49228->[::127.0.0.1]:3306 (CLOSE_WAIT)
>>java     569  root    9u  IPv6 0x0616dd70      0t0  TCP [:: 
>>127.0.0.1]:49247->[::127.0.0.1]:3306 (CLOSE_WAIT)
>>
>>These all seem to be wotaskd processes. Are there meant to be 10
>>wotaskd processes running?
>
>You definitely should only have 1 wotaskd running.  However, its
>unlikely that these are both wotaskd: on startup, wotaskd attaches to
>port 1085 and if there's a previous wotaskd attached, it fails and
>quits.
>
>
>>I tried sending a kill -QUIT signal to one of my hung WO instances
>>to force a stack trace, but nothing came out in the log file. I
>>have now read the posting about editing SpawnOfWotaskd.sh to get
>>the log output, so will do this for next time.
>
>This is helpful if your apps are deadlocked, but the infamous
>CLOSE_WAIT problem can occur without a deadlock.
>
>
>>As an aside, we try to use lsof every night in a script to detect
>>these problems and reboot, but lsof sometimes returns absolutely
>>nothing or "unable to read process table" or something close to
>>that. I have read this is a bug with 10.4.6. Not sure if 10.4.7
>>helps. Perhaps related to also having > 2G of RAM.
>
>I've had similar problems in the distant past (you can probably find
>my posts on the related MacOSX Admin list), but the only solution I
>found was to re-install.
>
>
>>These CLOSE_WAIT problems seem to appear with a release of WO (I
>>forget which) and never went away, though I am still using the WO
>>version one back from the latest (where WO is not bundled in XCode).
>>
>>Is the root of this problem likely to be a deadlock in our
>>instances? I will report back if I can get some stack trace info
>>next time.
>
>Unlikely a deadlock, especially if you're still using WO 5.1 or 5.2.1
>or (maybe?) 5.2.2.  These versions all have known problems with
>CLOSE_WAIT states.  Rebooting will work, but so will killing wotaskd
>and your apps, which should be faster to recover from.  Of course,
>your best bet is to upgrade to the lastest 5.2 or better yet, 5.3.
>
>
>sacha