CLOSE_WAIT issues
John Pollard <[email protected]> Tue, 25 Jul 2006 18:03:32 +0100
| Newsgroups | gmane.comp.web.webobjects.admin |
|---|---|
| Message-ID | <[email protected]> |
List, We have had similar problems in the past, but this seems to be happening more often now: Web site performance becomes erratic (some instances not responding) Viewing JavaMonitor one or two instances show 0 in the Transactions column, even though you know that is not the case (there will have been hundreds or thousands) Our WO app instances are all within the port range of 2000 - 2050: Running: /usr/sbin/lsof -i tcp:2001-2050 -P gives some ok LISTEN lines + some problem lines like this: java 553 root 20u IPv6 0x04bbcd68 0t0 TCP plug:2025- >plug:64477 (CLOSE_WAIT) java 553 root 21u IPv6 0x0646b2c8 0t0 TCP plug:2025- >plug:64483 (CLOSE_WAIT) When I send a kill -9 to the problem process, 533 these are of course cleared. Casting my lsof net a bit wider: /usr/sbin/lsof -i tcp:2001-2050 -P gives about 10 lines like this: java 281 root 9u IPv6 0x04bba470 0t0 TCP [::127.0.0.1]: 49228->[::127.0.0.1]:3306 (CLOSE_WAIT) java 569 root 9u IPv6 0x0616dd70 0t0 TCP [::127.0.0.1]: 49247->[::127.0.0.1]:3306 (CLOSE_WAIT) These all seem to be wotaskd processes. Are there meant to be 10 wotaskd processes running? I tried sending a kill -QUIT signal to one of my hung WO instances to force a stack trace, but nothing came out in the log file. I have now read the posting about editing SpawnOfWotaskd.sh to get the log output, so will do this for next time. As an aside, we try to use lsof every night in a script to detect these problems and reboot, but lsof sometimes returns absolutely nothing or "unable to read process table" or something close to that. I have read this is a bug with 10.4.6. Not sure if 10.4.7 helps. Perhaps related to also having > 2G of RAM. These CLOSE_WAIT problems seem to appear with a release of WO (I forget which) and never went away, though I am still using the WO version one back from the latest (where WO is not bundled in XCode). Is the root of this problem likely to be a deadlock in our instances? I will report back if I can get some stack trace info next time. Thanks John