Re: MacOSX 10.3 Server deployement

Hm... that was a bit too fast. Turns out that your trick did not solve 
my problem after all. lsof still returns tons of CLOSE_WAIT entries... 
and my apps are still going offline...

Does anyone have any ideas what could cause all these "hanging" open 
files?

-Will

On Wednesday, January 7, 2004, at 08:44 AM, Will Scheidegger wrote:

> Mark
>
> You're the greatest! I've been pulling my hair over this for the last 
> two Months, searching the archives, asking the list, reading the docs 
> and could not figure out how to resolve this problem. WOTASKD really 
> _is_ broken. Our deployment is terribly unreliable. We now even began 
> to play around with two Solaris servers, hoping that this would lead 
> to a more stable deployment.
>
> So same here: Apple, are you listening? We're in the process of 
> kicking out MacOS X in favor of Solaris! What a shame, especially 
> since I would be very eager to get my hands on one of those new slick 
> G5 XServes. Oh well, Apple is busy changing my iLife, no time for 
> "toys" like WO.
>
> -Will
>
> On Tuesday, January 6, 2004, at 04:41 PM, Mark Fisher wrote:
>
>> Pierre,
>>
>> I have the same problems you describe but my hardware environment is 
>> a bit different.  I run a cluster of XServers:  One 
>> webserver/application server, One database server and two dedicated 
>> WebObjects application servers.
>>
>> Before the change to 10.3 I had one application distributed to three 
>> of the servers running over 20 instances on each server.  Now (with 
>> the upgrade to 10.3) I'm running 10 instances on each server.
>>
>> I have been able to trace the problem down to WOTASKD - over time it 
>> opens too many TCP ports to the running instances and finally gets 
>> stuck with a "Too many files open" error - check your 
>> /var/log/webobjects.log for that message.
>>
>> I was able to work around the problem by writing a CRON task the 
>> counts the number of open TCP connections from WOTASKD and restarts 
>> WOTASKD when the number grows larger than 240 (the crash/hang occurs 
>> when your reach 255 TCP connections)
>>
>> Here is the line I use to identify the open TCP connections (note; 
>> you will have to adjust your port range if you changed the default or 
>> you have more instances running than I do):
>>
>> 	sudo lsof -i tcp:2001-2050 -P
>>
>> Here is the line I use to restart WOTASKD (I had to fix the startup 
>> script too)
>>
>> 	sudo SystemStarter restart 'WebObjects Services'
>>
>> I run this script every 10 min in a CRON job:
>>
>> #!/bin/sh
>> #
>> # Restart WOTASKD when it has too many TCP connections open
>> #
>> # by Mark Fisher 11/28/03
>> # must be run as "root"
>> #
>> maxprocs=240
>> PROCESSLIST=`sudo lsof -i tcp:2001-2050 -P`
>> i=0
>> for process in $PROCESSLIST
>> do
>> 	i=$((i+i))
>> done
>> if [ $i -gt $maxprocs ] ;then
>> 	echo "Restarting WOTASKD"
>> 	`SystemStarter restart 'WebObjects Services'`
>> fi
>> echo ""
>> # end shell script
>>
>> This solves the problem for now but I really, really wish I knew the 
>> cause of the problem - Is WOTASKD *really* that broken in 10.3?  I 
>> wonder how many others out there are having the same trouble getting 
>> 10.3 Server & WebObjects working reliably.
>>
>> If *ANYONE* (Apple are you listening) has any more information on 
>> this problem I would like to hear from you.
>>
>> Thanks in advance,
>>
>>
>> Mark FIsher
>> Chief Software Engineer,
>> 	Learning Quest, Inc.
>>
>> _______________________________________________
>> WebObjects-dev mailing list
>> [email protected]
>> http://www.omnigroup.com/mailman/listinfo/webobjects-dev
>
> _______________________________________________
> WebObjects-dev mailing list
> [email protected]
> http://www.omnigroup.com/mailman/listinfo/webobjects-dev