Re: Proxy Could not accept new connection

Glyph, thanks for the great feedback.  I'm looking into all of this now.
I don't think it's a ulimit issue as it seems to be "unlimited"

root@Proxy1:~# ulimit
unlimited

I think my first step will be to just make a log observer that will simply
restart the proxy script when it sees (EMFILE) so that I don't have to keep
logging in at night to restart it myself :).

Thanks for filing that ticket.  It would be great if there were a way to be
notified of the particular error inside the code.


I like your suggestion of keeping a list of my open connections.  I could
also keep a time stamp with them and close down the oldest, or any that are
over a certain time period.

I'm new to twisted.  Do you know in what object of the Proxy the new
connections are made and how to access them to store in a list?

Thanks again!

Dave Fowler





On Thu, Aug 6, 2009 at 9:31 PM, Glyph Lefkowitz <[email protected]>wrote:

> On Thu, Aug 6, 2009 at 6:22 PM, dave fowler <[email protected]> wrote:
>
>> I guys.  I'm running a twisted web proxy and after a day or so of usage it
>> stops working and start spitting out these errors.
>>
>> 2009/08/05 14:04 +0000 [__main__.ProxyFactory] Could not accept new
>> connection (EMFILE)
>> 2009/08/05 14:04 +0000 [__main__.ProxyFactory] Could not accept new
>> connection (EMFILE)
>> 2009/08/05 14:04 +0000 [__main__.ProxyFactory] Could not accept new
>> connection (EMFILE)
>> 2009/08/05 14:04 +0000 [__main__.ProxyFactory] Could not accept new
>> connection (EMFILE)
>> 2009/08/05 14:04 +0000 [__main__.ProxyFactory] Could not accept new
>> connection (EMFILE)
>> 2009/08/05 14:04 +0000 [__main__.ProxyFactory] Could not accept new
>> connection (EMFILE)
>>
>> Can anyone expand on what this error is or suggestions on fixing it or
>> debugging further?
>>
>> What is the EMFILE?
>>
>
> It means you're out of file descriptors for your process.
>
> See here for the root cause: <http://linux.die.net/man/2/accept>,
> specifically the part about EMFILE in the Errors section.  I'm actually glad
> to see this error report - Twisted used to fail in a *much* worse way in
> this error condition (<http://twistedmatrix.com/trac/ticket/662>), this is
> the first proof I've seen that our fix works in the wild :).
>
> To increase your per-process file-descriptor limit, use "ulimit -n".  Be
> aware that in order to increase this above 1024, you'll need to be root; I
> think there's a way to set it up to be bigger than that for non-root users
> but I don't know how.
>
> In order to free up file descriptors, keep a list of your open connections
> and shut down some of them.  Sadly, your Factory object won't be notified of
> this problem, so you might have to hack something gross together (like a log
> observer that watches for the string "(EMFILE)").  I just filed this ticket
> to provide a nice, clean notification: <
> http://twistedmatrix.com/trac/ticket/3958>.  This may be a duplicate of
> something, but if so it should be closed with a link to what it's a
> duplicate of.
>
> In some cases, connections won't shut down properly because there is still
> some buffered data and the other end is not available to acknowledge the
> connection's closure.  If this is the case, you can forcibly close the file
> descriptor (although this may cause some ugliness).  To get a good,
> supported way to destroy file descriptors in Twisted, see this ticket: <
> http://twistedmatrix.com/trac/ticket/78>.
>
> Good luck!  Sorry Twisted has not handled this error in the best way for
> you, I hope you help us fix some of these bugs for the next guy :).
>
>
> _______________________________________________
> Twisted-web mailing list
> [email protected]
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
>
>

_______________________________________________
Twisted-web mailing list
[email protected]
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web