[nevow.appserver.NevowSite] Could not accept new connection (EMFILE)

Werner Thie <[email protected]> Thu, 06 Sep 2007 10:13:54 +0200
Newsgroups gmane.comp.python.quotient.dev
Message-ID <[email protected]>
Hi all

I am running into a major problem when running an nevow site with 
LivePages under low load conditions:

After some time (non-determininstic), say 7-10 minutes with continous 
interaction of say 4 clients the server disconnects the clients with no 
apparent reason and logs a host of

Could not accept new connection (EMFILE)

After that, one or more tracebacks follow, which say that too many files 
are open in /tmp. Concurrent supervising of /tmp shows, that at most one 
open tmp file can be observed at any one time, except when the above 
line 'Could not accept new connection (EMFILE)' is logged.

uname -a
OpenBSD sfjass 4.1 GENERIC.MP_RAID#0 i386

ulimit -a
time(cpu-seconds)    unlimited
file(blocks)         unlimited
coredump(blocks)     unlimited
data(kbytes)         1048576
stack(kbytes)        16384
lockedmem(kbytes)    unlimited
memory(kbytes)       unlimited
nofiles(descriptors) 1024
processes            532

If I connect more clients (20), the problem shows up in the log as 
shorter burst of EMFILE entries, no tmp file tracebacks, no disconnects 
of clients at a rate of about once every three hours.

Does someone out there have a clue what could trigger this problem.

I am grateful for any hint, because this one could kill my project.

Thxs, Werner

...several hundred of the same line, same timestamp
2007-09-06 09:48:48+0200 [nevow.appserver.NevowSite] Could not accept 
new connection (EMFILE)
2007-09-06 09:48:48+0200 [HTTPChannel,7704,217.71.84.26] Unhandled Error
         Traceback (most recent call last):
         --- <exception caught here> ---
           File 
"/usr/local/lib/python2.5/site-packages/twisted/internet/selectreactor.py", 
line 139, in _doReadOrWrite
             why = getattr(selectable, method)()
           File 
"/usr/local/lib/python2.5/site-packages/twisted/internet/tcp.py", line 
362, in doRead
             return self.protocol.dataReceived(data)
           File 
"/usr/local/lib/python2.5/site-packages/twisted/protocols/basic.py", 
line 241, in dataReceived
             return self.rawDataReceived(data)
           File 
"/usr/local/lib/python2.5/site-packages/twisted/web/http.py", line 1081, 
in rawDataReceived
             self.allContentReceived()
           File 
"/usr/local/lib/python2.5/site-packages/twisted/web/http.py", line 1072, 
in allContentReceived
             req.requestReceived(command, path, version)
           File 
"/usr/local/lib/python2.5/site-packages/twisted/web/http.py", line 628, 
in requestReceived
             self.process()
           File 
"/usr/local/lib/python2.5/site-packages/nevow/appserver.py", line 125, 
in process
             environ={'REQUEST_METHOD': 'POST'})
           File "/usr/local/lib/python2.5/cgi.py", line 536, in __init__
             self.read_single()
           File "/usr/local/lib/python2.5/cgi.py", line 666, in read_single
             self.read_binary()
           File "/usr/local/lib/python2.5/cgi.py", line 676, in read_binary
             self.file = self.make_file('b')
           File "/usr/local/lib/python2.5/cgi.py", line 790, in make_file
             return tempfile.TemporaryFile("w+b")
           File "/usr/local/lib/python2.5/tempfile.py", line 466, in 
TemporaryFile
             (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
           File "/usr/local/lib/python2.5/tempfile.py", line 242, in 
_mkstemp_inner
             raise
           File "/usr/local/lib/python2.5/tempfile.py", line 236, in 
_mkstemp_inner
             fd = _os.open(file, flags, 0600)
         exceptions.OSError: [Errno 24] Too many open files: 
'/tmp/tmpea9YuM'

2