fastcgi processmanager, graceful restart and keepalive don't work together

I think there is a fundamental problem with mod_fastcgi
processmanagement, keepalives and apache graceful restarts.

Here is how I can trigger the issue 100% of the time with
apache 2.0.55 and mod_fastcgi-SNAP-0404142202:

- Do a fastcgi request in a way that will keep the connection alive
  (typically by using HTTP 1.1 and having keepalive not turned off in
  your apache config. Simply accessing the fastcgi URL from a modern
  browser like firefox 1.5 should be enough)
- do "apachectl restart"
- do the same fastcgi request again on the kept alive connection
  (all this needs to happen within the keepalive timeout).

The second request will fail

My analysis of the reason:

Apache starts up as a main server with a number of handler children.
In the fastcgi case there's also a fastcgi processmanager started.
All the handler children have a pipe connection to the processmanager
over which they can talk to the fastcgi processmanager.

When apache gets a graceful, the main apache process makes a connect to
all handler children which then gets immediately closed. The handler
children use this a trigger to exit. The fastcgi processmanager also
gets a SIGTERM and exits. A new processmanager is started and new
apache handler processes that again talk to to the processmanager over
a socket.

However, in case of keepalive the following happens:

the apache handler that handled the request doesn't immediately go
back to waiting for connects (accept state), but instead does a select
on the kept alive connection waiting for a certain time (the keepalive
timeout) for additional requests. If now a graceful restart happens,
this child doesn't get the empty connect and keeps on listening for
something to happen on the kept alive connection. All other handler
children not in keepalive exit and a few new ones may get spawned.
And, most importantly, the fastcgi process manager also exits and a new
one is started. Then another fastcgi request comes in on the apache
handler in keepalive state. It tries to talk to the processmanager, but
its socket went to the OLD one that's gone, and it fails. This handler
process will now exit and after a bit of time the original requestor
gets an internal server error message.

Workarounds:
  - Make your fastcgi programs always send a "Connection: close" header
    (drawback: if you use another processmanager like mod_fcgid this
    also stops keepalive, while it's not needed there)
  - Or configure apache not to do keepalive for the URLS backed by
    fastcgi (drawback: can only be configured for a whole virtual
    server)

A proper solution would probably be something like:
  - When an apache handler can't talk to the old processmanager, it
    should connect to the new one. That would need the processmanager
    to be able to accept new connections of course
  - Or make the new processmanager start up with the same pipe sockets
    as the old one.
(I suppose the second one is both easier to implement and faster)
___________________________________
fastcgi-developers mailing list
http://fastcgi.com/fastcgi-developers/