Re: ConnectionPools and removeSelf

On Saturday, Oct 18, 2003, at 18:55 US/Eastern, Stephen Wang wrote:

> I'd appreciate help squeezing out a bit more performance out of my
> backhand-assisted setup. I'm running a heavily-trafficked two-tier 
> setup
> (single master proxying to several slaves). I currently have
> BackhandConnectionPools off on the slave servers.

BackhandConnectionPools (on|off) shouldn't have any effect on the 
slaves at all.  The slaves don't have the opportunity to redirect 
connections from clients, so to use connection pooling or not isn't 
relevant.

> My first question: Whenever I turn BackhandConnectionPools off on the 
> master
> server, all 1024 apache processes get eaten up within a minute and the
> server dies. I don't think that apache is tearing down the connections 
> in a
> timely manner. Is there anything I can do to correct this? With 
> connection
> pools on, I'm still getting my apache error logs stuffed with the 
> "connect()
> time out" error messages.

This is really odd.  Usually we should see the opposite.  Things tend 
to be a bit less stable under heavy load with connection pooling on.  
The reason for this is that the moderator process becomes essential to 
serving each request.  The moderator establishes connections and the 
children request the open connection from the moderator, the moderator 
hands it to the Apache child, the child uses it and then the child 
hands it back.  With a vast number of children you see the moderator 
process high contended for.

As you said, perhaps it isn't tearing down the connections correctly.  
Assuming you want to disable connection pooling on the front end box, I 
would suggest turning keep-alives off on the backend boxes as well as 
force downgrading all client connections to HTTP 1.0.  The protocol is 
a bit simpler as it doesn't use chunked encoding and there is less 
plumbing required to proxy it back to a slave server.  There is a force 
downgrade example in the default Apache httpd.conf file (for MSIE), you 
should be able to generalize it to downgrade everything.

If you aren't going to use connection pools (which may achieve higher 
performance), then I would recommend setting keepalives off EVERYWHERE. 
  Most big (10 million+ hits / day) sites running Apache 1.3.x turn of 
keep alives to achieve more client concurrency at cost of (hardly 
noticeable) degraded client-perceived performance.

> Second question: I have RemoveSelf set on the master server. However, 
> it
> appears that whenever a slave server uses up all of it's apache 
> processes,
> the master server ignores RemoveSelf and tries to serve pages anyways. 
> Is
> there anyway to circumvent this behavior so that the master server will
> absolutely NEVER try to serve pages itself (and perhaps redirect to 
> another
> slave server)? I believe that this behavior was raised by another user 
> but I
> don't believe I saw an answer.

The answer is clear cut.  The problem is that the client has a 
connection to a server and has asked a question.  So, what if that 
server is absolutely unable to make a request to a slave server?  It 
has to serve the page from somewhere (even if it is an error).

The quick answer is that you can use MulticastStats two argument 
version to advertise the front end machine as "someone else".  This 
means the front end box will see all slave servers and itself as usual, 
but it will see "itself" as the IP:port of some other machine whether 
it be one of the slave servers or a little thttpd instance that serves 
"Sorry the system is overloaded" pages.  That set up combined with the 
BackhandSelfRedirect On option should solve your problem.

BackhandSelfRedirect on tells mod_backhand that even if it chooses 
itself, that is should proxy there no let apache internally serve it.  
The caveat here is that mod_backhand needs to be able to choose itself. 
  If it can't choose anyone, it will serve it internally.  So, assuming 
you have a cluster of 8 or so machines I would use something like:

MulticastStats fallbackip:fallbackport multicastip:port,ttl
BackhandSelfRedirect On

Backhand byAge
Backhand removeSelf
Backhand byRandom
Backhand byLogWindow
Backhand byBusyChildren
Backhand addSelf

This will give you a list of servers (the first of which should only be 
used) guaranteeing that the last server in the list is "yourself".  But 
yourself is really some fallback server and you will redirect there.

> Thanks in advance for any help.
>
> _______________________________________________
> backhand-users mailing list
> [email protected]
> http://lists.backhand.org/mailman/listinfo/backhand-users
>
// Theo Schlossnagle
// Principal Engineer -- http://www.omniti.com/~jesus/
// Postal Engine -- http://www.postalengine.com/
// Ecelerity: fastest MTA on earth