Re: where to run spread daemons?

On Saturday, July 20, 2002, at 03:19 , Aditya wrote:
> be a problem right now. What other horrible things can I expect to 
> happen if
> I don't run spread locally on each webserver?

If you have a cluster of 10 web servers on which each Apache instance 
has 256 children, that is 2560 TCP connections to one box.  That is not 
optimal.  Spread (with poll patch) should be able to handle that, but it 
will be abusively slow.  Also, you have to also consider the remote 
spread daemon's OS now has to maintain 2560 active, long-term TCP/IP 
sessions in addition to its normal workload (as opposed to a single unix 
domain socket session).

Now consider that you want more than just spreadlogd listening to the 
log stream.  You want a monitor here and there and an addition logger 
for fault-tolerance on another machine.  This too can be done of TCP/IP 
all to one Spread daemon, but you loose your fault tolerance AND all the 
log streams have to be sent back out on the network to be delivered to 
the monitor Spread clients.

TCP/IP isn't the optimal networking protocol to transmit these packets 
(log messages) across a local area network.  Spread however, is close to 
"as good as it gets".  So, if you have a Spread daemon running on each 
web server and on each monitoring server, you get all of the messages 
going everywhere at almost no additional cost.  You also only have 256 
or so open sockets to each webserver's Spread daemon and one or two unix 
domain connections on the logging and monitoring hosts.  You can have 
multiple sources (web servers) and multiple sinks (loggers, monitors, 
graphers, etc.) and it is still cheaper than the single logger model to 
which you currently subscribe.

If you are not pushing much traffic (less that 10 million log lines/day) 
you probably won't notice any logging bottlenecks the way you are doing 
it.  And, the way you are doing it now it a little easier to 
administrate if you aren't a networking/systems admin -- configuring and 
managing high traffic Spread rings (and keeping them stable) can be 
challenging at times.  So, if you are low traffic, then you probably 
won't run into any "show-stoppers" running with one remote Spread daemon.

On a side note, the messages are tagged by site (as the private spread 
user name is), so if you have one machine running spread, it can be hard 
to tell which web servers are logging which hits.  If you run an 
individual spread daemon on each, the message that are sent will have 
the hostname as a part of the sender's name, so you can actually break 
out which hits were served by which hosts.  This technique can be 
invaluable in a real-time (top-like) monitor.  With something like this, 
you can see if your load-balancing configuration has resource contention 
issues like assigning too many requests to a single machine.  These 
sorts of problem often present hard to decipher side-effects when 
looking at daily, hourly or even minute-to-minute summary information.

And lastly, Spread is designed to be a distributed message distributions 
system.  In your model, you are not using it as intended.  It will work 
that way, but it isn't the suggested configuration. So, if something 
goes wrong and you try to troubleshoot it and look for help in the user 
community, most likely they will tell you first to set it up in a 
distributed fashion and see if your problems go away :-D

Hope this helps a bit!

--
Theo Schlossnagle
Principal Consultant
OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
Phone:  +1 301 776 6376       Fax:  +1 410 880 4879
1024D/82844984/95FD 30F1 489E 4613 F22E  491A 7E88 364C 8284 4984
2047R/33131B65/71 F7 95 64 49 76 5D BA  3D 90 B9 9F BE 27 24 E7