Fw: Fairly critical issue with apache logging to error_log_spread.pl

> Greetings folks:
>
> Since we are using Apache2, we can't use mod_log_spread... at least
directly
> and officially in production - yet. However, we are making use of just a
> portion of it - specifically the included file "error_log_spread.pl". This
> small perl script is really intended for logging apache errors, but we use
> it for the transferlog so that all transfer logs go to a central host from
> multiple websites. For example, in most of our virtual host setups on
> apache, we have...
>
> CustomLog "|/usr/local/sbin/error_log_spread.pl -g hdcoutlet" combined
>
> We have made one small customization to the stock error_log_spread.pl
> program - any messages it receives via chomp have the trailing LF removed,
> so this one section of code is now...
>
> if(($ret = Spread::multicast($mbox,
>                              RELIABLE_MESS, $group, 0,
>                              ($hosttoggle?$hostname." ".$_:$_."\n")))>0) {
>   print STDERR "Successfully multicasted $ret bytes to [$group]\n" if
> $debug;
> } else {
>   print STDERR "Failed multicast $_ to $group: $sperrno\n" if $debug;
> }
>
> Note specifically the addition of "\n". This gets us the original log line
> written back out with the LF "intact".
>
> On to the problem - we have probably 15 or 20 virtual hosts that use the
> above CustomLog statement, piping through error_log_spread.pl, with each
> site having a different log group so spreadlogd can put the logs in
> different files. All is well and good, this works great. However, whenever
> we do an "apachectl stop" and then "apachectl start", sometimes, but not
all
> the time, SOME of the error_log_spread.pl pipes print up "Could Not
> Connect". It is kind of a random thing. If we do an apachectl stop and
> apachectl start again, most of the time they all start right (no
messages),
> but sometimes a few less fail, do it again, then none fail. I am making a
> wild guess that this is some kind of timing issue with 15 or so pipes
coming
> up trying to connect to spread. This becomes very problematic for us,
> because our developers release code with cvs in an automated fashion. They
> can "goose" a machine that will do a cvs export, then that machine will
> rsync the website code modules to all the webservers in the wackamole
> cluster. In order to keep the sites up while this is going on, this
> "controlling" machine needs to send ssh commands to the webservers, one at
a
> time, to do an apachectl stop, then a wackatrl -f, (the load is all on
other
> webservers in the wackamole cluster now), then it rsyncs the website code,
> then a wackatrl -s, then an apachectl start. It then does this in turn to
> each machine in the wackamole cluster. Of course, it's a real problem if
> when the apachectl start is done, that the logs dont come up because of a
> "could not connect" for just a few of the sites.
>
> Perhaps (wild uninformed guess here) all that needs to be done is for the
> error_log_spread.pl program to be changed to try to connect to spread
> several times, rather than just trying once and giving up. I'm not a perl
> programmer - if this is the issue, could someone show how the script
should
> be changed to do it? If this isn't the likely issue or best solution - any
> ideas??
>
> Thanks!
>
> Jay West
>

---
[This E-mail scanned for viruses by Declude Virus]