Re: synchronizing multiple services rotation

[FWIW I'm using runit, not daemontools]

* Daryl Tester <[email protected]> [20091109 03:37]:
> Joan Picanyol i Puig wrote:
> 
> >I'm setting up a system in which several consumers do some batch
> >processing of data logged from several producers, all of them services.
> >Producers are continuously running services, and on log rotation they
> >start once the consumer services (which detect what producer are being
> >rotated from reading $PWD). They are fed the rotated file, retrieve
> >saved state information including last execution on the filesystem, and
> >open the other consumer's current log.
> 
> This sounds ... confusing (which to me is usually a warning).

OK, let me try to state the requirements in a simpler way:

1. N > 1 "producer"s (data loggers), for which rotated logs must be
   kept
2. M > 1 "consumer"s (data processors), each of which need data from any 
   subset "producer"s

And let me try to explain again the system's architecture:

All "producer"s share a common processor script that reads it's
"consumer"s from the environment (set up in each "producer"'s log
configuration), signals them ('/command/sv once /service/consumer') and
feeds them 'current' through stdin upon rotation. "consumer"s use $PWD
to find out what they are being fed through stdin and go fetch the rest
of the data they need at the other's (not being currently rotated)
"producer"s log dir.

> >This way, the consumers are invoked at least as often as the fastest
> >rotating producer and no data is lost. 
> 
> So any of the participating process' multilog processor can process
> *all* of the "producer" log files?

Yes (assuming *all* refers to the set of "producer"s), but the data must
still be available for other "processor"s.

> It's essentially the same script?

"consumer"s are actually programs which process data, not just simple
scripts. That's one of the reasons for setting them up as services, so
processing can proceed asynchronously from log rotation (note that it's
not this that causes the race). The other is the fact that every
"producer"'s log might be needed any subset of "consumer"s.

> I guess it would have to be if it's processing another producer's
> current file (but then what does the processing script do with another's
> current file?  How is it stopped from being processed again?  Or aren't
> you looking at any of the previous data (is this where the saved state
> is coming in?).

"consumer"s use the saved state to avoid processing duplicate data and
to ensure correctness, since log rotation is not coupled with
"consumer"s logical intervals.

> >However, there's a race here if more than one producer rotates
> >simultaniously: the consumer might not see the logs from the producer
> >which didn't cause it's invocation.
> 
> Without really knowing enough of your problem, would:
> 
> *)  Having a common script for processing (possibly parameterised to
> know which producer invoked it)
> *)  Using setlock (or the singleton locking mechanism of your choice)
> within each multilogs's processor to fire off the above script

My reading of svlogd.c (and multilog.c for that matter) tells me that
rotation (i.e.: 'current' renaming) takes place before 'processor''s
invocation. Thus invoking setlock within the processor does not close
the race.

I hope the issue is clearer now. My only idea so far is to not feed
current through stdin but rather make "consumer"s scan all "producer"s
log directory for data.

Ideas? Suggestions? 
--
pica