Re: synchronizing multiple services rotation

(* Reply to dev null'd *)

Joan Picanyol i Puig wrote:

> I'm setting up a system in which several consumers do some batch
> processing of data logged from several producers, all of them services.
> Producers are continuously running services, and on log rotation they
> start once the consumer services (which detect what producer are being
> rotated from reading $PWD). They are fed the rotated file, retrieve
> saved state information including last execution on the filesystem, and
> open the other consumer's current log.

This sounds ... confusing (which to me is usually a warning).

> This way, the consumers are invoked at least as often as the fastest
> rotating producer and no data is lost. 

So any of the participating process' multilog processor can process
*all* of the "producer" log files?  It's essentially the same script?
I guess it would have to be if it's processing another producer's
current file (but then what does the processing script do with another's
current file?  How is it stopped from being processed again?  Or aren't
you looking at any of the previous data (is this where the saved state
is coming in?).

> However, there's a race here if more than one producer rotates
> simultaniously: the consumer might not see the logs from the producer
> which didn't cause it's invocation.

Without really knowing enough of your problem, would:

*)  Having a common script for processing (possibly parameterised to
know which producer invoked it)
*)  Using setlock (or the singleton locking mechanism of your choice)
within each multilogs's processor to fire off the above script

do what you want?


-- 
Regards,
  Daryl Tester

"Scheme is an exotic sports car. Fast. Manual transmission. No radio.
 Common Lisp is Howl's Moving Castle."
  -- Steve Yegge, comparing Lisp families to cars.