synchronizing multiple services rotation

Joan Picanyol i Puig <[email protected]> Thu, 5 Nov 2009 18:37:13 +0100
Newsgroups gmane.comp.djb.syslog
Message-ID <[email protected]>
Hi, 

I'm setting up a system in which several consumers do some batch
processing of data logged from several producers, all of them services.
Producers are continuously running services, and on log rotation they
start once the consumer services (which detect what producer are being
rotated from reading $PWD). They are fed the rotated file, retrieve
saved state information including last execution on the filesystem, and
open the other consumer's current log. This way, the consumers are
invoked at least as often as the fastest rotating producer and no data
is lost. 

However, there's a race here if more than one producer rotates
simultaniously: the consumer might not see the logs from the producer
which didn't cause it's invocation.

producer 1's log rotates
                            consumer is started once
producer 2's log rotates
                            consumer is fed producer 1's log through
                            stdin and goes reading producer 2's current
                            which is already empty
                    
The only way to block log rotation seems to be blocking the /previous
rotation/ processor, but I can't come up with a way to use this to
close the race across diferent services.

Ideas? Suggestions? I need to solve this for the general case (more than one
consumer), forward looking insights in that direction most welcome.

tks
-- 
pica