remote logging via socklog/multilog + Zebedee
Aldasair Grhone <[email protected]>
| Newsgroups | gmane.comp.djb.syslog |
|---|---|
| Message-ID | <[email protected]> |
In my attempt to get a couple of dozen hosts to log to a central log server over an untrusted network I have come to try a combination of socklog/multilog + Zebedee (an encrypted tunnel utility). In the standard configuration supervise hands socklog's output to multilog, which then logs to a file, and when that file gets to a certain size (4k minimum) it runs a "processor" (which in my case is socklog's "tryto" utility, which uses tcpclient to connect to the Zebedee tunnel on localhost, ending up on the other side of the tunnel on the log server, which also runs socklog). A diagram of my configuration for each of my hosts: socklog -> multilog -> tryto -> tcpclient -> Zebedee localhost -> Zebedee log svr -> socklog The main problem I've run across is that multilog waits too long to run its "processor" (socklog's "tryto" utility, which connects to the Zebedee tunnel). multilog will only run the processor when the current log fills up to a minimum of 4k. That means that if a single line got logged saying, for example, "disk full", the central log server may not hear about it immediately, if ever. That means my log server could miss critical information. Of course, I could hack multilog's source to decrease the minimum log file size at which the processor gets run (let's say to 40 bytes, which should catch most single line entries), but then if a large number of lines are logged in a short period of time there'll be an inordinate number of invokations of the tryto and Zebedee utilities on the host initiating the connection, not to mention the excess network traffic and the processes spawining on the log host to handle each seperate network connection. Clearly implementing this solution would throw me from the frying pan and in to the fire. The other possibility is to do some log filtering on each host individually, taking appropriate action when necessary, until the log gets to a reasonable size, at which time it would get logged to the log server, and processed there. But that would defeat the purpose of having a centralized log server, where all the log filtering should be taking place, not on each individual host. Please correct me if I'm mistaken, but what seems to be required is a change in architecture. multilog should allow the writing of its logs to a processor indefinitely, without having to spawn a new processor every X bytes. This would allow a persistent network connection to exist and for all data to get written to the log server as it arrives. Alternatively, something like tryto could be inserted between socklog and multilog. But instead of the normal behavior it would write both to STDOUT, which could be fed to multilog to log to the host's local log as usual, and at the same time try to feed it's input to another program (ie. tcpserver, etc). But then you'd run in to the problem of what tryto should do with its data should the program it tried to run fails (ie. the network connection was unavailable). At least in this scenario all the data would be saved locally, but it would be nice if any data that failed to transmit could be attempted to be retransmitted in the future. The final solution is to move to a completely different set of tools for doing remote logging. Tools that would continuously log every byte of data to my central log server, over an encrypted channel, would retry upon failure, and wouldn't spawn a new process nor open a new connection every 40 bytes. Is there such a beast? Aldasair