High performance syslog aggregation
Steve Bernacki <[email protected]> Fri, 30 Nov 2007 15:36:12 -0500
| Newsgroups | gmane.comp.sysutils.loganalysis |
|---|---|
| Message-ID | <[email protected]> |
First of all, thank you to those who responded to my last message regarding syslog load balancing. I'm currently researching how to best implement a high-peformance, high volume syslog aggregation. In our current environment, we have many devices logging to a small set of "front end" syslog aggregators which run syslog-ng. Currently, these front-end aggregators have a number of filters enabled, which negatively impacts thruput. What I'm looking to do is place new systems in front of the existing systems that simply capture, queue, and forward messages based on a very limit set of searchable criteria (no regexes needed!). These systems should also have the ability to queue incoming messages onto disk and replay them in the event that a receiver goes down or becomes temporarily overburdened. My first thought was to implement an architecture similar the following: Hosts --(UDP)--> (front end) --(TCP)-->(multiple receivers) In researching my "free" and "nearly free" options for doing this, syslog-ng community edition comes the closest, however only the commercial version supports "store and forward" for TCP syslog streams. rsyslog looks like a promising alternative option, although I haven't been able to confirm through its documentation whether it supports any type of "store and forward" mechanism. What other tools and/or solutions have I missed? I've considered using syslog-ng to log to a program which ultimately stores and forwards messages, although it seems like there must be a better way of doing this. Thanks once again for your guidance, Steve -- Steve Bernacki, Jr To date, the Pan-Massachusetts Challenge has raised 204 million dollars for cancer research. Get involved! http://www.pmc.org/