Re: Handling large log files

First, thanks for the great responses!  Aside from the fact that we
need a beefier system (2x P3 1.4 GHz, 3 GB RAM, RAID-5... ouch), it
looks like I have a lot of work to do.

Also, thanks for providing some idea of the specs I will need to use
for a central log server.  I believe our goal is to have around 300
servers sending logs (most of them should be less chatty than the
current ones).  If you don't mind me asking, roughly how many servers
should I expect to have generate 1 GB of logs?  I realize there really
isn't an accurate answer here, but I'm trying to get a rough ballpark
figure.

> Marcin wrote:
>
> - see if the architecture can be improved. Can you use multiple log
> servers? Is there
> a logical way of segmenting the log traffic - OS to box 1, db
> transactions to box 2, etc.?
> Post to the project's mailing list, there should be people who use it
> for larger installations,
> and willing/able to provide specific suggestions.

I'll see if this is an option.  Along these lines, I'd eventually like
to be able to turn log messages into events and be able to correlate
them with other messages, IDS alerts, etc.  I think that once I
compress the duplicates, and get rid of a lot of noise, I could
forward the results to an OSSIM box and use it for correlation,
alerts, etc.

> Paul wrote:
>
> What are you trying to achieve with your log analysis, as in, what
> sort of actions would the review of this daily log report trigger?
> Would you want to or should you move to a model where search/analysis
> is happening in near-real time instead of once daily?  That's going to
> be helpful in knowing what kind of solution you should be looking at.
> Also, while it's overpowering your logcheck scripts, 5GB/day of log
> data is nothing when you're talking about firewall logs.
>
> PaulM

We are primarily looking for security related events.  Real time
analysis/reporting of events is an eventual goal, but that seems a lot
more difficult to do in some regards.  Initially, I'd like to at least
have a summary I can look at daily (probably along the lines of what
David posted below) and then I could transition to more real-time
analysis.  Does that sound reasonable?

> David wrote:
>
> I don't like the idea of filtering out messages completely, the number of
> times that an otherwise 'unintersting' message shows up can be significant
> (if the number of requests for a web image per day suddenly jumps to 100
> times what it was before, that's a significant thing to know)

Duly noted.  Thanks!

>
> the key is to categorize and summarize the data. I have not found a good
> commercial tool to do this job (there are good tools for drilling down and
> querying the logs), the task of summarizing the data is just too site
> specific. I currently get 40-80G of logs per day and have a nightly process
> that summarizes them.

This is good to know as well.  I'd like to avoid commercial tools if
possible to save money (although Splunk seems pretty darn useful).

>
> *Solid plan of attack from David*
>

Thanks for all the great information from everyone.  I'll be jumping
into this today!

-Nate