Re: Analyzing tons of logs

Hi Chetan,

For the amount of data that you want to analyze, I
agree with Anton, there is no single solution
(commercial or open source) that can handle that.
They will all fail miserably... First of all, you
will have a huge network bandwidth usage, not speaking
about disk space and cpu/memory power to analyze
all of that (specially considering most tools use
regex).

What I would suggest is some form of segmentation or
partition of all this data. You can create one
log analysis "station" for each department or each
section of your company. This way you can perform
your analysis based on the goals of each department.

For example, on your main servers inside the DMZ, you 
can setup a "DMZ" log station, where you can monitor
the logs from there.. This way traffic doesn't need to
leave each segment and the memory/disk/cpu
requirements
can be easily manageable (oh, and it is scalable).


Hope it helps...

--
Daniel B. Cid
dcid ( at ) ossec.net






--- Chetan Gupta <[email protected]> escreveu:

> Dear List Members,
> 
> I am looking for opinion from the experts for a
> particluar problem.
> 
> How do we go about log analysis if we have tons
> (maybe in trillions) of
> logs from lets say tcpdump (raw logs) or some
> firewall (like netscreen or
> pix)?
> What would be the best way to normalize and analyze
> these logs in the
> shortest possible time?
> Import them into a database? Use a commercial
> application like arcsight?
> loglogic? simple text editor like editplus?
> Any suggestions/comments would be appreciated.
> 
> Regards,
> 
> Thanks and Regards,
> ERNST & YOUNG ®
> Ernst & Young Pvt. Ltd
> 
> Chetan Gupta
> Consultant
> Risk and Business Solutions
> FIDS
>
_______________________________________________________
> 
> 
> Mobile:      +91 - 9810718489
> Fax:          +91 - 11 - 2661 1012
> URL:          http://www.ey.com/in
>
_______________________________________________________
> 
> 
> 
>
----------------------------------------------------------
> The information contained in this communication is
> intended solely for the use of the individual or
> entity to whom it is addressed and others authorized
> to receive it.   It may contain confidential or
> legally privileged information.   If you are not the
> intended recipient you are hereby notified that any
> disclosure, copying, distribution or taking any
> action in reliance on the contents of this
> information is strictly prohibited and may be
> unlawful. If you have received this communication in
> error, please notify us immediately by responding to
> this email and then delete it from your system.
> Ernst & Young is neither liable for the proper and
> complete transmission of the information contained
> in this communication nor for any delay in its
receipt.>
_______________________________________________
> LogAnalysis mailing list
> [email protected]
>
http://www.loganalysis.org/mailman/listinfo/loganalysis


__________________________________________________
Fale com seus amigos  de graça com o novo Yahoo! Messenger 
http://br.messenger.yahoo.com/