Re: Analyzing tons of logs

Hi There...

I assume this is packet data (ie. libpcap).  I am amazed that no one mentioned tethereal (or is it twireshark 
now).  Regardless of the data size, you will have to have at least three times the disk space to do any kind 
of evidentiary work on to begin with.  I am sure that was part of the engagement planning, as you seem to 
be in possession of the data.  Regardless, most of the work can be done with 1.5 times the disk space of 
your sample as long as you don't keep the non-interesting files hanging around.

Obviously, this will require some measure of automation.  In order to prove what you find, you will have to 
make the case that you know your tools.  This almost requires a custom script.  If you don't write it, you 
must at least be able to read it.  Otherwise, you are working on trust, which is hardly a standard I want to be 
faced with in court.  As with any elephant, you can only eat it one bite at a time.  Here, a single bit can be a 
bite (sic).

twireshark will handle any packet capture.  It will do so on a box without a GUI, putting those resources into 
the work, rather than driving some silly screen saver.  It can cut out packets by time, or any single packet 
bit.  These are packets, so the opportunities for slices abound before you ever get to the content.  There is 
even a ring buffer mode to break the files out according to several criteria.  BTW, twireshark comes with 
wireshark, so you can fire up your GUI if you feel the need.

tcpdump can also break out the base anomalies.  If you want to look at the DNS traffic, or !(udp[0:2] = 0x35 
or udp[2:2] = 0x35), you can do that.  I have run tcpdump filter files as the big as 5Kb, every hour of every 
day.  The nice thing is that you can divide your working copy into viewable chunks and track them with a 
new vi file.  You were going type into your tools anyhow, right?

I assume that the snaplen on these packets is not 65535.  If it is, you can run them through snort, using the 
default signature set, or imagine your own.

The key here is that by reducing the code that you use, YOU present the data you found.  YOU have the 
best opportunities to find false positives, as well as false negatives.  You also reduce the time needed to 
assure yourself that the tools are tracking everything you think they are.   I have not met a machine that can 
out imagine the human mind.  Any thing else is a shortcut.

B.

On 30 Mar 2007, a message purporting to be from Chetan Gupta appeared:

Date sent:	Fri, 30 Mar 2007 08:57:33 +0530
From:	"Chetan Gupta" <[email protected]>
To:	"Peter Sicilia" <[email protected]>
Subject:	Re: [logs] Analyzing tons of logs
Copies to:	[email protected]

> Dear list members,
> Thanks a lot for your wonderful insights on this topic. I've noted down all
> the points mentioned and am gonna try out the various solutions
> suggested.Precisely, I am gonna try and lay my hands on the following tools
> in the coming weeks:
> 
> 
>    1. OSSIM
>    2. Manageengine adventnet
>    3. Splunk
>    4. Logpp
> 
> I would post regarding my experience with these tools.
> How about sguil? Thats an NSM tool I guess. Has any one tried it?
> 
> Can anyone suggest some good repositories for sample logs containing some
> attacks. I know I could get some from honeynet.org site or loganalysis.org.
> Any other aprt from these two?
> Thanks again,
> 
> -- 
> Chetan Gupta ENCE, GCIA, GCFA, CEH, CCNA, CIW Sec. Analyst
> Forensic Consultant
> 
> Mobile: +91 9810718489
> ------------------------------------------------------
> Online Computer Forensics Magazine
> http://www.niiconsulting.com/checkmate
> ------------------------------------------------------
> 

Bill Scherr IV, GSEC, GCIA
Principal Security Engineer
EWA Information and Infrastructure Technologies
[email protected]
[email protected]
703-478-7608