Re: SnortSnarf memory usage on large files

Hello Jason,

(comments inline)

At 4:35 AM -0500 2/2/03, Jason Falciola wrote:
>Greetings,
>
>I've read through the archives of this list and whenever someone has
>problems with SnortSnarf consuming memory when processing "large" files,
>the answers seem to be along the lines of this snippet from
><http://www.silicondefense.com/pipermail/snortsnarf-users/2002-March/000258.html>:
...

>The problem is that many are faced with a scenario where the last 2
>suggestions aren't an option.  The SANS GCIA practical requires the
>processing of files that are in the range of 300 MB.  All alerts must be
>processed to give a thorough analysis.
>
>A friend has run into problems processing these files on a dual PIII 1 GhZ
>machine w/ 1 Gig of RAM and 500 MB of swap space runing RH Linux 6.2.  I
>would have thought that this box would have enough resources to handle
>these files, but Snortsnarf quits with the message that it ran out of
>memory.  Perl is version 5.005_03 and kernel is 2.2.14-5.0smp #1. 
>I'm wondering if an upgrade would help, or if the same
>issues would arise.
>
>I guess my fundamental question is whether Snortsnarf should die or just
>take a long time to run when processing large files.  I would have thought
>the latter, but maybe I'm wrong.

SnortSnarf will run to completion as long as the OS keeps giving Perl 
all the memory it needs.  If that stops being the case (e.g., RAM and 
swap are exhausted), then Perl will quit with an out of memory error. 
This sounds like what you are running into.


>   What cpu/mem requirements are there for
>processing 300 mb of snort log files, assuming the box isn't doing
>anything else?

CPU doesn't matter for SnortSnarf's ability to complete, its just the 
main part of the divisor on time to completion.  I've never studied 
the memory requirements.

I suggest dividing the processing of logs up by time using -mintime 
and -maxtime and referring between the sets of HTML as needed.  You 
might find the annotations feature helpful with getting a complete 
view of the activity even when it is divided.  SISR may be useful in 
tracking all the incidents in the log; you can even use to combine an 
incident that was divided between SnortSnarf runs.

>Another interesting suggestion was from
><http://www.silicondefense.com/pipermail/snortsnarf-users/2002-March/000261.html>,
>but I'm unclear on how to actually do this.  Any pointers?
>
>"there were huge savings to be had in caching dns answers (if you use that
>option), and especially in storing and sorting the alert lists."

DNS caching is mainly useful if you are using -dns to resolve 
addresses other than local ones (which I don't recommend).  (This 
does not sound like your problem.)  DNS caching and whatever the 
second part of the sentence is referring to aren't a features that's 
available with SnortSnarf at present.

Good luck,

   Jim
-- 
|*     Jim Hoagland, Associate Researcher, Silicon Defense     *|
|*    --- Silicon Defense: The Cyberwar Defense Company ---    *|
|*   [email protected], http://www.silicondefense.com/    *|
|*  Voice: (530) 756-7317                 Fax: (530) 756-7297  *|