SnortSnarf memory usage on large files

Greetings,

I've read through the archives of this list and whenever someone has 
problems with SnortSnarf consuming memory when processing "large" files, 
the answers seem to be along the lines of this snippet from 
<http://www.silicondefense.com/pipermail/snortsnarf-users/2002-March/000258.html>:

<snip>

+ The #1 thing you can do is add more physical memory (or run it on a 
machine with more RAM).  When you need to start using swap space, it 
takes alot more time to complete (though it will eventually complete 
unless you run out of swap space).

+ Run it on a machine with a faster CPU if possible.  Or a less-used CPU.

+ Break it into smaller files.  (Although you loose the benefit of 
seeing it all together.  But see a post I am planning to make.)

+ Have SnortSnarf exclude certain alerts from its processing using 
and input filter.  At present these are -minprio, -sipin, and -dipin.

</snip>

The problem is that many are faced with a scenario where the last 2 
suggestions aren't an option.  The SANS GCIA practical requires the 
processing of files that are in the range of 300 MB.  All alerts must be 
processed to give a thorough analysis.

A friend has run into problems processing these files on a dual PIII 1 GhZ 
machine w/ 1 Gig of RAM and 500 MB of swap space runing RH Linux 6.2.  I 
would have thought that this box would have enough resources to handle 
these files, but Snortsnarf quits with the message that it ran out of 
memory.  Perl is version 5.005_03 and kernel is 2.2.14-5.0smp #1.  I'm wondering if an upgrade would help, or if the same 
issues would arise.

I guess my fundamental question is whether Snortsnarf should die or just 
take a long time to run when processing large files.  I would have thought 
the latter, but maybe I'm wrong.  What cpu/mem requirements are there for 
processing 300 mb of snort log files, assuming the box isn't doing 
anything else?

Another interesting suggestion was from 
<http://www.silicondefense.com/pipermail/snortsnarf-users/2002-March/000261.html>, 
but I'm unclear on how to actually do this.  Any pointers?

"there were huge savings to be had in caching dns answers (if you use that 
option), and especially in storing and sorting the alert lists."

Thanks!

Jason Falciola
Information Security Analyst
IBM Managed Security Services
[email protected]