Re: performance testing

Thanks Scott.

Are you using the commercial version of PREWikka as well?

My thoughts were that the bottleneck was in the interface that PREWikka
is taking to the PreludeDB. I'm actually able to query the PreludeDB
very quickly if I do it directly or even via scripts/code that uses
libprelude. There seems to be something about the way  PREWikka 
structures it's queries for the overview pegs the Postgres database at
99% cpu for a while.

Here is what I've found w/ our particular setup...

If there are about 1000 alerts in the database I can dump the entire
thing through PREWikka in a few seconds, and can perform just about any
query through the PREWikka interface quickly.

If there are about 5000 alerts in the DB (our IDS's that report to
Prelude often generate that amount in a bad day). I can query for about
200 alerts through the general interface before PREWikka times out.
Specific queries (i.e. clicking on the alert subjects or querying via a
source or dest) return quickly (under 5 seconds). After 5000 alerts
things become very slow. Around 10k entries, we can only return about
50-60 alerts before the database times out, and those returns are slow.
Performance seems steady at that rate until a few million alerts. 

As per our sensors, only our Snort sensors and a custom log analysis
sensor report to Prelude atm. Most of our "bad days", are usually when I
upload/enable some new Emerging Threats rules or add some other rules
that have false positives.

I have noticed that if there are many alerts being generated at that
time period, performance slows down significantly (which makes sense).

With the XLR plugin and your setup how long does it take to query for:
24 hours of data?
48 hours of data?
72 hours of data?
96 hours of data?





ScottO wrote:
> Not exactly what you are looking for, but we quickly needed to move
> off of the open source prelude db, as prewikka was getting sluggish to
> refresh pages, etc.  We had < 50 sensors, and under a million total
> alerts, maybe 10k/day.  Default page refreshes would take some time to
> refresh on occassion.
>
> On the XLR plugin we have noticed no problems now with almost 7million
> alerts and ~75-100 sensors, this being data collected over the last
> ~18 months.
>
>
> On Wed, Jan 14, 2009 at 4:07 PM, Aaron Shelmire <[email protected]
> <mailto:[email protected]>> wrote:
>
>     I was wondering if anyone has done any performance testing w/ prelude
>     and prewikka; and if they would be willing to share those results?
>
>     I'm most interested in how many alerts are in the database until users
>     begin to see the prewikka interface become slow, become unusable,
>     etc...
>     and if the issues I've encountered are typical of others.
>
>     -aaron
>     _______________________________________________
>     Prelude-user site list
>     [email protected] <mailto:[email protected]>
>     http://lists.prelude-ids.org/mailman/listinfo/prelude-user
>
>

_______________________________________________
Prelude-user site list
[email protected]
http://lists.prelude-ids.org/mailman/listinfo/prelude-user