Re: performance testing

On Thu, Jan 15, 2009 at 2:33 PM, Aaron Shelmire <[email protected]> wrote:

>  Thanks Scott.
>
> Are you using the commercial version of PREWikka as well?
>

Yes.

>
>
> My thoughts were that the bottleneck was in the interface that PREWikka is
> taking to the PreludeDB. I'm actually able to query the PreludeDB very
> quickly if I do it directly or even via scripts/code that uses libprelude.
> There seems to be something about the way  PREWikka  structures it's queries
> for the overview pegs the Postgres database at 99% cpu for a while.
>

I'm pretty sure we've used the open source Prewikka w/XLR in testing and
seen good results, but can't remember as that was a while ago.


>
> Here is what I've found w/ our particular setup...
>
> If there are about 1000 alerts in the database I can dump the entire thing
> through PREWikka in a few seconds, and can perform just about any query
> through the PREWikka interface quickly.
>
> If there are about 5000 alerts in the DB (our IDS's that report to Prelude
> often generate that amount in a bad day). I can query for about 200 alerts
> through the general interface before PREWikka times out. Specific queries
> (i.e. clicking on the alert subjects or querying via a source or dest)
> return quickly (under 5 seconds). After 5000 alerts things become very slow.
> Around 10k entries, we can only return about 50-60 alerts before the
> database times out, and those returns are slow. Performance seems steady at
> that rate until a few million alerts.
>
>

>
> As per our sensors, only our Snort sensors and a custom log analysis sensor
> report to Prelude atm. Most of our "bad days", are usually when I
> upload/enable some new Emerging Threats rules or add some other rules that
> have false positives.
>
> I have noticed that if there are many alerts being generated at that time
> period, performance slows down significantly (which makes sense).
>
> With the XLR plugin and your setup how long does it take to query for:
> 24 hours of data?
> 48 hours of data?
> 72 hours of data?
> 96 hours of data?
>
>
this was only a quick one run of each, not sure if the previous query caches
or sways the data in any way:

24 == 6s
48 == 6s
72 == 14s
96 == 5s



>
>
>
> ScottO wrote:
>
> Not exactly what you are looking for, but we quickly needed to move off of
> the open source prelude db, as prewikka was getting sluggish to refresh
> pages, etc.  We had < 50 sensors, and under a million total alerts, maybe
> 10k/day.  Default page refreshes would take some time to refresh on
> occassion.
>
> On the XLR plugin we have noticed no problems now with almost 7million
> alerts and ~75-100 sensors, this being data collected over the last ~18
> months.
>
>
> On Wed, Jan 14, 2009 at 4:07 PM, Aaron Shelmire <[email protected]> wrote:
>
>> I was wondering if anyone has done any performance testing w/ prelude
>> and prewikka; and if they would be willing to share those results?
>>
>> I'm most interested in how many alerts are in the database until users
>> begin to see the prewikka interface become slow, become unusable, etc...
>> and if the issues I've encountered are typical of others.
>>
>> -aaron
>> _______________________________________________
>> Prelude-user site list
>> [email protected]
>> http://lists.prelude-ids.org/mailman/listinfo/prelude-user
>>
>
>
>
_______________________________________________
Prelude-user site list
[email protected]
http://lists.prelude-ids.org/mailman/listinfo/prelude-user