Re: Prewikka performance

Yeah, performance with any reasonably-sized system is abysmal with the
free version. I gave up long ago with their CTO admitted to me in
writing that "the open source version is obviously not intended for an
enterprise deployment". They want you to upgrade to their commercial
offering which for a large network is priced comparably to market
leaders such as Q1 and ArcSight with nowhere near the functionality or
performance. I finally got their tech support folks to explain to me
what they used to sell as their "database accelerator" product (now
integrated into the main commercial offering) - it's simply an optimized
schema and database library that avoids doing thousands of multi-table
JOINs just to render a single page. The same things happens when trying
to ingest large numbers of sensor events into the database - the
preludedb library is *not* multithreaded so if you have several chatty
sensors feeding into it the INSERTs quickly become CPU-bound and more
cores will not help you (I was running my database and prewikka instance
on a 16-core machine with 64GB of RAM, and 15 cores sat idle 90% of the
time).

The gist of it is, they've deliberately crippled the open-source version
of the software by employing criminally stupid database logic in an
effort to force you into buying their incredibly overpriced commercial
solution. For a project that has so much potential, this is inexcusable.
It's like driving a ferarri with a lawnmower engine.

-- Robert Vineyard


On 8/6/2010 7:17 PM, Russell Fulton wrote:
> we have around 1 million events in a mysql database and response from prewikka is far from sparkling.  I too would be interested in knowing how well prelude scales.   Looking at the DB schema there are a lot of tables and I suspect that you end up doing a *lot* of joins for most queries.
> 
> Russell
> 
> 
> On 6/08/2010, at 9:23 PM, Honza Mach wrote:
> 
>> Hi everybody,
>>
>> does anyone has any information about prewikka performance to share with
>> me?
>>
>> Regards
>>
>> ---
>> Jan Mach
>> Cesnet, z.s.p.o.
>> www.cesnet.cz
>>
>> From: Honza Mach <[email protected]>
>> Date: 30 July 2010 1:40:43 AM NZST
>> To: <[email protected]>
>> Subject: [prelude-user] Prewikka performance
>> Reply-To: <[email protected]>
>>
>>
>> Hi everybody,
>>
>> I was wondering, if anyone could give me some information regarding
>> Prewikka performance.
>>
>> I have recently installed Prelude on Debian Lenny system and connected
>> some heavy sensors we have. After the weekend, the MySQL database
>> contained about 6 million entries and the Prewikka interface was
>> absolutely unusable - it could not even render the page after tens of
>> minutes. 
>>
>> My first guess was that the database has poor performance. I made some
>> tuning and optimization of InnoDB tables according to the tips here
>> http://www.wzdftpd.net/trac/wiki/Misc/PreludeTips and here
>> http://www.mysqlperformanceblog.com/2007/11/03/choosing-innodb_buffer_pool_size/
>>
>> , but with absolutely no positive results. Database responded normally
>> via the linux command line, incoming new entries were written in real
>> time and did not get stalled anywhere.
>>
>> So it brought me to the Prewikka. I disconnected some of the heaviest
>> senzors, turned on the MySQL logging and discovered, that for Prewikka
>> to display the alerts listing page with about 2000 entries in the
>> database it has to do about 2000 QUERIES to the database !!! This is
>> absolutely out of my expectations and no wonder that it was impossible
>> to render the alert listing page with 6 million entries in DB. Prewikka
>> reads the entries one by one.
>>
>> Have you any experience with using Prewikka with large amount of data?
>> Are there any options to optimize the queries, so the Prewikka will be
>> usable even for huge amounts of data? Or are there any lightweight
>> alternatives to Prewikka? Have you any experience with Piwi? It seems
>> dead to me.
>>
>> I have considered the pre-filtering of the incoming data in the
>> prelude-manager, so the database will not contain so much of junk, but
>> still, I would like to keep as many of the data as possible.
>>
>> Perhaps this is not the issue of the Prewikka, but instead of the
>> underlaying libpreludedb library.
>>
>> I have to admit, that we are testing the Prelude on the lower-end system
>> (Dell PowerEdge 2650, 4GB RAM), so I did not expected flashing speed,
>> but this in my opinion is too much.
>>
>> Thanks for your time in advance
>>
>> Jan Mach
>> Cesnet, z.s.p.o.
>> www.cesnet.cz
>> _______________________________________________
>> Prelude-user site list
>> [email protected]
>> http://lists.prelude-technologies.com/mailman/listinfo/prelude-user
>>
>> _______________________________________________
>> Prelude-user site list
>> [email protected]
>> http://lists.prelude-technologies.com/mailman/listinfo/prelude-user
> 
> _______________________________________________
> Prelude-user site list
> [email protected]
> http://lists.prelude-technologies.com/mailman/listinfo/prelude-user
_______________________________________________
Prelude-user site list
[email protected]
http://lists.prelude-technologies.com/mailman/listinfo/prelude-user