Re: Prelude-LML 100%CPU usage after ~3-5 minutes post startup

yes ive seen this happen but only in a very specific instance.
while tailing a more the ten log files that each grew to over 10GB and 
the lml instance had more then 1000 rules.
one prelude-lml does not need python its complaining about messages in 
the log it cant parse properly that happen to be python errors coming 
from prewikka.

what the error message is talking about is that a messages
 "TypeError: not indexable", ' File 
"/usr/share/prewikka/cgi-bin/prewikka.cgi", line 83, in <module>', ' 
raise TypeError, "not indexable"', ' File "/usr/lib/python2.6/cgi.py", 
line 583, in keys'
do not match the prefix-regex in the prelude-lml.conf file.

it looks like the file does not have the standard syslog headers also it 
looks like you have severe issues with your prewikka install.

1) first things first fix your prewikka install

2 fix the prefix-regex in the prelude-lml.conf

2) One way to make it more efficient is to have syslog feed it via UDP 
directly because tailing a file and keeping track of the last read line 
combine with the file rotation detection is heavy not to mention the 
fact that the lml records and updates this information in a file so it 
can recover from a process failure. under light load the overhead is 
minimal however under heavy load it becomes noticeable and can even max 
out your CPU due mostly to IO wait and check sum calculations.

3) if you have a lot of rules comment out the ones that you don't use 
add drop rules at the top of the first rule file processed to 
automatically drop messages you know are spam like there is a common one 
on RHEL from pam that logs  a login by cron for every user that has a 
populated crontab every minute when it polls every user crontab to see 
if a cron job needs to be run.


chances are solution 1 and 2 will fix your issues.

I also highly recommend solution 3 its only down side is since 
prelude-lml only support udp syslog and not tcp syslog you cant use a 
more advanced syslog server like rsyslog with disk queuing to ensure 
delivery so if your box gets over too over loaded or your lml process 
goes down you will loose messages.
by the way ive had over 600 servers sending there syslog messages to one 
syslog server with a lml process receiving a relayed feed over the loop 
back from the log servers syslog process, when i did that i never missed 
a message and the cpu load of the lml process never went over 5%
 


Tim Arneaud wrote:
> Hi All,
> I'm wondering if anyone else is experiencing this behavior and/or has a
> pointer for me to check.
>
> As the subject states, I'm running a fairly default installation of
> Prelude-LML that maxes out the CPU about 3-5 minutes after daemon
> startup. 
>
> Prelude-lml successfully registers with the Manager and events do appear
> to be updating the database prior to hitting %100 utilization. 
>
> However after a few minutes it becomes completely unresponsive and also
> is shown as offline via Manager/Prewikka.
>
> There don't appear to be instances of others seeing this behavior (via
> google and list searches) that I can find.
>
> However I've duplicated this on two separate systems (Ubuntu/Debian)
> with minimal configuration (per the documentation) and am surprised that
> others are not seeing this.
>
> *Prior* to maxing 100% CPU, my /var/log/messages logfile shows relevant
> entries ->
>
> Nov 23 17:04:35 host prelude-lml: WARNING: no appropriate format defined
> for log entry: '  File "/usr/share/prewikka/cgi-bin/prewikka.cgi", line
> 83, in <module>'.
> Nov 23 17:04:35 host prelude-lml: WARNING: no appropriate format defined
> for log entry: '    request.init()'.
> Nov 23 17:04:35 host prelude-lml: WARNING: no appropriate format defined
> for log entry: '  File "/usr/lib/python2.6/cgi.py", line 583, in keys'.
> Nov 23 17:04:35 host prelude-lml: WARNING: no appropriate format defined
> for log entry: '    raise TypeError, "not indexable"'.
> Nov 23 17:04:35 host prelude-lml: WARNING: no appropriate format defined
> for log entry: 'TypeError: not indexable'.
>
> Others clearly have this working so I'm at a loss as to how my fairly
> basic/default configuration would require significant python library
> dependency changes.
>
> *- Is anyone else experiencing this?
> *- Can anyone point me in a new direction that I should look at?
>
> Any information or pointers are most welcome, as I'm very keen to get
> this working and giving Prelude a full run :D
>
> Thanks in advance!
>
> Kind Regards,
> Tim Arneaud
> The University of Melbourne, Carlton
> Victoria 3053, Australia
>
>
> _______________________________________________
> Prelude-user site list
> [email protected]
> http://lists.prelude-ids.org/mailman/listinfo/prelude-user
>   

_______________________________________________
Prelude-user site list
[email protected]
http://lists.prelude-ids.org/mailman/listinfo/prelude-user