Re: remote sensor limit

Steve
I severely doubt its number of sensors. I have had several hundred
sensors reporting to a single prelude-manager instance. my only issues
were time delays during alert floods. however i never had an issue
with heartbeats being dropped.

Here are some question and some suggestions ment to help you provide
more useful debugging information to help track down the source of the
issue.

1) (question) what is the sensor.
for example I wrote a custom sensor which had a similar issue using
the Perl binding as an agent for snmptrapd using its embedded libperl
implementation. however independent daemons I wrote using almost the
same code did not have the issue. Also the snort sensor is very
different in its behavior than the Prelude-LML  because of different
design philosophies about attempting to reconnect after a tcp failure.
while most sensors will try to reconnect after a connection issue the
developers of snort preferr not to try to reconnect to remote
destinations for data because snort generally in most implementations
pushes the limits of the CPU (or core) its running on and the remote
reporting is considered to be a "nice to have" but not to be relied
upon feature unless your infrastructure functions flawlessly( which no
ones does 100% of the time).

2) (suggestion ) You used an ngrep to track the issue have you
captures a tcpdump that would be very useful because it might provide
better insight into whats going on' however I would understand if you
didnt want to porvide a tcpdump for security reasons especially on a
public forum.

3) (question and suggestion ) There was a request I made several
months ago which was implemented that might be helpfull. in the
failover mechanisim if you have enabled it there is a file it records
of uninsertable alerts. The reason why I requested this is I had alert
which were exceeding the maximum field length of some of the database
fields (cfengine in combination with the sudo rules was the primary
culprit ). The way this issue manifests itself is it fails to insert
which makes prelude think that there is an issue with the database
instead of the data the result is it took a long time before it would
finally drop the alert and at the same time it would drop a few
others. the change i requested detects these uninsertable alerts an
sets them aside into a file.if  you don't have failover on the
database enabled please do so if it produces a file containing alerts
that it can not be inserted that file might help us explain the issue.

4) (question) has your database produced any insert errors in the
logs? if so this may also explain the cause of the issue.
keep in mind that an alert (including heartbeats) result in several
inserts in an ACID transaction ("Atomicity, Consistency, Isolation,
Durability" http://en.wikipedia.org/wiki/ACID .  in short words it
requires a commit at the end) and if one insert fails in an ACID
transaction all of modification within the transactions are rolled
back (undone  http://en.wikipedia.org/wiki/Rollback_(SQL) )

5) (suggestion ) An strace ( http://sourceforge.net/projects/strace/
)of prelude manager might also be helpfull



On Tue, Sep 16, 2008 at 9:27 AM, Yoann Vandoorselaere
<[email protected]> wrote:
> Hello Steve,
>
> Le mardi 09 septembre 2008 à 14:06 -0500, Steve Williamson a écrit :
>> I have 11 remote sensors and all of of them regularly check in and
>> show active status in Prewikka.  Except for the 11th one that checks
>> in.  It eventually shows missing. and never returns.  A check at the
>> sensor shows all programs running and working fine.  (Prelude-manager,
>> prelude-lml and snort.  Some even have OSSEC)
>>
>> I removed all alerts, and heartbeats.  Then I stopped all remote
>> sensors and restarted them in a different order.   The 11th one to
>> check in eventually shows missing, and never returns.
>>
>> I checked to see if they were sending out heartbeats via a ngrep
>> packet capture and the packets did arrive on the receiving prelude-
>> manager network.
>
> Could you enable the prelude-manager 'debug' plugin, and make sure
> heartbeat of the 11th agent are printed?
>
> Additionally, can you confirm the issue happen randomly, independently
> of the type of sensor?
>
> --
> Yoann Vandoorselaere | Responsable R&D / CTO | PreludeIDS Technologies
> Tel: +33 (0)8 70 70 21 58                  Fax: +33(0)4 78 42 21 58
> http://www.prelude-ids.com
>
> _______________________________________________
> Prelude-user site list
> [email protected]
> http://lists.prelude-ids.org/mailman/listinfo/prelude-user
_______________________________________________
Prelude-user site list
[email protected]
http://lists.prelude-ids.org/mailman/listinfo/prelude-user