Re: Bro Logs Ingestion

There are some good patterns here. We observed that it helps a lot to just
ship logs from each NSM sensor as soon as data is collected, with minimal,
if any, processing. That's why we ship logs (with syslog-ng) to a RabbitMQ
instance, via AMQPS. No extra processing is done there, it's just
buffering. We then have a set of python workers fetching messages from
Rabbit and doing all of the processing.

No Kafka here but just simple solutions and avoiding any processing on
endpoints.



On Sun, May 5, 2019 at 7:12 PM Mustafa Qasim <[email protected]> wrote:

> the biggest reason is absorbing back pressure from logstash or other
> ingesting tools. In past the back pressure from logstash would cause CPU
> spikes on the originating endpoint.
>
> second, we can write programs to clean, modify and enrich data before
> throwing at the ingesting tools making our log processing pipelines
> indipendedent. Giving us flexibility of migrating from logstash to Humio or
> Splunk and not worry about wasting all the efforts you put into logstash
> pipelines.
>
> ------
> *Mustafa Qasim*
> PGP: C57E0A7C
> <http://pgp.mit.edu/pks/lookup?op=get&search=0x0A9C8A5EC57E0A7C>
>
>
> On Mon, May 6, 2019 at 7:08 AM David Decker <[email protected]> wrote:
>
>> Sorry beginner question here:
>>
>> But I know you can ingest logs into Splunk, and Elastic Search.
>>
>> So I know SecurityOnion has an ELK stack and it looks like they get sent
>> right to Logstash - ES - Kibana
>>
>> RockNSM looks almost the same but it has a stop off at Kafka before
>> forwarding to Logstash.
>>
>> Trying to figure out is there a benefit for Kafka.
>>
>> Also looking at using Splunk instead of ES.
>> I know I can use the TA and monitor the logs from splunk, but would it be
>> better to monitor from Kafka?
>>
>> I guess I need to understand more of how Kafka fits.
>>
>> Thanks
>> Dave
>> _______________________________________________
>> Zeek mailing list
>> [email protected]
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> _______________________________________________
> Zeek mailing list
> [email protected]
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

_______________________________________________
Zeek mailing list
[email protected]
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek