Re: Bro Logs Ingestion

also it's a good security practice to offload logs from endpoints ASAP.
During incidnet endpoints can go offline or the data can be impacted
especially if the endpoints are compromised.

------
*Mustafa Qasim*
PGP: C57E0A7C
<http://pgp.mit.edu/pks/lookup?op=get&search=0x0A9C8A5EC57E0A7C>


On Mon, May 6, 2019 at 12:17 PM Michał Purzyński <[email protected]>
wrote:

> There are some good patterns here. We observed that it helps a lot to just
> ship logs from each NSM sensor as soon as data is collected, with minimal,
> if any, processing. That's why we ship logs (with syslog-ng) to a RabbitMQ
> instance, via AMQPS. No extra processing is done there, it's just
> buffering. We then have a set of python workers fetching messages from
> Rabbit and doing all of the processing.
>
> No Kafka here but just simple solutions and avoiding any processing on
> endpoints.
>
>
>
> On Sun, May 5, 2019 at 7:12 PM Mustafa Qasim <[email protected]> wrote:
>
>> the biggest reason is absorbing back pressure from logstash or other
>> ingesting tools. In past the back pressure from logstash would cause CPU
>> spikes on the originating endpoint.
>>
>> second, we can write programs to clean, modify and enrich data before
>> throwing at the ingesting tools making our log processing pipelines
>> indipendedent. Giving us flexibility of migrating from logstash to Humio or
>> Splunk and not worry about wasting all the efforts you put into logstash
>> pipelines.
>>
>> ------
>> *Mustafa Qasim*
>> PGP: C57E0A7C
>> <http://pgp.mit.edu/pks/lookup?op=get&search=0x0A9C8A5EC57E0A7C>
>>
>>
>> On Mon, May 6, 2019 at 7:08 AM David Decker <[email protected]> wrote:
>>
>>> Sorry beginner question here:
>>>
>>> But I know you can ingest logs into Splunk, and Elastic Search.
>>>
>>> So I know SecurityOnion has an ELK stack and it looks like they get sent
>>> right to Logstash - ES - Kibana
>>>
>>> RockNSM looks almost the same but it has a stop off at Kafka before
>>> forwarding to Logstash.
>>>
>>> Trying to figure out is there a benefit for Kafka.
>>>
>>> Also looking at using Splunk instead of ES.
>>> I know I can use the TA and monitor the logs from splunk, but would it
>>> be better to monitor from Kafka?
>>>
>>> I guess I need to understand more of how Kafka fits.
>>>
>>> Thanks
>>> Dave
>>> _______________________________________________
>>> Zeek mailing list
>>> [email protected]
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>> _______________________________________________
>> Zeek mailing list
>> [email protected]
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>

_______________________________________________
Zeek mailing list
[email protected]
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek