Re: log delays and logger CPUs

So there's a couple of things you could do.

1. Indeed using pin_cpus for the manager or the logging will result in all
threads being pinned to the same CPU, definitely not ideal and worth
filling a bug

2. You could (per Justin's idea from Zeek Week) use the path func and write
a script that will split biggest log files into two or more files - and
each will get a separate thread

https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_dns.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_http.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_intel.zeek

3. And maybe implement some filtering for traffic you don't care for, if
there is any, examples here

https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_files.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_mysql.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_ssl.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_x509.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_conn.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_input.zeek


On Wed, Oct 16, 2019 at 10:17 AM Justin Azoff <[email protected]> wrote:

>
>
> On Wed, Oct 16, 2019 at 12:23 PM Chris Herdt <[email protected]> wrote:
>
>> I have a cluster running Bro 2.6.4. One host runs a manager and logger, 8
>> other hosts run proxy and worker nodes.
>>
>> Lately the logger node has not been able to keep up with the logs, and
>> I've noticed that the most recent entries in the current/conn.log are
>> significantly delayed (I've seen delays as high as 90 minutes).
>>
>> The logger process has maxed out CPU usage on core 1. The node.cfg file
>> specifies 8 CPU cores (all on the same NUMA node as the NVMe drive where
>> the logs are written):
>>
>> [logger]
>> type=logger
>> host=bromanager-01.umn.edu
>> pin_cpus=1,3,5,7,9,11,13,15
>>
>> `broctl nodes` shows that only 1 CPU core is pinned:
>>
>
>
>> Can pin_cpus be used with a logger node? Any other suggestions for
>> improving logger performance?
>>
>
> Ah.. pin_cpus is more intended to work with multiple worker processes.
> It's definitely doing the wrong thing in your case.  It's pinning the first
> logger (out of one total) to core 1, and then it never uses 3,5,7....
> You're better off removing that setting.  It should run fine across all
> cores, even with the numa hit.. the volume of logs that would go across the
> numa bus would only be a small fraction of the total bandwidth.
>
> You could still pin it to those cores, but you'd have to do it manually
> using taskset for now.  This is probably something that could be fixed in
> broctl(zeekctl) to better handle pin_cpus option when only a single process
> is being started.
>
> --
> Justin
> _______________________________________________
> Zeek mailing list
> [email protected]
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

_______________________________________________
Zeek mailing list
[email protected]
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek