Re: Increased memory usage by Zeek..

Hmm, I will disable the SMB analyzer in local.bro and see if it helps..
Thanks Jon! :-)


On Fri, Sep 6, 2019 at 12:58 PM Jon Siwek <[email protected]> wrote:

> Biggest changes from 2.5.x to 2.6.x that I can recall are (1)
> switching remote communication to use the new Broker library and (2)
> enabling SMB analysis by default.
>
> Had you manually enabled SMB in your previous 2.5.x deployment?  If
> not, you could see if disabling it helps:
>
>     redef Analyzer::disabled_analyzers += { Analyzer::ANALYZER_SMB };
>
> That's my first guess because we've recently seen/suspected (but not
> yet fixed) some state management issues in the SMB analysis scripts
> that might explain high memory usage.
>
> - Jon
>
> On Fri, Sep 6, 2019 at 8:46 AM fatema bannatwala
> <[email protected]> wrote:
> >
> > Hi All,
> >
> > Couple of months ago I upgraded the Zeek cluster from 2.5 to 2.6.1
> (compiled with the jemalloc support).
> > I have started seeing increased memory usage by the workers.
> >
> > I have two physical sensors, each running 18 Zeek worker processes LB by
> PF_RING.
> > Not loaded any custom scripts, just the basic scripts that are enabled
> by default in local.bro (also have misc/scan disabled).
> >
> > I just did a top on one of the boxes and here's the output (specially
> two Zeek processes -13632, 13611 using >10% memory which is ~11G)
> > Also, attaching a weekly available free memory graph for the system.
> >
> > Tasks: 455 total,   9 running, 443 sleeping,   0 stopped,   3 zombie
> > %Cpu(s): 18.3 us,  1.7 sy,  0.0 ni, 79.5 id,  0.0 wa,  0.0 hi,  0.4 si,
> 0.0 st
> > KiB Mem : 98783960 total, 32963660 free, 64807572 used,  1012728
> buff/cache
> > KiB Swap:  4194300 total,  3572200 free,   622100 used. 33221356 avail
> Mem
> >
> >   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+
> COMMAND
> > 13589 bro       20   0 3662052   3.4g  73340 R    90.4     3.6
> 1072:47    /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-5 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13533 bro       20   0 1847972   1.6g  73188 S    50.3     1.7
> 1098:05    /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-6 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13512 bro       20   0 1291260   1.1g  73052 S    49.7     1.1
> 1080:30    /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-1 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13628 bro       20   0 2347952   2.1g  73328 R    49.0     2.2
> 1109:31    /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-12 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13516 bro       20   0  973260 799176  72844 R  47.0     0.8    1036:29
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-3 local.bro broctl base/frameworks/cluster broctl/auto
> > 13539 bro       20   0 6374956   6.0g  73456 S    46.0     6.3
> 1147:08   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-2 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13591 bro       20   0  865952 726516  73020 S  44.7     0.7    1052:29
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-4 local.bro broctl base/frameworks/cluster broctl/auto
> > 13632 bro       20   0   12.2g  12.0g  73584 R      43.7    12.8
> 1068:17   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-15 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13540 bro       20   0 2146844   1.9g  73348 R    41.4     2.0
> 1149:38   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-7 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13611 bro       20   0   17.0g  16.7g  73404 S      39.7    17.8
> 1172:14   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-9 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13640 bro       20   0 2624300   2.1g  73328 S    39.7     2.3
> 1043:50   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-18 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13586 bro       20   0 3347044   3.1g  73468 S    39.1     3.2
> 1042:24   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-10 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13641 bro       20   0 2274788   2.0g  73424 R    39.1     2.2
> 1029:58   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-17 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13614 bro       20   0 1954780   1.7g  73188 S    38.4     1.8
>  995:00.54   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-13 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13627 bro       20   0 2756520   2.5g  73288 S    38.4     2.6
>  1035:18   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-14 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13638 bro       20   0 1206548 853056  72328 R  37.4    0.9   952:10.00
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-16 local.bro broctl base/frameworks/cluster broctl/auto
> > 13623 bro       20   0 8998324   2.1g  73284 S     37.1    2.2
>  1073:31   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-11 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13575 bro       20   0  871396 706148  73128 R    36.4   0.7
>  1028:30   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-8 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13336 bro       20   0  266244 133920  33388 S    12.6   0.1
>  400:27.62   /usr/local/bro/2.6.1/bin/bro -U .status -p broctl -p
> broctl-live -p local -p proxy-2 local.bro broctl base/frameworks/cluster
> broctl/auto
> >
> > Any suggestions?
> >
> > Thanks!
> > Fatema
> > _______________________________________________
> > Zeek mailing list
> > [email protected]
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>

_______________________________________________
Zeek mailing list
[email protected]
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek