Re: Zeek Digest, Vol 165, Issue 6
James Offer <[email protected]> Mon, 6 Jan 2020 15:07:42 -0500
| Newsgroups | gmane.comp.security.detection.bro |
|---|---|
| Message-ID | <CA+FQEy6RmQkeOzg_XoM98t7zsB=ZMmA8brkYA4rcF2GK=B5kPg@mail.gmail.com> |
--===============1909781725== Content-Type: multipart/alternative; boundary="000000000000fb52c4059b7e36c8" --000000000000fb52c4059b7e36c8 Content-Type: text/plain; charset="UTF-8" All, This is helpful. Other than my sysadmins' preference, is there any reason to choose one or the other, between rsyslog and syslog-ng? Thanks, Jim On Mon, Jan 6, 2020 at 3:00 PM <[email protected]> wrote: > Send Zeek mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Zeek digest..." > > > Today's Topics: > > 1. Re: Zeek with ELK (Darren S.) > 2. Re: Zeek with ELK (duhang) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 5 Jan 2020 17:40:02 -0700 > From: "Darren S." <[email protected]> > Subject: Re: [Zeek] Zeek with ELK > To: sec-x sec-x <[email protected]> > Cc: [email protected] > Message-ID: > < > CAKVSOJWSnfCrzPyreChQtVVU5LS1yEKO3ufV3S2ZdyJXwj4-9w@mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > I read OP question as "I have Zeek running on FreeBSD, what is a > sensible option for shipping logs from the sensor to an Elastic > Stack?" Apologies if it's the wrong read. > > In that case I wouldn't want to install either the whole stack nor > even Logstash on the sensor as it alone tends to consume an excessive > amount of memory, not what you want on a sensor. Filebeat (a small > footprint data collector/shipper) is the way to go if you're shipping > remotely. > > If Filebeat isn't an option on the platform, maybe explore Fluent Bit: > > https://github.com/fluent/fluent-bit > https://fluentbit.io/ > > Fluent Bit can output directly to Elasticsearch: > https://fluentbit.io/documentation/0.14/output/elasticsearch.html > > Even a Fluentd can run with typically lower memory consumption than > Logstash, so perhaps worth exploring both/either: > > https://github.com/fluent/fluentd > https://www.fluentd.org/ > > Fluentd can also output to Elasticsearch: > https://docs.fluentd.org/output/elasticsearch > > There are other options for shippers too, such as Syslog-ng: > > https://www.syslog-ng.com/community/b/blog/posts/logging-to-elasticsearch-made-simple-with-syslog-ng > > - Darren > > On Sun, Jan 5, 2020 at 9:11 AM Michael Shirk <[email protected]> > wrote: > > > > You should be able to fire up Elastic, Logstash and Kibana on FreeBSD, > using recommend Logstash configs to read in the log files from the file > system. I can check about the Filebeat port to see if that can be updated > or fixed. > > > > I myself just use the CLI tools but have been working on something "Not > Java" to ingest log files into other than Splunk. > > > > > > -- > > Michael Shirk > > Daemon Security, Inc. > > https://www.daemon-security.com > > > > On Sun, Jan 5, 2020, 10:35 sec-x sec-x <[email protected]> wrote: > >> > >> Hi, > >> > >> I recently used zeek IDS on FreeBSD 12.1 - Default Policy (GetTraffic > >> from TAP on the network) and i want to send all the logs to ELK in > >> realtime. > >> > >> I saw Filebeat ports on BSD is old and has problems. > >> > >> How can i send the logs from the BSD to the Elastic (what is the > >> correct/best way)? > >> > >> > >> Thanks, > >> > >> CM. > >> _______________________________________________ > >> Zeek mailing list > >> [email protected] > >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > > > _______________________________________________ > > Zeek mailing list > > [email protected] > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > > > -- > Darren Spruell > [email protected] > > > ------------------------------ > > Message: 2 > Date: Mon, 6 Jan 2020 09:52:52 +0800 > From: duhang <[email protected]> > Subject: Re: [Zeek] Zeek with ELK > To: sec-x sec-x <[email protected]> > Cc: [email protected] > Message-ID: > < > CAG+yijM94rhd5m9PifrbnEAf1yRii-N4aWA8-qfDJzCGnr9u9Q@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > You can try rsyslog imfile module to send logs to logstash. The following > is my configuration. > > $ModLoad imfile > $InputFileName /usr/local/bro/logs/current/dns.log > $InputFileTag dns: > $InputFileStateFile stat-dns > $InputFileSeverity info > $InputFileFacility local2 > $InputRunFileMonitor > > $SystemLogRateLimitInterval 0 > $SystemLogRateLimitBurst 0 > $MaxMessageSize 64k > > sec-x sec-x <[email protected]> ?2020?1?5??? ??11:36??? > > > Hi, > > > > I recently used zeek IDS on FreeBSD 12.1 - Default Policy (GetTraffic > > from TAP on the network) and i want to send all the logs to ELK in > > realtime. > > > > I saw Filebeat ports on BSD is old and has problems. > > > > How can i send the logs from the BSD to the Elastic (what is the > > correct/best way)? > > > > > > Thanks, > > > > CM. > > _______________________________________________ > > Zeek mailing list > > [email protected] > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200106/91801d18/attachment-0001.html > > ------------------------------ > > _______________________________________________ > Zeek mailing list > [email protected] > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > > End of Zeek Digest, Vol 165, Issue 6 > ************************************ > -- Jim Offer Network Security Analyst Saint Joseph's University (610) 660-1573 --000000000000fb52c4059b7e36c8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">All,<div><br></div><div>This is helpful. Other than my sys= admins' preference, is there any reason to choose one or the other, bet= ween rsyslog and syslog-ng?</div><div><br></div><div>Thanks,</div><div>Jim<= /div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_a= ttr">On Mon, Jan 6, 2020 at 3:00 PM <<a href=3D"mailto:zeek-request@zeek= .org">[email protected]</a>> wrote:<br></div><blockquote class=3D"gm= ail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,= 204,204);padding-left:1ex">Send Zeek mailing list submissions to<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a><br> <br> To subscribe or unsubscribe via the World Wide Web, visit<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"http://mailman.ICSI.Berkeley.EDU/mai= lman/listinfo/zeek" rel=3D"noreferrer" target=3D"_blank">http://mailman.ICS= I.Berkeley.EDU/mailman/listinfo/zeek</a><br> or, via email, send a message with subject or body 'help' to<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a><br> <br> You can reach the person managing the list at<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a><br> <br> When replying, please edit your Subject line so it is more specific<br> than "Re: Contents of Zeek digest..."<br> <br> <br> Today's Topics:<br> <br> =C2=A0 =C2=A01. Re: Zeek with ELK (Darren S.)<br> =C2=A0 =C2=A02. Re: Zeek with ELK (duhang)<br> <br> <br> ----------------------------------------------------------------------<br> <br> Message: 1<br> Date: Sun, 5 Jan 2020 17:40:02 -0700<br> From: "Darren S." <<a href=3D"mailto:[email protected]" ta= rget=3D"_blank">[email protected]</a>><br> Subject: Re: [Zeek] Zeek with ELK<br> To: sec-x sec-x <<a href=3D"mailto:[email protected]" target=3D"_blan= k">[email protected]</a>><br> Cc: <a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a><br= > Message-ID:<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 <<a href=3D"mailto:CAKVSOJWSnfCrzPyreChQtVVU= [email protected]" target=3D"_blank">CAKVSOJWSnfCrz= [email protected]</a>><br> Content-Type: text/plain; charset=3D"UTF-8"<br> <br> I read OP question as "I have Zeek running on FreeBSD, what is a<br> sensible option for shipping logs from the sensor to an Elastic<br> Stack?" Apologies if it's the wrong read.<br> <br> In that case I wouldn't want to install either the whole stack nor<br> even Logstash on the sensor as it alone tends to consume an excessive<br> amount of memory, not what you want on a sensor. Filebeat (a small<br> footprint data collector/shipper) is the way to go if you're shipping<b= r> remotely.<br> <br> If Filebeat isn't an option on the platform, maybe explore Fluent Bit:<= br> <br> <a href=3D"https://github.com/fluent/fluent-bit" rel=3D"noreferrer" target= =3D"_blank">https://github.com/fluent/fluent-bit</a><br> <a href=3D"https://fluentbit.io/" rel=3D"noreferrer" target=3D"_blank">http= s://fluentbit.io/</a><br> <br> Fluent Bit can output directly to Elasticsearch:<br> <a href=3D"https://fluentbit.io/documentation/0.14/output/elasticsearch.htm= l" rel=3D"noreferrer" target=3D"_blank">https://fluentbit.io/documentation/= 0.14/output/elasticsearch.html</a><br> <br> Even a Fluentd can run with typically lower memory consumption than<br> Logstash, so perhaps worth exploring both/either:<br> <br> <a href=3D"https://github.com/fluent/fluentd" rel=3D"noreferrer" target=3D"= _blank">https://github.com/fluent/fluentd</a><br> <a href=3D"https://www.fluentd.org/" rel=3D"noreferrer" target=3D"_blank">h= ttps://www.fluentd.org/</a><br> <br> Fluentd can also output to Elasticsearch:<br> <a href=3D"https://docs.fluentd.org/output/elasticsearch" rel=3D"noreferrer= " target=3D"_blank">https://docs.fluentd.org/output/elasticsearch</a><br> <br> There are other options for shippers too, such as Syslog-ng:<br> <a href=3D"https://www.syslog-ng.com/community/b/blog/posts/logging-to-elas= ticsearch-made-simple-with-syslog-ng" rel=3D"noreferrer" target=3D"_blank">= https://www.syslog-ng.com/community/b/blog/posts/logging-to-elasticsearch-m= ade-simple-with-syslog-ng</a><br> <br> - Darren<br> <br> On Sun, Jan 5, 2020 at 9:11 AM Michael Shirk <<a href=3D"mailto:shirkdog= [email protected]" target=3D"_blank">[email protected]</a>> wrote:<br> ><br> > You should be able to fire up Elastic, Logstash and Kibana on FreeBSD,= using recommend Logstash configs to read in the log files from the file sy= stem. I can check about the Filebeat port to see if that can be updated or = fixed.<br> ><br> > I myself just use the CLI tools but have been working on something &qu= ot;Not Java" to ingest log files into other than Splunk.<br> ><br> ><br> > --<br> > Michael Shirk<br> > Daemon Security, Inc.<br> > <a href=3D"https://www.daemon-security.com" rel=3D"noreferrer" target= =3D"_blank">https://www.daemon-security.com</a><br> ><br> > On Sun, Jan 5, 2020, 10:35 sec-x sec-x <<a href=3D"mailto:center.mn= [email protected]" target=3D"_blank">[email protected]</a>> wrote:<br> >><br> >> Hi,<br> >><br> >> I recently used zeek IDS on FreeBSD 12.1 - Default Policy (GetTraf= fic<br> >> from TAP on the network) and i want to send all the logs to ELK in= <br> >> realtime.<br> >><br> >> I saw Filebeat ports on BSD is old and has problems.<br> >><br> >> How can i send the logs from the BSD to the Elastic (what is the<b= r> >> correct/best way)?<br> >><br> >><br> >> Thanks,<br> >><br> >> CM.<br> >> _______________________________________________<br> >> Zeek mailing list<br> >> <a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</= a><br> >> <a href=3D"http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek"= rel=3D"noreferrer" target=3D"_blank">http://mailman.ICSI.Berkeley.EDU/mail= man/listinfo/zeek</a><br> ><br> > _______________________________________________<br> > Zeek mailing list<br> > <a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a><b= r> > <a href=3D"http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" rel= =3D"noreferrer" target=3D"_blank">http://mailman.ICSI.Berkeley.EDU/mailman/= listinfo/zeek</a><br> <br> <br> <br> -- <br> Darren Spruell<br> <a href=3D"mailto:[email protected]" target=3D"_blank">phatbuckett@gmai= l.com</a><br> <br> <br> ------------------------------<br> <br> Message: 2<br> Date: Mon, 6 Jan 2020 09:52:52 +0800<br> From: duhang <<a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a>><br> Subject: Re: [Zeek] Zeek with ELK<br> To: sec-x sec-x <<a href=3D"mailto:[email protected]" target=3D"_blan= k">[email protected]</a>><br> Cc: <a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a><br= > Message-ID:<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 <<a href=3D"mailto:CAG%2ByijM94rhd5m9PifrbnE= [email protected]" target=3D"_blank">CAG+yijM94rh= [email protected]</a>><br> Content-Type: text/plain; charset=3D"utf-8"<br> <br> You can try rsyslog imfile module to send logs to logstash. The following<b= r> is my configuration.<br> <br> $ModLoad imfile<br> $InputFileName /usr/local/bro/logs/current/dns.log<br> $InputFileTag dns:<br> $InputFileStateFile stat-dns<br> $InputFileSeverity info<br> $InputFileFacility local2<br> $InputRunFileMonitor<br> <br> $SystemLogRateLimitInterval 0<br> $SystemLogRateLimitBurst 0<br> $MaxMessageSize 64k<br> <br> sec-x sec-x <<a href=3D"mailto:[email protected]" target=3D"_blank">c= [email protected]</a>> ?2020?1?5??? ??11:36???<br> <br> > Hi,<br> ><br> > I recently used zeek IDS on FreeBSD 12.1 - Default Policy (GetTraffic<= br> > from TAP on the network) and i want to send all the logs to ELK in<br> > realtime.<br> ><br> > I saw Filebeat ports on BSD is old and has problems.<br> ><br> > How can i send the logs from the BSD to the Elastic (what is the<br> > correct/best way)?<br> ><br> ><br> > Thanks,<br> ><br> > CM.<br> > _______________________________________________<br> > Zeek mailing list<br> > <a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a><b= r> > <a href=3D"http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" rel= =3D"noreferrer" target=3D"_blank">http://mailman.ICSI.Berkeley.EDU/mailman/= listinfo/zeek</a><br> ><br> -------------- next part --------------<br> An HTML attachment was scrubbed...<br> URL: <a href=3D"http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments= /20200106/91801d18/attachment-0001.html" rel=3D"noreferrer" target=3D"_blan= k">http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200106/918= 01d18/attachment-0001.html</a> <br> <br> ------------------------------<br> <br> _______________________________________________<br> Zeek mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a><br> <a href=3D"http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" rel=3D"n= oreferrer" target=3D"_blank">http://mailman.ICSI.Berkeley.EDU/mailman/listi= nfo/zeek</a><br> <br> <br> End of Zeek Digest, Vol 165, Issue 6<br> ************************************<br> </blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"= class=3D"gmail_signature"><div dir=3D"ltr"><div><div dir=3D"ltr"><span sty= le=3D"font-family:Arial,sans-serif;color:black">Jim Offer<br>Network Securi= ty Analyst<br>Saint Joseph's University<br>(610) 660-<span>1573</span><= /span></div></div></div></div> --000000000000fb52c4059b7e36c8-- --===============1909781725== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Zeek mailing list [email protected] http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek --===============1909781725==--