Re: State of p0f support

@Michal,

That's a really good suggestion!  Latest p0f is from 2016, so it's not that
maintained anyway I guess.  The thing I noticed for software.log is that
the OS info gets logged only via software calls from apps like
Firefox/Chrome/etc; it would be nice to not rely on this.

Thanks,

On Mon, Jun 17, 2019 at 9:22 PM <[email protected]> wrote:

> Send Zeek mailing list submissions to
>         [email protected]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> or, via email, send a message with subject or body 'help' to
>         [email protected]
>
> You can reach the person managing the list at
>         [email protected]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Zeek digest..."
>
>
> Today's Topics:
>
>    1. State of p0f support (Robin Sommer)
>    2. Re: State of p0f support (Micha? Purzy?ski)
>    3. Re: Hui Lin_DNP3 analyzer not working in current version of
>       zeek (Jon Siwek)
>    4. Re: Hui Lin_DNP3 analyzer not working in current version of
>       zeek (Hui Lin (Hugo))
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 17 Jun 2019 13:45:27 -0700
> From: Robin Sommer <[email protected]>
> Subject: [Zeek] State of p0f support
> To: zeek <[email protected]>
> Message-ID: <[email protected]>
> Content-Type: text/plain; charset=us-ascii
>
> Looking for some input here.
>
> Zeek has provided support for passive OS fingerprinting for a long
> time through p0f. However, we are using using a very outdated version
> of the p0f engine, and the signature set is likewise stale (last
> update from 2011!).
>
> Unfortunately p0f has changed quite a bit in meantime, so that it's
> not easy to upgrade. While we'd certainly be happy to do that if
> anybody wanted to work on it, for now we are considering to remove the
> old engine that's currently shipping with Zeek because it doesn't seem
> to provide much value anymore.
>
> Please chime in if that would be a problem for you. Is anybody still
> relying on the p0f support in Zeek as it is today?
>
> Thanks,
>
> Robin
>
>
> --
> Robin Sommer * Corelight, Inc. * [email protected] * www.corelight.com
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 17 Jun 2019 15:05:18 -0700
> From: Micha? Purzy?ski <[email protected]>
> Subject: Re: [Zeek] State of p0f support
> To: Robin Sommer <[email protected]>
> Cc: zeek <[email protected]>
> Message-ID:
>         <CAJ6bFK1MzSH=fpWAcDQu2ANFTLAGOiicBHA33ZNL=
> [email protected]>
> Content-Type: text/plain; charset="utf-8"
>
> There is so much data in various logs, like software.log, http.log, SSL,
> DNS, known_*, x509 and even in the conn.log that recognizing the OS is most
> of the time trivial. I would rather invest into correlation and build a
> scoring engine that logs a verdict "based on A, B and C I think this is a
> Windows 10"
>
> On Mon, Jun 17, 2019 at 1:55 PM Robin Sommer <[email protected]> wrote:
>
> > Looking for some input here.
> >
> > Zeek has provided support for passive OS fingerprinting for a long
> > time through p0f. However, we are using using a very outdated version
> > of the p0f engine, and the signature set is likewise stale (last
> > update from 2011!).
> >
> > Unfortunately p0f has changed quite a bit in meantime, so that it's
> > not easy to upgrade. While we'd certainly be happy to do that if
> > anybody wanted to work on it, for now we are considering to remove the
> > old engine that's currently shipping with Zeek because it doesn't seem
> > to provide much value anymore.
> >
> > Please chime in if that would be a problem for you. Is anybody still
> > relying on the p0f support in Zeek as it is today?
> >
> > Thanks,
> >
> > Robin
> >
> >
> > --
> > Robin Sommer * Corelight, Inc. * [email protected] * www.corelight.com
> > _______________________________________________
> > Zeek mailing list
> > [email protected]
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190617/3f89f3bf/attachment-0001.html
>
> ------------------------------
>
> Message: 3
> Date: Mon, 17 Jun 2019 19:46:57 -0700
> From: Jon Siwek <[email protected]>
> Subject: Re: [Zeek] Hui Lin_DNP3 analyzer not working in current
>         version of      zeek
> To: "Hui Lin (Hugo)" <[email protected]>
> Cc: zeek <[email protected]>
> Message-ID:
>         <
> CAMzgZ0+RHVhvSv5UHVhOz-4cgEg3_km4eDd+NcCic9SR8eEuXg@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Found the difference:
>
> This behavior changed in Bro 2.5.4 (really what changed is in BinPAC
> 0.49, not Bro), but the new parsing behavior is legitimate.  The old
> behavior just caused broken protocol grammars to possibly parse more
> things than they should have, such as in cases where there wasn't enough
> data to fill an array.  So it appeared to be working for you, but it
> was not.
>
> In this case, the DNP3 protocol grammar we use is either incomplete or
> needs a further fix.  With some debugging for this example pcap, you can
> see where the parsing is failing:
>
> protocol violation, [orig_h=10.0.0.3, orig_p=37147/tcp,
> resp_h=10.0.0.1, resp_p=20000/tcp], Analyzer::ANALYZER_DNP3_TCP,
> Binpac exception: binpac exception: out_of_bound:
> Request_Objects:ojbects: 8 > 0
> protocol violation, [orig_h=10.0.0.3, orig_p=55021/tcp,
> resp_h=10.0.0.2, resp_p=20000/tcp], Analyzer::ANALYZER_DNP3_TCP,
> Binpac exception: binpac exception: out_of_bound:
> Request_Objects:ojbects: 8 > 0
>
> That's here:
>
>
> https://github.com/zeek/zeek/blob/e2dc0092f3a1caea1ebc71e347663e723298fb6b/src/analyzer/protocol/dnp3/dnp3-protocol.pac#L97
>
> You can look in the pcap (e.g. Wireshark) and see in the first READ
> request that there's no objects being sent for us to parse even though
> our protocol definition is written to expect that.  So that protocol
> violation is legit in the sense that we've defined the protocol in a way
> that differs from what's being sent on the wire.  And a protocol
> violation in the case of DNP3 disables all further analysis.
>
> You maybe understand DNP3 better than me, so please create an issue
> or pull request if you come up with a fix that improves the DNP3 parser.
> Attached is a naive patch that seems to generate the same number of
> requests/responses as before Bro 2.5.4; maybe it helps as a
> starting point or reference.
>
> - Jon
>
> On Mon, Jun 17, 2019 at 11:53 AM Hui Lin (Hugo) <[email protected]>
> wrote:
> >
> > Thanks Jon,
> >
> > The version that is working is version 2.5-457. I have attached the
> sample pcap here.
> >
> > Best,
> >
> > Hui Lin
> >
> > On Mon, Jun 17, 2019 at 10:32 AM Jon Siwek <[email protected]> wrote:
> >>
> >> On Sun, Jun 16, 2019 at 3:17 PM Hui Lin (Hugo) <[email protected]>
> wrote:
> >> > Actually for the same pcap, in a version that I git last year, bro
> works fine by printing all messages. Any idea what happens? If needed, I
> can provide the pcap for the testing.
> >>
> >> Not sure, please either try to debug / explore the diffs, or provide a
> >> pcap and say which was the last known working version.
> >>
> >> - Jon
> >
> >
> >
> > --
> > Hui Lin
> > Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
> > DEPEND (http://depend.csl.illinois.edu/)
> > ECE, Uni. of Illinois at Urbana-Champaign
> >
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: dnp3-naive.patch
> Type: application/octet-stream
> Size: 1229 bytes
> Desc: not available
> Url :
> http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190617/e6d65dd9/attachment-0001.obj
>
> ------------------------------
>
> Message: 4
> Date: Mon, 17 Jun 2019 21:17:40 -0700
> From: "Hui Lin (Hugo)" <[email protected]>
> Subject: Re: [Zeek] Hui Lin_DNP3 analyzer not working in current
>         version of      zeek
> To: Jon Siwek <[email protected]>
> Cc: zeek <[email protected]>
> Message-ID:
>         <
> CAKq214mGdRLKPDZtaXe-w6juvBcpUpmmz6SFYrxgBOfsPwVVQA@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Jon,
>
> This is a very confusing part of DNP3 protocol, but only in the request.
> The "number_of_item" obtained from "object_header" in the request is mainly
> applied to the response, telling outstation that this is the number of
> items that you should include in the response. But there are a few
> exceptions where the request is also using the "number_of_item", especially
> the control operations, meaning that these are the operations that we would
> like to apply to the outstation.
>
> Back then when I had implemented it, I use a common record
> type, Request_Data_Object, to hide those differences. In the READ request,
> there still should be Request_Data_Object[8]. However, according to the
> value of the function code (which is the input of the Request_Data_Object),
> the size of each Request_Data_Object becomes 0 if it is READ request. Even
> though there are 8 objects, but each of them has 0 bytes, so totally there
> are no data coming. That is why actually there are no more data coming,
> which is not a mistake. I guess that it is probably how binpac changes the
> way to handle this type of situation now, making the analyzer fail to work.
>
> For me, the workaround should not be difficult (also I am pretty sure that
> Robin and Seth would like to keep the binpac works as it is now).  As there
> are only a very few exceptions, I just include those exceptions directly
> there (similar to what you did in the patch), and then default most other
> requests to empty objects.
>
> I probably will do that on July as I am catching a deadline on July 1st. I
> will contact you privately as I may need you to help me to input a
> different public key for upload as I have already graduated.
>
> Best,
>
> Hui Lin
>
> On Mon, Jun 17, 2019 at 7:57 PM Jon Siwek <[email protected]> wrote:
>
> > Found the difference:
> >
> > This behavior changed in Bro 2.5.4 (really what changed is in BinPAC
> > 0.49, not Bro), but the new parsing behavior is legitimate.  The old
> > behavior just caused broken protocol grammars to possibly parse more
> > things than they should have, such as in cases where there wasn't enough
> > data to fill an array.  So it appeared to be working for you, but it
> > was not.
> >
> > In this case, the DNP3 protocol grammar we use is either incomplete or
> > needs a further fix.  With some debugging for this example pcap, you can
> > see where the parsing is failing:
> >
> > protocol violation, [orig_h=10.0.0.3, orig_p=37147/tcp,
> > resp_h=10.0.0.1, resp_p=20000/tcp], Analyzer::ANALYZER_DNP3_TCP,
> > Binpac exception: binpac exception: out_of_bound:
> > Request_Objects:ojbects: 8 > 0
> > protocol violation, [orig_h=10.0.0.3, orig_p=55021/tcp,
> > resp_h=10.0.0.2, resp_p=20000/tcp], Analyzer::ANALYZER_DNP3_TCP,
> > Binpac exception: binpac exception: out_of_bound:
> > Request_Objects:ojbects: 8 > 0
> >
> > That's here:
> >
> >
> >
> https://github.com/zeek/zeek/blob/e2dc0092f3a1caea1ebc71e347663e723298fb6b/src/analyzer/protocol/dnp3/dnp3-protocol.pac#L97
> >
> > You can look in the pcap (e.g. Wireshark) and see in the first READ
> > request that there's no objects being sent for us to parse even though
> > our protocol definition is written to expect that.  So that protocol
> > violation is legit in the sense that we've defined the protocol in a way
> > that differs from what's being sent on the wire.  And a protocol
> > violation in the case of DNP3 disables all further analysis.
> >
> > You maybe understand DNP3 better than me, so please create an issue
> > or pull request if you come up with a fix that improves the DNP3 parser.
> > Attached is a naive patch that seems to generate the same number of
> > requests/responses as before Bro 2.5.4; maybe it helps as a
> > starting point or reference.
> >
> > - Jon
> >
> > On Mon, Jun 17, 2019 at 11:53 AM Hui Lin (Hugo) <[email protected]>
> > wrote:
> > >
> > > Thanks Jon,
> > >
> > > The version that is working is version 2.5-457. I have attached the
> > sample pcap here.
> > >
> > > Best,
> > >
> > > Hui Lin
> > >
> > > On Mon, Jun 17, 2019 at 10:32 AM Jon Siwek <[email protected]>
> wrote:
> > >>
> > >> On Sun, Jun 16, 2019 at 3:17 PM Hui Lin (Hugo) <[email protected]>
> > wrote:
> > >> > Actually for the same pcap, in a version that I git last year, bro
> > works fine by printing all messages. Any idea what happens? If needed, I
> > can provide the pcap for the testing.
> > >>
> > >> Not sure, please either try to debug / explore the diffs, or provide a
> > >> pcap and say which was the last known working version.
> > >>
> > >> - Jon
> > >
> > >
> > >
> > > --
> > > Hui Lin
> > > Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
> > > DEPEND (http://depend.csl.illinois.edu/)
> > > ECE, Uni. of Illinois at Urbana-Champaign
> > >
> > _______________________________________________
> > Zeek mailing list
> > [email protected]
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> --
> Hui Lin
> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
> DEPEND (http://depend.csl.illinois.edu/)
> ECE, Uni. of Illinois at Urbana-Champaign
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190617/06350065/attachment.html
>
> ------------------------------
>
> _______________________________________________
> Zeek mailing list
> [email protected]
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
> End of Zeek Digest, Vol 158, Issue 22
> *************************************
>

_______________________________________________
Zeek mailing list
[email protected]
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek