Re: State of p0f support
Michał Purzyński <[email protected]>
| Newsgroups | gmane.comp.security.detection.bro |
|---|---|
| Message-ID | <[email protected]> |
What’s needed here is some heuristics. If I see, for example, windows crypto api, BITS, calls to MS services for reporting what usb devices were plugged in, certain DNS lookups - that’s MS. Apple also has similar services. So does iOS and Android. It’s more an art than a science ;) JA3 would also be great. > On Jun 17, 2019, at 11:34 PM, TQ <[email protected]> wrote: > > @Michal, > > That's a really good suggestion! Latest p0f is from 2016, so it's not that maintained anyway I guess. The thing I noticed for software.log is that the OS info gets logged only via software calls from apps like Firefox/Chrome/etc; it would be nice to not rely on this. > > Thanks, > >> On Mon, Jun 17, 2019 at 9:22 PM <[email protected]> wrote: >> Send Zeek mailing list submissions to >> [email protected] >> >> To subscribe or unsubscribe via the World Wide Web, visit >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >> or, via email, send a message with subject or body 'help' to >> [email protected] >> >> You can reach the person managing the list at >> [email protected] >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of Zeek digest..." >> >> >> Today's Topics: >> >> 1. State of p0f support (Robin Sommer) >> 2. Re: State of p0f support (Micha? Purzy?ski) >> 3. Re: Hui Lin_DNP3 analyzer not working in current version of >> zeek (Jon Siwek) >> 4. Re: Hui Lin_DNP3 analyzer not working in current version of >> zeek (Hui Lin (Hugo)) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Mon, 17 Jun 2019 13:45:27 -0700 >> From: Robin Sommer <[email protected]> >> Subject: [Zeek] State of p0f support >> To: zeek <[email protected]> >> Message-ID: <[email protected]> >> Content-Type: text/plain; charset=us-ascii >> >> Looking for some input here. >> >> Zeek has provided support for passive OS fingerprinting for a long >> time through p0f. However, we are using using a very outdated version >> of the p0f engine, and the signature set is likewise stale (last >> update from 2011!). >> >> Unfortunately p0f has changed quite a bit in meantime, so that it's >> not easy to upgrade. While we'd certainly be happy to do that if >> anybody wanted to work on it, for now we are considering to remove the >> old engine that's currently shipping with Zeek because it doesn't seem >> to provide much value anymore. >> >> Please chime in if that would be a problem for you. Is anybody still >> relying on the p0f support in Zeek as it is today? >> >> Thanks, >> >> Robin >> >> >> -- >> Robin Sommer * Corelight, Inc. * [email protected] * www.corelight.com >> >> >> ------------------------------ >> >> Message: 2 >> Date: Mon, 17 Jun 2019 15:05:18 -0700 >> From: Micha? Purzy?ski <[email protected]> >> Subject: Re: [Zeek] State of p0f support >> To: Robin Sommer <[email protected]> >> Cc: zeek <[email protected]> >> Message-ID: >> <CAJ6bFK1MzSH=fpWAcDQu2ANFTLAGOiicBHA33ZNL=5De2NLQuA@mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> There is so much data in various logs, like software.log, http.log, SSL, >> DNS, known_*, x509 and even in the conn.log that recognizing the OS is most >> of the time trivial. I would rather invest into correlation and build a >> scoring engine that logs a verdict "based on A, B and C I think this is a >> Windows 10" >> >> On Mon, Jun 17, 2019 at 1:55 PM Robin Sommer <[email protected]> wrote: >> >> > Looking for some input here. >> > >> > Zeek has provided support for passive OS fingerprinting for a long >> > time through p0f. However, we are using using a very outdated version >> > of the p0f engine, and the signature set is likewise stale (last >> > update from 2011!). >> > >> > Unfortunately p0f has changed quite a bit in meantime, so that it's >> > not easy to upgrade. While we'd certainly be happy to do that if >> > anybody wanted to work on it, for now we are considering to remove the >> > old engine that's currently shipping with Zeek because it doesn't seem >> > to provide much value anymore. >> > >> > Please chime in if that would be a problem for you. Is anybody still >> > relying on the p0f support in Zeek as it is today? >> > >> > Thanks, >> > >> > Robin >> > >> > >> > -- >> > Robin Sommer * Corelight, Inc. * [email protected] * www.corelight.com >> > _______________________________________________ >> > Zeek mailing list >> > [email protected] >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >> > >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190617/3f89f3bf/attachment-0001.html >> >> ------------------------------ >> >> Message: 3 >> Date: Mon, 17 Jun 2019 19:46:57 -0700 >> From: Jon Siwek <[email protected]> >> Subject: Re: [Zeek] Hui Lin_DNP3 analyzer not working in current >> version of zeek >> To: "Hui Lin (Hugo)" <[email protected]> >> Cc: zeek <[email protected]> >> Message-ID: >> <CAMzgZ0+RHVhvSv5UHVhOz-4cgEg3_km4eDd+NcCic9SR8eEuXg@mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> Found the difference: >> >> This behavior changed in Bro 2.5.4 (really what changed is in BinPAC >> 0.49, not Bro), but the new parsing behavior is legitimate. The old >> behavior just caused broken protocol grammars to possibly parse more >> things than they should have, such as in cases where there wasn't enough >> data to fill an array. So it appeared to be working for you, but it >> was not. >> >> In this case, the DNP3 protocol grammar we use is either incomplete or >> needs a further fix. With some debugging for this example pcap, you can >> see where the parsing is failing: >> >> protocol violation, [orig_h=10.0.0.3, orig_p=37147/tcp, >> resp_h=10.0.0.1, resp_p=20000/tcp], Analyzer::ANALYZER_DNP3_TCP, >> Binpac exception: binpac exception: out_of_bound: >> Request_Objects:ojbects: 8 > 0 >> protocol violation, [orig_h=10.0.0.3, orig_p=55021/tcp, >> resp_h=10.0.0.2, resp_p=20000/tcp], Analyzer::ANALYZER_DNP3_TCP, >> Binpac exception: binpac exception: out_of_bound: >> Request_Objects:ojbects: 8 > 0 >> >> That's here: >> >> https://github.com/zeek/zeek/blob/e2dc0092f3a1caea1ebc71e347663e723298fb6b/src/analyzer/protocol/dnp3/dnp3-protocol.pac#L97 >> >> You can look in the pcap (e.g. Wireshark) and see in the first READ >> request that there's no objects being sent for us to parse even though >> our protocol definition is written to expect that. So that protocol >> violation is legit in the sense that we've defined the protocol in a way >> that differs from what's being sent on the wire. And a protocol >> violation in the case of DNP3 disables all further analysis. >> >> You maybe understand DNP3 better than me, so please create an issue >> or pull request if you come up with a fix that improves the DNP3 parser. >> Attached is a naive patch that seems to generate the same number of >> requests/responses as before Bro 2.5.4; maybe it helps as a >> starting point or reference. >> >> - Jon >> >> On Mon, Jun 17, 2019 at 11:53 AM Hui Lin (Hugo) <[email protected]> wrote: >> > >> > Thanks Jon, >> > >> > The version that is working is version 2.5-457. I have attached the sample pcap here. >> > >> > Best, >> > >> > Hui Lin >> > >> > On Mon, Jun 17, 2019 at 10:32 AM Jon Siwek <[email protected]> wrote: >> >> >> >> On Sun, Jun 16, 2019 at 3:17 PM Hui Lin (Hugo) <[email protected]> wrote: >> >> > Actually for the same pcap, in a version that I git last year, bro works fine by printing all messages. Any idea what happens? If needed, I can provide the pcap for the testing. >> >> >> >> Not sure, please either try to debug / explore the diffs, or provide a >> >> pcap and say which was the last known working version. >> >> >> >> - Jon >> > >> > >> > >> > -- >> > Hui Lin >> > Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/) >> > DEPEND (http://depend.csl.illinois.edu/) >> > ECE, Uni. of Illinois at Urbana-Champaign >> > >> -------------- next part -------------- >> A non-text attachment was scrubbed... >> Name: dnp3-naive.patch >> Type: application/octet-stream >> Size: 1229 bytes >> Desc: not available >> Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190617/e6d65dd9/attachment-0001.obj >> >> ------------------------------ >> >> Message: 4 >> Date: Mon, 17 Jun 2019 21:17:40 -0700 >> From: "Hui Lin (Hugo)" <[email protected]> >> Subject: Re: [Zeek] Hui Lin_DNP3 analyzer not working in current >> version of zeek >> To: Jon Siwek <[email protected]> >> Cc: zeek <[email protected]> >> Message-ID: >> <CAKq214mGdRLKPDZtaXe-w6juvBcpUpmmz6SFYrxgBOfsPwVVQA@mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> Hi Jon, >> >> This is a very confusing part of DNP3 protocol, but only in the request. >> The "number_of_item" obtained from "object_header" in the request is mainly >> applied to the response, telling outstation that this is the number of >> items that you should include in the response. But there are a few >> exceptions where the request is also using the "number_of_item", especially >> the control operations, meaning that these are the operations that we would >> like to apply to the outstation. >> >> Back then when I had implemented it, I use a common record >> type, Request_Data_Object, to hide those differences. In the READ request, >> there still should be Request_Data_Object[8]. However, according to the >> value of the function code (which is the input of the Request_Data_Object), >> the size of each Request_Data_Object becomes 0 if it is READ request. Even >> though there are 8 objects, but each of them has 0 bytes, so totally there >> are no data coming. That is why actually there are no more data coming, >> which is not a mistake. I guess that it is probably how binpac changes the >> way to handle this type of situation now, making the analyzer fail to work. >> >> For me, the workaround should not be difficult (also I am pretty sure that >> Robin and Seth would like to keep the binpac works as it is now). As there >> are only a very few exceptions, I just include those exceptions directly >> there (similar to what you did in the patch), and then default most other >> requests to empty objects. >> >> I probably will do that on July as I am catching a deadline on July 1st. I >> will contact you privately as I may need you to help me to input a >> different public key for upload as I have already graduated. >> >> Best, >> >> Hui Lin >> >> On Mon, Jun 17, 2019 at 7:57 PM Jon Siwek <[email protected]> wrote: >> >> > Found the difference: >> > >> > This behavior changed in Bro 2.5.4 (really what changed is in BinPAC >> > 0.49, not Bro), but the new parsing behavior is legitimate. The old >> > behavior just caused broken protocol grammars to possibly parse more >> > things than they should have, such as in cases where there wasn't enough >> > data to fill an array. So it appeared to be working for you, but it >> > was not. >> > >> > In this case, the DNP3 protocol grammar we use is either incomplete or >> > needs a further fix. With some debugging for this example pcap, you can >> > see where the parsing is failing: >> > >> > protocol violation, [orig_h=10.0.0.3, orig_p=37147/tcp, >> > resp_h=10.0.0.1, resp_p=20000/tcp], Analyzer::ANALYZER_DNP3_TCP, >> > Binpac exception: binpac exception: out_of_bound: >> > Request_Objects:ojbects: 8 > 0 >> > protocol violation, [orig_h=10.0.0.3, orig_p=55021/tcp, >> > resp_h=10.0.0.2, resp_p=20000/tcp], Analyzer::ANALYZER_DNP3_TCP, >> > Binpac exception: binpac exception: out_of_bound: >> > Request_Objects:ojbects: 8 > 0 >> > >> > That's here: >> > >> > >> > https://github.com/zeek/zeek/blob/e2dc0092f3a1caea1ebc71e347663e723298fb6b/src/analyzer/protocol/dnp3/dnp3-protocol.pac#L97 >> > >> > You can look in the pcap (e.g. Wireshark) and see in the first READ >> > request that there's no objects being sent for us to parse even though >> > our protocol definition is written to expect that. So that protocol >> > violation is legit in the sense that we've defined the protocol in a way >> > that differs from what's being sent on the wire. And a protocol >> > violation in the case of DNP3 disables all further analysis. >> > >> > You maybe understand DNP3 better than me, so please create an issue >> > or pull request if you come up with a fix that improves the DNP3 parser. >> > Attached is a naive patch that seems to generate the same number of >> > requests/responses as before Bro 2.5.4; maybe it helps as a >> > starting point or reference. >> > >> > - Jon >> > >> > On Mon, Jun 17, 2019 at 11:53 AM Hui Lin (Hugo) <[email protected]> >> > wrote: >> > > >> > > Thanks Jon, >> > > >> > > The version that is working is version 2.5-457. I have attached the >> > sample pcap here. >> > > >> > > Best, >> > > >> > > Hui Lin >> > > >> > > On Mon, Jun 17, 2019 at 10:32 AM Jon Siwek <[email protected]> wrote: >> > >> >> > >> On Sun, Jun 16, 2019 at 3:17 PM Hui Lin (Hugo) <[email protected]> >> > wrote: >> > >> > Actually for the same pcap, in a version that I git last year, bro >> > works fine by printing all messages. Any idea what happens? If needed, I >> > can provide the pcap for the testing. >> > >> >> > >> Not sure, please either try to debug / explore the diffs, or provide a >> > >> pcap and say which was the last known working version. >> > >> >> > >> - Jon >> > > >> > > >> > > >> > > -- >> > > Hui Lin >> > > Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/) >> > > DEPEND (http://depend.csl.illinois.edu/) >> > > ECE, Uni. of Illinois at Urbana-Champaign >> > > >> > _______________________________________________ >> > Zeek mailing list >> > [email protected] >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >> >> >> >> -- >> Hui Lin >> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/) >> DEPEND (http://depend.csl.illinois.edu/) >> ECE, Uni. of Illinois at Urbana-Champaign >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190617/06350065/attachment.html >> >> ------------------------------ >> >> _______________________________________________ >> Zeek mailing list >> [email protected] >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >> >> >> End of Zeek Digest, Vol 158, Issue 22 >> ************************************* > _______________________________________________ > Zeek mailing list > [email protected] > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek _______________________________________________ Zeek mailing list [email protected] http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek