Re: Multithreaded nDPI
Luca Deri <[email protected]> Sun, 10 Mar 2019 10:26:45 +0100
| Newsgroups | gmane.linux.ntop.devel |
|---|---|
| Message-ID | <[email protected]> |
--===============1223754599060986228== Content-Type: multipart/alternative; boundary="Apple-Mail=_5F3AEB9C-06EA-40F6-AEC5-BC1AF770B059" --Apple-Mail=_5F3AEB9C-06EA-40F6-AEC5-BC1AF770B059 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On 8 Mar 2019, at 17:51, Steve Hill <[email protected]> wrote: >=20 >=20 > I'm looking for some advice on using nDPI in multithreaded code. >=20 > Some Googling tells me that I need a separate = ndpi_detection_module_struct for each thread, but beyond that I can't = find a lot of information. The example ndpiReader code seems to only = use threads when processing multiple independent pcap files or NICs. = That's not directly applicable to my situation, as I'm processing a = stream of packets from a single NIC so don't have completely independent = data sources like that. yes. You need to have a separate module per thread to avoid locks. = However if for some reason you want to follow that path , this is also = an option=20 >=20 > So, I can use a 5-tuple hash to pick a consistent thread for each = flow. That would mean that each flow would consistently be processed = with the same ndpi_detection_module_struct. The module contains the information about protocols etc, but for each = flow you need to have the ndpi peers and flow info. So make sure that = when a thread starts processing one flow, it is the only one that does = that =20 >=20 > The documentation isn't very clear, but presumably each endpoint = should have a single ndpi_id_struct, irrespective of how many flows it = has. Therefore each of the ndpi_id_structs are shared between multiple = flows and may therefore be used by any of the threads in association = with any of the ndpi_detection_module_structs. I can obviously lock the = ndpi_id_structs so that each can only be used by one thread at a time, = but I'm not sure whether using the same endpoint with multiple = ndpi_detection_module_structs is going to be a problem? The best example is the ntopng code. have a look at it. Please avoid = locks, they are a waste of time every time you lock/unlock. You can = avoid them, see the ntopng code >=20 > Finally, I'm not sure how much information nDPI shares between flows = via the ndpi_detection_module_struct. i.e. when a flow is processed, = does nDPI store any information in ndpi_detection_module_struct about = what it has learnt and then use that information when processing other = flows? If so, splitting the flows between multiple = ndpi_detection_module_structs is fundamentally problematic. No the flow information is stored inside the flow struct = https://github.com/ntop/ntopng/blob/dev/include/Flow.h#L63 >=20 >=20 > I've also got a quick query about the ndpi_id_structs that are passed = to ndpi_detection_process_packet. The documentation just says that = these are "source" and "destination", but isn't clear on whether these = are the source/destination of the packet or of the flow. Obviously = reply packets will have the source/destination the opposite way around = to the flow's source/destination. Correct. Again please see = https://github.com/ntop/ntopng/blob/dev/include/Flow.h#L84 = <https://github.com/ntop/ntopng/blob/dev/include/Flow.h#L84> Luca >=20 >=20 > Many thanks. >=20 > --=20 > - Steve Hill > Technical Director | Cyfarwyddwr Technegol > Opendium Online Safety & Web Filtering = http://www.opendium.com > Diogelwch Ar-Lein a Hidlo Gwefan >=20 > Enquiries | Ymholiadau: [email protected] +44-1792-824568 > Support | Cefnogi: [email protected] +44-1792-825748 >=20 > = ------------------------------------------------------------------------ > Opendium Limited is a company registered in England and Wales. > Mae Opendium Limited yn gwmni sydd wedi'i gofrestru yn Lloegr a = Chymru. >=20 > Company No. | Rhif Cwmni: 5465437 > Highfield House, 1 Brue Close, Bruton, Somerset, BA10 0HY, England. > _______________________________________________ > Ntop-dev mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-dev --Apple-Mail=_5F3AEB9C-06EA-40F6-AEC5-BC1AF770B059 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><br = class=3D""><div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On 8 Mar 2019, at 17:51, Steve Hill <<a = href=3D"mailto:[email protected]" class=3D"">[email protected]</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = class=3D""><br class=3D"">I'm looking for some advice on using nDPI in = multithreaded code.<br class=3D""><br class=3D"">Some Googling tells me = that I need a separate ndpi_detection_module_struct for each thread, but = beyond that I can't find a lot of information. The example = ndpiReader code seems to only use threads when processing multiple = independent pcap files or NICs. That's not directly applicable to my = situation, as I'm processing a stream of packets from a single NIC so = don't have completely independent data sources like that.<br = class=3D""></div></div></blockquote>yes. You need to have a separate = module per thread to avoid locks. However if for some reason you want to = follow that path , this is also an option <br class=3D""><blockquote = type=3D"cite" class=3D""><div class=3D""><div class=3D""><br = class=3D"">So, I can use a 5-tuple hash to pick a consistent thread for = each flow. That would mean that each flow would consistently be = processed with the same ndpi_detection_module_struct.<br = class=3D""></div></div></blockquote>The module contains the information = about protocols etc, but for each flow you need to have the ndpi peers = and flow info. So make sure that when a thread starts processing one = flow, it is the only one that does that <br class=3D""><blockquote = type=3D"cite" class=3D""><div class=3D""><div class=3D""><br = class=3D"">The documentation isn't very clear, but presumably each = endpoint should have a single ndpi_id_struct, irrespective of how many = flows it has. Therefore each of the ndpi_id_structs are shared between = multiple flows and may therefore be used by any of the threads in = association with any of the ndpi_detection_module_structs. I can = obviously lock the ndpi_id_structs so that each can only be used by one = thread at a time, but I'm not sure whether using the same endpoint with = multiple ndpi_detection_module_structs is going to be a problem?<br = class=3D""></div></div></blockquote><div><br class=3D""></div><div>The = best example is the ntopng code. have a look at it. Please avoid locks, = they are a waste of time every time you lock/unlock. You can avoid them, = see the ntopng code</div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div class=3D""><br class=3D"">Finally, I'm = not sure how much information nDPI shares between flows via the = ndpi_detection_module_struct. i.e. when a flow is processed, does = nDPI store any information in ndpi_detection_module_struct about what it = has learnt and then use that information when processing other flows? If = so, splitting the flows between multiple ndpi_detection_module_structs = is fundamentally problematic.<br = class=3D""></div></div></blockquote><div><br class=3D""></div>No the = flow information is stored inside the flow struct <a = href=3D"https://github.com/ntop/ntopng/blob/dev/include/Flow.h#L63" = class=3D"">https://github.com/ntop/ntopng/blob/dev/include/Flow.h#L63</a><= br class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = class=3D""><br class=3D""><br class=3D"">I've also got a quick query = about the ndpi_id_structs that are passed to = ndpi_detection_process_packet. The documentation just says that = these are "source" and "destination", but isn't clear on whether these = are the source/destination of the packet or of the flow. Obviously = reply packets will have the source/destination the opposite way around = to the flow's source/destination.<br = class=3D""></div></div></blockquote><div><br class=3D""></div>Correct. = Again please see <a = href=3D"https://github.com/ntop/ntopng/blob/dev/include/Flow.h#L84" = class=3D"">https://github.com/ntop/ntopng/blob/dev/include/Flow.h#L84</a><= /div><div><br class=3D""></div><div>Luca<br class=3D""><blockquote = type=3D"cite" class=3D""><div class=3D""><div class=3D""><br = class=3D""><br class=3D"">Many thanks.<br class=3D""><br class=3D"">-- = <br class=3D""> - Steve Hill<br class=3D""> Technical = Director | Cyfarwyddwr Technegol<br class=3D""> Opendium = Online Safety & Web Filtering = <a href=3D"http://www.opendium.com" = class=3D"">http://www.opendium.com</a><br class=3D""> = &n= bsp; Diogelwch Ar-Lein a Hidlo Gwefan<br class=3D""><br class=3D""> = Enquiries | Ymholiadau: <a = href=3D"mailto:[email protected]" class=3D"">[email protected]</a> = +44-1792-824568<br class=3D""> = Support | Cefnogi: = <a href=3D"mailto:[email protected]" = class=3D"">[email protected]</a> +44-1792-825748<br = class=3D""><br = class=3D"">---------------------------------------------------------------= ---------<br class=3D"">Opendium Limited is a company registered in = England and Wales.<br class=3D"">Mae Opendium Limited yn gwmni sydd = wedi'i gofrestru yn Lloegr a Chymru.<br class=3D""><br class=3D"">Company = No. | Rhif Cwmni: 5465437<br class=3D"">Highfield House, 1 = Brue Close, Bruton, Somerset, BA10 0HY, England.<br = class=3D"">_______________________________________________<br = class=3D"">Ntop-dev mailing list<br class=3D""><a = href=3D"mailto:[email protected]" = class=3D"">[email protected]</a><br = class=3D"">http://listgateway.unipi.it/mailman/listinfo/ntop-dev<br = class=3D""></div></div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_5F3AEB9C-06EA-40F6-AEC5-BC1AF770B059-- --===============1223754599060986228== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ntop-dev mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-dev --===============1223754599060986228==--