Re: Multithreaded nDPI

Hi Steve
processing traffic from a single interface with multiple streams/thread is similar 
to processing traffic from multiple interfaces with one thread per interface, as 
long as traffic is distributed across streams based on 5-tuple. This means that
you need (to avoid locking, for performance) separate data structures including
the ndpi struct.

Alfredo

> On 8 Mar 2019, at 17:51, Steve Hill <[email protected]> wrote:
> 
> 
> I'm looking for some advice on using nDPI in multithreaded code.
> 
> Some Googling tells me that I need a separate ndpi_detection_module_struct for each thread, but beyond that I can't find a lot of information.  The example ndpiReader code seems to only use threads when processing multiple independent pcap files or NICs. That's not directly applicable to my situation, as I'm processing a stream of packets from a single NIC so don't have completely independent data sources like that.
> 
> So, I can use a 5-tuple hash to pick a consistent thread for each flow. That would mean that each flow would consistently be processed with the same ndpi_detection_module_struct.
> 
> The documentation isn't very clear, but presumably each endpoint should have a single ndpi_id_struct, irrespective of how many flows it has. Therefore each of the ndpi_id_structs are shared between multiple flows and may therefore be used by any of the threads in association with any of the ndpi_detection_module_structs.  I can obviously lock the ndpi_id_structs so that each can only be used by one thread at a time, but I'm not sure whether using the same endpoint with multiple ndpi_detection_module_structs is going to be a problem?
> 
> Finally, I'm not sure how much information nDPI shares between flows via the ndpi_detection_module_struct.  i.e. when a flow is processed, does nDPI store any information in ndpi_detection_module_struct about what it has learnt and then use that information when processing other flows? If so, splitting the flows between multiple ndpi_detection_module_structs is fundamentally problematic.
> 
> 
> I've also got a quick query about the ndpi_id_structs that are passed to ndpi_detection_process_packet.  The documentation just says that these are "source" and "destination", but isn't clear on whether these are the source/destination of the packet or of the flow.  Obviously reply packets will have the source/destination the opposite way around to the flow's source/destination.
> 
> 
> Many thanks.
> 
> -- 
> - Steve Hill
>   Technical Director | Cyfarwyddwr Technegol
>   Opendium    Online Safety & Web Filtering     http://www.opendium.com
>               Diogelwch Ar-Lein a Hidlo Gwefan
> 
>   Enquiries | Ymholiadau:   [email protected]     +44-1792-824568
>   Support   | Cefnogi:      [email protected]   +44-1792-825748
> 
> ------------------------------------------------------------------------
> Opendium Limited is a company registered in England and Wales.
> Mae Opendium Limited yn gwmni sydd wedi'i gofrestru yn Lloegr a Chymru.
> 
> Company No. | Rhif Cwmni:   5465437
> Highfield House, 1 Brue Close, Bruton, Somerset, BA10 0HY, England.
> _______________________________________________
> Ntop-dev mailing list
> [email protected]
> http://listgateway.unipi.it/mailman/listinfo/ntop-dev