Re: Multithreaded nDPI

On 10/03/2019 09:26, Luca Deri wrote:

> The best example is the ntopng code. have a look at it. Please avoid 
> locks, they are a waste of time every time you lock/unlock. You can 
> avoid them, see the ntopng code

I've looked through the ntopng code, and as far as I can see it doesn't 
provide an example of processing a single stream of packets (i.e. 
receiving on a single NIC) across multiple threads.

As far as I can see (and please correct me if I'm wrong), ntopng uses 
one thread per NIC and treats each NIC more or less independently.  This 
obviously solves some concurrency problems since it makes it likely 
(although not guaranteed) that the flows handled by different threads 
are completely unrelated to each other.

In ntopng, each thread appears to maintain its own pool of flows and 
peers, which means that if a client attached to eth0 is talking to the 
same server as a client attached to eth1, nDPI cannot share information 
about the server between the threads.

My understanding is that nDPI can use knowledge of what was detected in 
one flow to infer things about a different flow that involves the same 
peer.  If this is the case, surely it is impacted by partitioning this 
data between threads?


When processing packets received from a single NIC, we have a much 
harder job in splitting up "unrelated" flows.  Take the following flows 
as an example:
   (1) udp, 10.0.0.1:1234 <-> 192.168.0.1:5678
   (2) udp, 10.0.0.1:1234 <-> 192.168.0.2:5678
In a single threaded process, nDPI sees the first flow, records 
information about it in the flow[1], peer[10.0.0.1] and 
peer[192.168.0.1] structures.  It then sees the second flow and records 
information about it in the flow[2], peer[10.0.0.1] and 
peer[192.168.0.2] structures.  While processing the second flow it can 
use what it already knows about peer[10.0.0.1] to make inferences about 
the flow.

In a multithreaded process using a 5-tuple hash, the flows may be 
allocated to different threads.  Thread 1 sees the first flow, records 
information about it in the thread[1]->flow[1], 
thread[1]->peer[10.0.0.1] and thread[1]->peer[192.168.0.1] structures. 
Thread 2 then sees the second flow and records information about it in 
the thread[2]->flow[2], thread[2]->peer[10.0.0.1] and 
thread[2]->peer[192.168.0.2] structures.  It cannot access the 
thread[1]->peer[10.0.0.1] structure, so is unable to use that 
information to make inferences about the second flow.

How does this impact the detection process?

> No the flow information is stored inside the flow struct 
> https://github.com/ntop/ntopng/blob/dev/include/Flow.h#L63

ndpi_detection_module_struct appears to include various caches that are 
shared between flows, such as ookla_cache.  If different flows see 
different versions of those caches, is that not going to break some 
detections?


-- 
  - Steve Hill
    Technical Director | Cyfarwyddwr Technegol
    Opendium    Online Safety & Web Filtering     http://www.opendium.com
                Diogelwch Ar-Lein a Hidlo Gwefan

    Enquiries | Ymholiadau:   [email protected]     +44-1792-824568
    Support   | Cefnogi:      [email protected]   +44-1792-825748

------------------------------------------------------------------------
Opendium Limited is a company registered in England and Wales.
Mae Opendium Limited yn gwmni sydd wedi'i gofrestru yn Lloegr a Chymru.

Company No. | Rhif Cwmni:   5465437
Highfield House, 1 Brue Close, Bruton, Somerset, BA10 0HY, England.