Re: n2disk automated capture archive workflow

Alfredo Cardigliano <[email protected]> Fri, 30 Mar 2018 20:20:05 +0200
Newsgroups	gmane.linux.ntop.general
Message-ID	<[email protected]>
Hi Raoul
please read my comments inline.

> On 30 Mar 2018, at 20:07, Raoul Duke <[email protected]> wrote:
> 
> Hi list,
> 
> I'm evaluating n2disk for a specific use-case but I'm not clear how to map my workflow to its mental model so I wanted to ask your advice.
> 
> my desired workflow is:
> 
> * I start capturing with n2disk.  I'm using the -I option to make on-the-fly index
> * when a certain session event occurs in my application I want to archive the pcap for that session (in the event I will know which remote UDP ip/port were used in the session)
> 
> I have had success performing the above steps manuall/independently but my attempts to automate them have become protracted and I wanted to ask if perhaps I'm missing something simpler way to do this.
> 
> the main issue I have is that when my application event occurs (lets say I know that a given application session $sessionid just ended involving remote $ip and $port) then I'd like the knowledge of that application session ending to trigger a npcapextract for $ip and $port to archive the session under a filename of $sessionid.  I have all these variables and have a script which is ready to automatically run the extract...
> 
> the problem I have is:
> * n2disk has not flushed the data to disk yet and so I can't run the extract yet.  so how can I know when it is safe to run the extract?

You can send a SIGUSR1 to the n2disk process to flush the current PCAP to disk, if the disk is fast enough you should be able to extract traffic after a couple of seconds at most.

> * I read in release notes that it was possible to us " kill -USR1 to close and flush the current pcap in order to make live traffic immediately available" which works but I notice every time I call it it generates a new index file.  Which then leads me to the question of: how do I know which index file to run npcapextract against?  e.g. if the latest index was 1.idx and I do a kill -USR do I have to guess that my application events would be found in 1.idx / 1.pcap or is there a another way to do this?

I recommend you to enable the timeline, and just specify the time interval in npcapextract, using the timeline as data source instead of the specific pcap/index.

> * looking at all this another way.  I'd be happy to defer the npcapextract until the data is naturally flushed to disk. but this leads me to 2 questions:
>  - how can I know when all the relevant data is flushed to disk so I can take action on the npcapextract?   e.g. is there some concept of a hook/trigger I can call when pcap / index data is flushed to disk?

You probably need to know what is the timestamp of the last packet dumped to disk, maybe we can write it under /proc/net/pf_ring/stats/<n2disk stats>. If this works for you we can add it to the features list.

> - how can I know which index file is the "current” one

You can check the time of last index on disk, however as I said you probably do not need this if you use the timeline and we export the time of last packet dumped to disk in the stats.

> I am new to n2disk so I am likely just coming at this with some flawed mental model.   I hope at least my question articulates what I'm trying to accomplish.
> 
> I'd be very grateful for any input on how I can accomplish my use-case.
> 
> Thanks,
> RD

Best Regards
Alfredo

> 
> 
> 
> 
> _______________________________________________
> Ntop mailing list
> [email protected]
> http://listgateway.unipi.it/mailman/listinfo/ntop

_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
signature.asc (application/pgp-signature, 833 B)