Re: How to Analyse MikroTik Traffic Using nprobe -> ntopng on Windows

Hi,

This could be a timeout issue. Please try to add the following options 
to nprobe:

     --disable-cache

     --zmq-disable-buffering

Please also review the flow timeout in the ntopng cache preferences.

Regards,

Emanuele

On 12/24/18 10:32 PM, [email protected] wrote:
> Update to prev mail:
>
> Starting ntopng with:
>
> ntopng /c -i tcp://*:5556c
>
> and nprobe with:
>
> nprobe /c --zmq "tcp://127.0.0.1:5556" --zmq-probe-mode -i none -n 
> none --collector-port 2055 -T "@NTOPNG@"
>
> Results in traffic being parsed to GUI running on:
>
> http://127.0.0.1:3000/lua/hosts_stats.lua
>
> However when selecting Hosts, Filter Hosts, Local Hosts, it outputs: 
> No results found, yet we can see some of our local IPs listed under 
> Hosts (main menu).
>
> Our primary requirement right now is analyzing / recording LAN users 
> internet bandwidth usage.
>
> C:\Program Files\ntopng>ntopng /c -i tcp://*:5556c
>
> ===================================================================
> Starting ntopng
> Running ntopng.
> 24/Dec/2018 23:26:30 [Ntop.cpp:1545] Setting local networks to 
> 127.0.0.0/8
> 24/Dec/2018 23:26:30 [Redis.cpp:132] Successfully connected to redis 
> 127.0.0.1@0
> 24/Dec/2018 23:26:30 [Redis.cpp:132] Successfully connected to redis 
> 127.0.0.1@0
> 24/Dec/2018 23:26:30 [NtopPro.cpp:310] [LICENSE] Reading license from 
> Redis
> 24/Dec/2018 23:26:30 [NtopPro.cpp:451] WARNING: [LICENSE] Invalid or 
> missing license
> 24/Dec/2018 23:26:30 [NtopPro.cpp:468] WARNING: [LICENSE] ntopng will 
> now run in enterprise edition for 10 minutes
> 24/Dec/2018 23:26:30 [NtopPro.cpp:470] WARNING: [LICENSE] before 
> returning to community mode
> 24/Dec/2018 23:26:30 [NtopPro.cpp:472] WARNING: [LICENSE] You can buy 
> a permanent license at http://shop.ntop.org
> 24/Dec/2018 23:26:30 [NtopPro.cpp:474] WARNING: [LICENSE] or run 
> ntopng in community mode starting
> 24/Dec/2018 23:26:30 [NtopPro.cpp:475] WARNING: [LICENSE] ntopng 
> --community
> 24/Dec/2018 23:26:30 [Ntop.cpp:1639] Registered interface 
> tcp://*:5556c [id: 9]
> 24/Dec/2018 23:26:31 [HTTPserver.cpp:945] HTTPS Disabled: missing SSL 
> certificate C:\Program Files\ntopng\httpdocs/ssl/ntopng-cert.pem
> 24/Dec/2018 23:26:31 [HTTPserver.cpp:947] Please read 
> https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to 
> enable SSL.
> 24/Dec/2018 23:26:31 [HTTPserver.cpp:1114] Web server dirs [C:\Program 
> Files\ntopng\httpdocs][C:\Program Files\ntopng\scripts]
> 24/Dec/2018 23:26:31 [HTTPserver.cpp:1117] HTTP server listening on 3000
> 24/Dec/2018 23:26:31 [main.cpp:393] Working directory: 
> Z:\Cloud\OneDrive\MyPC\Documents\ntopng
> 24/Dec/2018 23:26:31 [main.cpp:395] Scripts/HTML pages directory: 
> C:\Program Files\ntopng
> 24/Dec/2018 23:26:31 [Ntop.cpp:390] Welcome to ntopng x64 v.3.7.180929 
> - (C) 1998-18 ntop.org
> 24/Dec/2018 23:26:31 [Ntop.cpp:400] Built on Windows
> 24/Dec/2018 23:26:31 [NtopPro.cpp:633] [LICENSE] System Id: 
> 2152224034-9206A1D8
> 24/Dec/2018 23:26:31 [NtopPro.cpp:634] [LICENSE] Edition: Enterprise
> 24/Dec/2018 23:26:31 [NtopPro.cpp:635] [LICENSE] License Type: 
> Time-Limited License
> 24/Dec/2018 23:26:31 [NtopPro.cpp:644] [LICENSE] Validity: Until Mon 
> Dec 24 23:36:30 2018
> 24/Dec/2018 23:26:31 [PeriodicActivities.cpp:68] Started periodic 
> activities loop...
> 24/Dec/2018 23:26:32 [PeriodicActivities.cpp:109] Each periodic 
> activity script will use 2 threads
> 24/Dec/2018 23:26:32 [NetworkInterface.cpp:2581] Started packet 
> polling on interface tcp://*:5556c [id: 9]...
> 24/Dec/2018 23:26:32 [CollectorInterface.cpp:122] Collecting flows on 
> tcp://*:5556c
> ====================================================================================== 
>
>
> C:\Program Files\nProbe>nprobe /c --zmq "tcp://127.0.0.1:5556" 
> --zmq-probe-mode -i none -n none --collector-port 2055 -T "@NTOPNG@"
>
> ============================================================================== 
>
> Running nProbe for Windows.
> 24/Dec/2018 23:26:40 [nprobe.c:4168] Valid nProbe license found
> 24/Dec/2018 23:26:40 [nprobe.c:6092] WARNING: The output interfaceId 
> is set to 0: did you forget to use -Q perhaps ?
> 24/Dec/2018 23:26:40 [nprobe.c:6095] WARNING: The input interfaceId is 
> set to 0: did you forget to use -u perhaps ?
> 24/Dec/2018 23:26:40 [nprobe.c:6182] Welcome to nProbe v.8.6.181004 
> ($Revision: 4384 $) for Windows
> 24/Dec/2018 23:26:40 [nprobe.c:6192] Running on Windows
> 24/Dec/2018 23:26:40 [nprobe.c:6203] [LICENSE] nProbe SystemId: 
> 2152224034-9206A1D8
> 24/Dec/2018 23:26:40 [nprobe.c:6270] Sample rate [packet: 1][flow 
> collection/export: 1/1]
> 24/Dec/2018 23:26:40 [nprobe.c:8966] Welcome to nProbe v.8.6.181004 
> for Windows
> 24/Dec/2018 23:26:40 [nprobe.c:7870] WARNING: Adding 
> %EXPORTER_IPV4_ADDRESS to the template as nProbe is working as collector
> 24/Dec/2018 23:26:40 [nprobe.c:7976] Using NetFlow Packet Payload Len: 
> 1472
> 24/Dec/2018 23:26:40 [nprobe.c:7906] @NTOPNG@ expanded to " %L7_PROTO 
> %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR 
> %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %IN_BYTES %IN_PKTS 
> %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %SRC_VLAN  
> %EXPORTER_IPV4_ADDRESS"
> 24/Dec/2018 23:26:40 [plugin.c:1238] 0 plugin(s) enabled
> 24/Dec/2018 23:26:40 [nprobe.c:8422] Each flow is 82 bytes long
> 24/Dec/2018 23:26:40 [nprobe.c:8423] The # flows per packet has been 
> set to 16
> 24/Dec/2018 23:26:40 [nprobe.c:8426] IP TOS is accounted
> 24/Dec/2018 23:26:40 [nprobe.c:8452] Non IPv4/v6 traffic is discarded 
> according to the template
> 24/Dec/2018 23:26:40 [nprobe.c:9231] Flows ASs will not be computed 
> (missing libmxminddb support)
> 24/Dec/2018 23:26:40 [nprobe.c:9334] Not capturing packet from 
> interface (collector mode)
> 24/Dec/2018 23:26:40 [util.c:4719] Initializing ZMQ as client
> 24/Dec/2018 23:26:40 [util.c:4738] Exporting flows towards ZMQ 
> endpoint tcp://127.0.0.1:5556
> 24/Dec/2018 23:26:40 [collect.c:142] Flow collector listening on port 
> 2055 (IPv4/v6)
> 24/Dec/2018 23:26:40 [nprobe.c:9582] nProbe started successfully
> 24/Dec/2018 23:30:26 [nprobe.c:567] Received shutdown request... 
> [signal: 2]
> 24/Dec/2018 23:30:27 [nprobe.c:6317] Flushing active flows
> 24/Dec/2018 23:30:27 [engine.c:3169] About to flush hash (threadId 0)
> 24/Dec/2018 23:30:27 [engine.c:3171] Completed hash walk (thread 0)
> ================================================================================== 
>
>
> Assistance greatly appreciated.
>
> Best,
>
> Johan.
>
>
> On 2018-12-24 20:50, [email protected] wrote:
>> Hi Emanuele,
>>
>> Both below Windows CMD terminals run as Administrator:
>>
>> C:\Program Files\ntopng>ntopng /c i -i tcp://*:5556c
>>
>> =============================================
>> Starting ntopng
>> Running ntopng.
>> 24/Dec/2018 20:39:33 [Ntop.cpp:1545] Setting local networks to 
>> 127.0.0.0/8
>> 24/Dec/2018 20:39:33 [Redis.cpp:132] Successfully connected to redis 
>> 127.0.0.1@0
>> 24/Dec/2018 20:39:33 [Redis.cpp:132] Successfully connected to redis 
>> 127.0.0.1@0
>> 24/Dec/2018 20:39:33 [NtopPro.cpp:310] [LICENSE] Reading license from 
>> Redis
>> 24/Dec/2018 20:39:33 [NtopPro.cpp:451] WARNING: [LICENSE] Invalid or
>> missing license
>> 24/Dec/2018 20:39:33 [NtopPro.cpp:468] WARNING: [LICENSE] ntopng will
>> now run in enterprise edition for 10 minutes
>> 24/Dec/2018 20:39:33 [NtopPro.cpp:470] WARNING: [LICENSE] before
>> returning to community mode
>> 24/Dec/2018 20:39:33 [NtopPro.cpp:472] WARNING: [LICENSE] You can buy
>> a permanent license at http://shop.ntop.org
>> 24/Dec/2018 20:39:33 [NtopPro.cpp:474] WARNING: [LICENSE] or run
>> ntopng in community mode starting
>> 24/Dec/2018 20:39:33 [NtopPro.cpp:475] WARNING: [LICENSE] ntopng 
>> --community
>> 24/Dec/2018 20:39:34 [CollectorInterface.cpp:66] ERROR: Unable to bind
>> to ZMQ endpoint tcp://*:5556 [collector]
>> 24/Dec/2018 20:39:35 [main.cpp:239] ERROR: An exception occurred
>> during tcp://*:5556c interface creation[2]: No such file or directory
>> 24/Dec/2018 20:39:35 [main.cpp:293] ERROR: Startup error: missing
>> super-user privileges ?
>>
>> C:\Program Files\ntopng>ntopng /c i -i tcp://*:5556c
>> Starting ntopng
>> Running ntopng.
>> 24/Dec/2018 20:40:36 [Ntop.cpp:1545] Setting local networks to 
>> 127.0.0.0/8
>> 24/Dec/2018 20:40:36 [Redis.cpp:132] Successfully connected to redis 
>> 127.0.0.1@0
>> 24/Dec/2018 20:40:36 [Redis.cpp:132] Successfully connected to redis 
>> 127.0.0.1@0
>> 24/Dec/2018 20:40:36 [NtopPro.cpp:310] [LICENSE] Reading license from 
>> Redis
>> 24/Dec/2018 20:40:36 [NtopPro.cpp:451] WARNING: [LICENSE] Invalid or
>> missing license
>> 24/Dec/2018 20:40:36 [NtopPro.cpp:468] WARNING: [LICENSE] ntopng will
>> now run in enterprise edition for 10 minutes
>> 24/Dec/2018 20:40:36 [NtopPro.cpp:470] WARNING: [LICENSE] before
>> returning to community mode
>> 24/Dec/2018 20:40:36 [NtopPro.cpp:472] WARNING: [LICENSE] You can buy
>> a permanent license at http://shop.ntop.org
>> 24/Dec/2018 20:40:36 [NtopPro.cpp:474] WARNING: [LICENSE] or run
>> ntopng in community mode starting
>> 24/Dec/2018 20:40:36 [NtopPro.cpp:475] WARNING: [LICENSE] ntopng 
>> --community
>> 24/Dec/2018 20:40:37 [CollectorInterface.cpp:66] ERROR: Unable to bind
>> to ZMQ endpoint tcp://*:5556 [collector]
>> 24/Dec/2018 20:40:37 [main.cpp:239] ERROR: An exception occurred
>> during tcp://*:5556c interface creation[2]: No such file or directory
>> 24/Dec/2018 20:40:37 [main.cpp:293] ERROR: Startup error: missing
>> super-user privileges ?
>> ================================================
>>
>> C:\Program Files\nProbe>nprobe /c my_nprobe --zmq
>> "tcp://<192.168.88.2>:5556" --zmq-probe-mode -i none -n none
>> --collector-port 2055 -T "@NTOPNG@"
>>
>> ============================================================
>> Running nProbe for Windows.
>> 24/Dec/2018 20:41:38 [nprobe.c:4168] Valid nProbe license found
>> 24/Dec/2018 20:41:38 [nprobe.c:6092] WARNING: The output interfaceId
>> is set to 0: did you forget to use -Q perhaps ?
>> 24/Dec/2018 20:41:38 [nprobe.c:6095] WARNING: The input interfaceId is
>> set to 0: did you forget to use -u perhaps ?
>> 24/Dec/2018 20:41:38 [nprobe.c:6182] Welcome to nProbe v.8.6.181004
>> ($Revision: 4384 $) for Windows
>> 24/Dec/2018 20:41:38 [nprobe.c:6192] Running on Windows
>> 24/Dec/2018 20:41:38 [nprobe.c:6203] [LICENSE] nProbe SystemId:
>> 2152224034-9206A1D8
>> 24/Dec/2018 20:41:38 [nprobe.c:6270] Sample rate [packet: 1][flow
>> collection/export: 1/1]
>> 24/Dec/2018 20:41:38 [nprobe.c:8966] Welcome to nProbe v.8.6.181004 
>> for Windows
>> 24/Dec/2018 20:41:38 [nprobe.c:7870] WARNING: Adding
>> %EXPORTER_IPV4_ADDRESS to the template as nProbe is working as
>> collector
>> 24/Dec/2018 20:41:38 [nprobe.c:7976] Using NetFlow Packet Payload 
>> Len: 1472
>> 24/Dec/2018 20:41:38 [nprobe.c:7906] @NTOPNG@ expanded to " %L7_PROTO
>> %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR
>> %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %IN_BYTES %IN_PKTS
>> %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %SRC_VLAN
>> %EXPORTER_IPV4_ADDRESS"
>> 24/Dec/2018 20:41:38 [plugin.c:1238] 0 plugin(s) enabled
>> 24/Dec/2018 20:41:38 [nprobe.c:8422] Each flow is 82 bytes long
>> 24/Dec/2018 20:41:38 [nprobe.c:8423] The # flows per packet has been 
>> set to 16
>> 24/Dec/2018 20:41:38 [nprobe.c:8426] IP TOS is accounted
>> 24/Dec/2018 20:41:38 [nprobe.c:8452] Non IPv4/v6 traffic is discarded
>> according to the template
>> 24/Dec/2018 20:41:38 [nprobe.c:9231] Flows ASs will not be computed
>> (missing libmxminddb support)
>> 24/Dec/2018 20:41:38 [nprobe.c:9334] Not capturing packet from
>> interface (collector mode)
>> 24/Dec/2018 20:41:38 [util.c:4719] Initializing ZMQ as client
>> 24/Dec/2018 20:41:38 [util.c:4736] ERROR: Unable to export flows
>> towards ZMQ endpoint tcp://<192.168.88.2>:5556: Invalid argument
>> 24/Dec/2018 20:41:38 [collect.c:142] Flow collector listening on port
>> 2055 (IPv4/v6)
>> 24/Dec/2018 20:41:38 [nprobe.c:9582] nProbe started successfully
>> 24/Dec/2018 20:46:29 [nprobe.c:567] Received shutdown request... 
>> [signal: 2]
>> 24/Dec/2018 20:46:29 [nprobe.c:6317] Flushing active flows
>> 24/Dec/2018 20:46:31 [nprobe.c:3127] Processed packets: 0 (max bucket 
>> search: 0)
>> 24/Dec/2018 20:46:31 [nprobe.c:3110] Fragment queue length: 0
>> 24/Dec/2018 20:46:31 [nprobe.c:3137] Flow collection stats:
>> [collected pkts: 0][processed flows: 0]
>> 24/Dec/2018 20:46:31 [nprobe.c:3140] Flow export stats:      [0
>> bytes/0 pkts][0 flows/0 pkts sent]
>> 24/Dec/2018 20:46:31 [nprobe.c:3146] Flow export drop stats: [0
>> bytes/0 pkts][0 flows]
>> 24/Dec/2018 20:46:31 [nprobe.c:3151] Total flow stats:       [0
>> bytes/0 pkts][0 flows/0 pkts sent]
>> ====================================================================
>>
>> Am not sure what to do / try form here, assistance appreciated,
>>
>> Best,
>>
>> Johan.
>>
>>
>> On 2018-12-24 16:02, Emanuele Faranda wrote:
>>> Hi,
>>>
>>> Please try to replace /i with /c so that you can see the commands 
>>> output.
>>>
>>> Regards,
>>>
>>> Emanuele
>>>
>>> On 12/24/18 12:17 AM, [email protected] wrote:
>>>> Update to the below, as per what Ive posted to the mailing list:
>>>>
>>>> We have Multiple nProbe sites with Mikrotik routers, and want to 
>>>> send flows to one remote ntopng instance running on a Windows machine.
>>>>
>>>> Starting with the local site all behind the same Firewall / on same 
>>>> LAN:
>>>>
>>>> Mikrotik is setup to send NetFlow to the IP of the host running 
>>>> nprobe & ntopng: 192.168.88.2
>>>>
>>>> ntopng started as service with the below CMD:
>>>>
>>>> ntopng /i -i tcp://*:5556c
>>>>
>>>> And nprobe with:
>>>>
>>>> nprobe /i my_nprobe --zmq "tcp://<192.168.88.2>:5556" 
>>>> --zmq-probe-mode -i none -n none --collector-port 2055 -T "@NTOPNG@"
>>>>
>>>> As per the steps outlined here:
>>>>
>>>> https://www.ntop.org/nprobe/best-practices-for-the-collection-of-flows-with-ntopng-and-nprobe/ 
>>>> However ntopng when loaded shows only:
>>>>
>>>> No packet has been received yet on interface tcp://*:5556c. Please 
>>>> wait 6 seconds until this page reloads.
>>>>
>>>> Have also tried the steps outlined below to no avail:
>>>>
>>>> https://www.ntop.org/ntopng/how-to-analyse-mikrotik-traffic-using-ntopng/ 
>>>>
>>>>
>>>> Any help greatly appreciated,
>>>>
>>>> Johan.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 2018-12-23 13:12, [email protected] wrote:
>>>>> Hi there,
>>>>>
>>>>> We have one simple requirement:
>>>>>
>>>>> To accurately record how much bandwidth each user is using, across 
>>>>> our
>>>>> several sites, over a day / week / month / year. Realtime data 
>>>>> nice to
>>>>> have but not necessary.
>>>>>
>>>>> I say 'simple requirement' however having tried many ways to achieve
>>>>> this over years its been anything but simple. (For us anyhow.)
>>>>>
>>>>> With ntopng now being able to record historical data we're feeling
>>>>> encouraged to try ntop again.
>>>>>
>>>>> As such we've acquired the needed licenses, instructed our 
>>>>> Mikrotik to
>>>>> send NetFlow to the Windows PC running nProbe & ntopng, and created
>>>>> the needed license file.
>>>>>
>>>>> However I cannot figure out how to start nprobe service to capture 
>>>>> the
>>>>> Mikrotik flows and send them to ntopng.
>>>>>
>>>>> What are the correct Windows cmd's to start nprobe & ntopng, to
>>>>> capture NetFlow from Mikrotik please?
>>>>>
>>>>> Lots of tutorials like the one below for starting on Linux but no so
>>>>> much on Windows:
>>>>>
>>>>> https://www.ntop.org/ntopng/how-to-analyse-mikrotik-traffic-using-ntopng/ 
>>>>> We seem to need the Windows equivalent of the below however:
>>>>>
>>>>> nprobe -i none -n none -3 2055 --zmq tcp://127.0.0.1:1234
>>>>> ntopng -i tcp://127.0.0.1:1234
>>>>>
>>>>> Help greatly appreciated,
>>>>>
>>>>> Best,
>>>>>
>>>>> Johan.
>>>>>
>>>>> _______________________________________________
>>>>> Ntop mailing list
>>>>> [email protected]
>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>> _______________________________________________
>>>> Ntop mailing list
>>>> [email protected]
>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>> _______________________________________________
>>> Ntop mailing list
>>> [email protected]
>>> http://listgateway.unipi.it/mailman/listinfo/ntop
> _______________________________________________
> Ntop mailing list
> [email protected]
> http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop