Re: housekeeping - no enough memory

Emanuele Faranda <[email protected]>
Newsgroups gmane.linux.ntop.general
Message-ID <[email protected]> 
Hi Roberto,

Can you monitor the ntopng memory usage over some hours to see if that 
is the process which consumes memory?

I've just committed a fix for a memory leak on the Mac.cpp but it 
cannot affect you since you are receiving flows from nProbe. It will be 
very helpful if you could run ntopng with valgrind and send me the 
valgrind log (just the last 200 lines of log or so). Please execute
	sudo valgrind --leak-check=full ntopng /etc/ntopng.conf

Regards,
Emanuele

On Thu, Aug 24, 2017 at 6:40 PM, Roberto Alvarado 
<[email protected]> wrote:
> Hi,
> 
> The problem is after some time, here is the log from the start of 
> ntopng to the die of the process:
> 
> 23/Aug/2017 11:20:05 [Ntop.cpp:1437] Setting local networks to 
> [REMOVED]
> 23/Aug/2017 11:20:05 [Redis.cpp:109] Successfully connected to redis 
> 127.0.0.1:6379@0
> 23/Aug/2017 11:20:05 [NtopPro.cpp:183] [LICENSE] Reading license from 
> /etc/ntopng.license
> 23/Aug/2017 11:20:05 [NtopPro.cpp:118] [LICENSE] /etc/ntopng.license: 
> unable to validate Enterprise license [License mismatch error]
> 23/Aug/2017 11:20:05 [NtopPro.cpp:121] [LICENSE] /etc/ntopng.license: 
> found valid Professional license
> 23/Aug/2017 11:20:06 [MySQLDB.cpp:680] Attempting to connect to MySQL 
> for interface tcp://127.0.0.1:5556...
> 23/Aug/2017 11:20:06 [MySQLDB.cpp:720] Successfully connected to 
> MySQL [localhost:ntop] for interface tcp://127.0.0.1:5556
> 23/Aug/2017 11:20:06 [Ntop.cpp:1563] Registered interface 
> tcp://127.0.0.1:5556 [id: 0]
> 23/Aug/2017 11:20:06 [main.cpp:252] PID stored in file 
> /var/tmp/ntopng.pid
> 23/Aug/2017 11:20:06 [HTTPserver.cpp:797] Please read 
> https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to 
> enable SSL.
> 23/Aug/2017 11:20:06 [Utils.cpp:427] User changed to nobody
> 23/Aug/2017 11:20:06 [HTTPserver.cpp:851] Web server dirs 
> [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
> 23/Aug/2017 11:20:06 [HTTPserver.cpp:854] HTTP server listening on 
> port(s) 3000
> 23/Aug/2017 11:20:06 [MySQLDB.cpp:667] Disconnected from MySQL for 
> interface tcp://127.0.0.1:5556...
> 23/Aug/2017 11:20:06 [MySQLDB.cpp:680] Attempting to connect to MySQL 
> for interface tcp://127.0.0.1:5556...
> 23/Aug/2017 11:20:06 [MySQLDB.cpp:720] Successfully connected to 
> MySQL [localhost:ntop] for interface tcp://127.0.0.1:5556
> 23/Aug/2017 11:20:06 [main.cpp:320] Working directory: /data/ntopng
> 23/Aug/2017 11:20:06 [main.cpp:322] Scripts/HTML pages directory: 
> /usr/share/ntopng
> 23/Aug/2017 11:20:06 [Ntop.cpp:340] Welcome to ntopng x86_64 
> v.3.1.170821 - (C) 1998-17 ntop.org
> 23/Aug/2017 11:20:06 [Ntop.cpp:345] Built on Debian GNU/Linux 8.2 
> (jessie)
> 23/Aug/2017 11:20:06 [NtopPro.cpp:383] [LICENSE] System Id:     
> [REMOVED]
> 23/Aug/2017 11:20:06 [NtopPro.cpp:384] [LICENSE] Edition:       
> Professional
> 23/Aug/2017 11:20:06 [NtopPro.cpp:385] [LICENSE] License Type:  
> Permanent License
> 23/Aug/2017 11:20:06 [NtopPro.cpp:388] [LICENSE] License:       
> [REMOVED]
> 23/Aug/2017 11:20:06 [NtopPro.cpp:398] [LICENSE] Maintenance:   Until 
> Thu Mar 22 12:28:01 2018 [211 days left]
> 23/Aug/2017 11:20:06 [PeriodicActivities.cpp:50] Started periodic 
> activities loop...
> 23/Aug/2017 11:20:06 [NetworkInterface.cpp:2241] Started packet 
> polling on interface tcp://127.0.0.1:5556 [id: 0]...
> 23/Aug/2017 11:20:07 [MySQLDB.cpp:680] Attempting to connect to MySQL 
> for interface tcp://127.0.0.1:5556...
> 23/Aug/2017 11:20:07 [CollectorInterface.cpp:116] Collecting flows on 
> tcp://127.0.0.1:5556
> 23/Aug/2017 11:20:08 [NetworkInterface.cpp:1202] WARNING: A flow has 
> been seen from multiple exprters or from multiple IN/OUT interfaces. 
> Check exporters configuration.
> 23/Aug/2017 12:07:26 [MySQLDB.cpp:720] Successfully connected to 
> MySQL [localhost:ntop] for interface tcp://127.0.0.1:5556
> 23/Aug/2017 17:57:02 [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
> 23/Aug/2017 17:58:01 [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
> 
> 
> My config file:
> 
> root@mhost:/var/log/ntopng# cat /etc/ntopng/ntopng.conf
> -G=/var/tmp/ntopng.pid
> -i="tcp://127.0.0.1:5556"
> -d="/data/ntopng"
> -w="3000"
> #-v=5
> --max-num-hosts=300000
> --local-networks=[REMOVED]
> --dump-flows="mysql;localhost;ntop;[REMOVED]"
> root@mhost:/var/log/ntopng#
> 
> A screenshoot of my dashboard:
> 
> https://i.dmtinc.cl/i/2017/08/24/CapturaNTOPNG428fd.png
> 
> More info from syslog:
> 
> Aug 22 22:20:00 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
> Aug 22 22:21:01 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
> Aug 22 22:22:00 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
> Aug 22 22:23:01 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
> Aug 22 22:24:00 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
> Aug 22 22:24:09 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
> Aug 22 22:24:10 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/housekeeping.lua][not enough 
> memory]
> Aug 22 22:24:10 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
> Aug 22 22:24:11 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
> Aug 22 22:24:12 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
> Aug 22 22:24:13 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/housekeeping.lua][not enough 
> memory]
> Aug 22 22:24:13 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
> Aug 22 22:24:14 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
> Aug 22 22:24:15 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
> Aug 22 22:24:16 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/housekeeping.lua][not enough 
> memory]
> Aug 22 22:24:16 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
> Aug 22 22:24:17 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
> Aug 22 22:24:18 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
> Aug 22 22:24:19 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/housekeeping.lua][not enough 
> memory]
> Aug 22 22:24:19 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][/usr/share/ntopng/scripts/lua/modules/i18n/init.lua:186: 
> not enough memory]
> Aug 22 22:24:20 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][/usr/share/ntopng/scripts/lua/modules/i18n/init.lua:186: 
> not enough memory]
> Aug 22 22:24:21 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][/usr/share/ntopng/scripts/lua/modules/i18n/init.lua:186: 
> not enough memory]
> Aug 22 22:24:22 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/i18n/init.lua:186: 
> not enough memory]
> Aug 22 22:24:22 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][/usr/share/ntopng/scripts/lua/modules/i18n/init.lua:186: 
> not enough memory]
> Aug 22 22:24:23 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][/usr/share/ntopng/scripts/lua/modules/i18n/init.lua:186: 
> not enough memory]
> Aug 22 22:24:24 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][error loading module 
> 'lua_utils' from file 
> '/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
> enough memory]
> Aug 22 22:24:25 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/housekeeping.lua][error loading 
> module 'lua_utils' from file 
> '/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
> enough memory]
> Aug 22 22:24:25 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][error loading module 
> 'lua_utils' from file 
> '/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
> enough memory]
> Aug 22 22:24:26 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][error loading module 
> 'lua_utils' from file 
> '/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
> enough memory]
> Aug 22 22:24:27 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][error loading module 
> 'lua_utils' from file 
> '/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
> enough memory]
> Aug 22 22:24:28 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/housekeeping.lua][error loading 
> module 'lua_utils' from file 
> '/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
> enough memory]
> Aug 22 22:24:28 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/second.lua][error loading module 
> 'lua_utils' from file 
> '/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
> enough memory]
> Aug 23 11:20:02 mhost logger: ntopng stop
> Aug 23 11:20:05 mhost logger: ntopng start
> Aug 23 11:20:08 mhost ntopng: [NetworkInterface.cpp:1202] WARNING: A 
> flow has been seen from multiple exprters or from multiple IN/OUT 
> interfaces. Check exporters configuration.
> Aug 23 17:57:02 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
> Aug 23 17:58:01 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
> [/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory
> 
> 
> Additional preferences:
> 
> {
>   "ntopng.prefs.housekeeping_frequency": 5,
>   "ntopng.prefs.local_host_cache_duration": 3600,
>   "ntopng.prefs.local_host_max_idle": 300,
>   "ntopng.prefs.non_local_host_max_idle": 60,
>   "ntopng.prefs.flow_max_idle": 60,
>   "ntopng.prefs.active_local_host_cache_interval": 3600,
>   "ntopng.prefs.intf_rrd_raw_days": 1,
>   "ntopng.prefs.intf_rrd_1min_days": 30,
>   "ntopng.prefs.intf_rrd_1h_days": 100,
>   "ntopng.prefs.intf_rrd_1d_days": 365,
>   "ntopng.prefs.other_rrd_raw_days": 1,
>   "ntopng.prefs.other_rrd_1min_days": 30,
>   "ntopng.prefs.other_rrd_1h_days": 100,
>   "ntopng.prefs.other_rrd_1d_days": 365,
>   "ntopng.prefs.host_top_sites_creation": false,
>   "ntopng.prefs.is_local_host_cache_enabled": true,
>   "ntopng.prefs.is_active_local_host_cache_enabled": true,
>   "ntopng.prefs.tiny_flows_export_enabled": true,
>   "ntopng.prefs.max_num_alerts_per_entity": 1024,
>   "ntopng.prefs.max_num_flow_alerts": 16384,
>   "ntopng.prefs.flow_device_port_rrd_creation": false,
>   "ntopng.prefs.disable_alerts_generation": true,
>   "ntopng.prefs.probing_alerts": false,
>   "ntopng.prefs.ssl_alerts": false,
>   "ntopng.prefs.alerts_syslog": true,
>   "ntopng.prefs.alerts.slack_notifications_enabled": false,
>   "ntopng.prefs.alerts.dump_alerts_when_iface_is_alerted": false,
>   "ntopng.prefs.max_num_packets_per_tiny_flow": 3,
>   "ntopng.prefs.max_num_bytes_per_tiny_flow": 64,
>   "ntopng.prefs.safe_search_dns": 2078163920,
>   "ntopng.prefs.global_dns": 4294967295,
>   "ntopng.prefs.secondary_dns": 4294967295,
>   "ntopng.prefs.enable_captive_portal": false,
>   "ntopng.prefs.redirection_url": "http:\/\/www.ntop.org",
>   "ntopng.prefs.max_ui_strlen": 24,
>   "ntopng.prefs.host_mask": 0,
>   "ntopng.prefs.ldap.bind_dn": "",
>   "ntopng.prefs.ldap.bind_pwd": "",
>   "ntopng.prefs.ldap.search_path": "",
>   "ntopng.prefs.ldap.user_group": "",
>   "ntopng.prefs.ldap.admin_group": "",
>   "ntopng.prefs.alerts.mysql_check_open_files_limit": "1",
>   "ntopng.prefs.nagios_nsca_host": "localhost",
>   "ntopng.prefs.nagios_nsca_port": "5667",
>   "ntopng.prefs.nagios_send_nsca_executable": 
> "\/usr\/local\/nagios\/bin\/send_nsca",
>   "ntopng.prefs.nagios_send_nsca_config": 
> "\/usr\/local\/nagios\/etc\/send_nsca.cfg",
>   "ntopng.prefs.nagios_host_name": "ntopng-host",
>   "ntopng.prefs.nagios_service_name": "NtopngAlert",
>   "ntopng.prefs.default_snmp_community": "public",
>   "ntopng.prefs.google_apis_browser_key": "",
>   "ntopng.prefs.show_advanced_prefs": "true",
>   "ntopng.prefs.host_blacklist": "0"
> }
> 
> Regards!
> 
> Roberto
>> Sobre 23-08-2017 4:14:34, Emanuele Faranda <[email protected]> 
>> escribió:
>> 
>> _______________________________________________ Ntop mailing list 
>> [email protected] 
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>> Hi Roberto,
>> 
>> What is the content of your ntopng.conf file? Does the problem occur 
>> immediately or after some time? Can you post a screenshot of your 
>> ntopng dashboard?
>> 
>> Emanuele
>> 
>> On Wed, Aug 23, 2017 at 1:46 AM, Roberto Alvarado 
>> <[email protected]> wrote:
>>> Hi Folks,
>>> 
>>> My ntopng is crashing every time when the housekeeping is executed, 
>>> the message of the log is explicit “no enough memory”, but this 
>>> problems comes out after upgrade my install to the latest devel 
>>> version, do you have any clue about this, of how much memory is 
>>> needed?
>>> 
>>> 
>>> 
>>> 21/Aug/2017 20:14:22 [Lua.cpp:6214] WARNING: Script failure 
>>> [/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
>>> 21/Aug/2017 20:14:22 [Lua.cpp:6214] WARNING: Script failure 
>>> [/usr/share/ntopng/scripts/callbacks/housekeeping.lua][not enough 
>>> memory]
>>> 
>>> root@mhost:/var/log/ntopng# free -m
>>>             total       used       free     shared    buffers     
>>> cached
>>> Mem:         32163      30833       1329        428        575      
>>> 25141
>>> -/+ buffers/cache:       5116      27046
>>> Swap:        18530        399      18131
>>> root@mhost:/var/log/ntopng#
>>> 
>>> root@mhost:/var/log/ntopng# ps aux --sort -rss
>>> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME 
>>> COMMAND
>>> nobody   26971 28.1  5.8 3375504 1942968 ?     Sl   16:12  76:01 
>>> ntopng /etc/ntopng/ntopng.conf
>>> mysql    19755 15.0  1.7 1809120 588236 ?      Ssl  Aug17 1173:54 
>>> /usr/sbin/mysqld
>>> root       838  0.0  0.1  90724 51236 ?        Ss   Feb17  14:29 
>>> /lib/systemd/systemd-journald
>>> 
>>> root@mhost:/var/log/ntopng# ntopng -V
>>> v.3.1.170821    [Enterprise/Professional build]
>>> GIT rev:        
>>> dev:d13a4a219d885e47a46b008c61fb5e24b49b4514:20170821
>>> Pro rev:        r1130
>>> Built on:       Debian GNU/Linux 8.2 (jessie)
>>> System Id:
>>> Edition:        Professional
>>> License Type:   Permanent
>>> Maintenance:    Until Thu Mar 22 12:28:01 2018 [211 days left]
>>> License:
>>> root@mhost:/var/log/ntopng#
>>> 
>>> Thanks
>>> 
>>> Regards
>>> Roberto
>>> 
>>> 
>>> _______________________________________________
>>> Ntop mailing list
>>> [email protected]
>>> http://listgateway.unipi.it/mailman/listinfo/ntop

_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.