Re: Critical: Memory leak in snort 2.9 and FreeBSD >= 10.4
James via Snort-devel <[email protected]> Fri, 25 May 2018 09:05:34 +0300
| Newsgroups | gmane.comp.security.ids.snort.devel |
|---|---|
| Message-ID | <CACgEDaSfDoasi=1xfvU0xLXd_KzxbcOO6Vacv6UDPR1Wcn=uDQ@mail.gmail.com> |
Please unsubscribe On Thu, 24 May 2018 at 17:12 <[email protected]> wrote: > > Hi list (and Zi from FreeBSD ports)! > > I sent the below question to snort-users in March but got no response. > > Now I have upgraded a couple of more systems. This time from FreeBSD > 10.3 to 10.4 (not from 10.3 to 11.1 as before) and snort start leaking > memory on all of them, just as it did in FreeBSD 11.1! > > > FreeBSD 10.3: snort --version > ,,_ -*> Snort! <*- > o" )~ Version 2.9.11.1 (Build 268) > '''' By Martin Roesch & The Snort Team: > http://www.snort.org/contact#team > Copyright (C) 2014-2017 Cisco and/or its affiliates. All > rights > reserved. > Copyright (C) 1998-2013 Sourcefire, Inc., et al. > Using libpcap version 1.8.1 > Using PCRE version: 8.40 2017-01-11 > Using ZLIB version: 1.2.8 > Snort is working fine. No memory leak. :-) > > > FreeBSD 10.4: snort --version > ,,_ -*> Snort! <*- > o" )~ Version 2.9.11.1 (Build 268) > '''' By Martin Roesch & The Snort Team: > http://www.snort.org/contact#team > Copyright (C) 2014-2017 Cisco and/or its affiliates. All > rights > reserved. > Copyright (C) 1998-2013 Sourcefire, Inc., et al. > Using libpcap version 1.8.1 > Using PCRE version: 8.42 2018-03-20 > Using ZLIB version: 1.2.11 > Snort has a memory leak. :-( > > > FreeBSD 11.1: snort --version > ,,_ -*> Snort! <*- > o" )~ Version 2.9.11.1 (Build 268) > '''' By Martin Roesch & The Snort Team: > http://www.snort.org/contact#team > Copyright (C) 2014-2017 Cisco and/or its affiliates. All rights > reserved. > Copyright (C) 1998-2013 Sourcefire, Inc., et al. > Using libpcap version 1.8.1 > Using PCRE version: 8.40 2017-01-11 > Using ZLIB version: 1.2.11 > Snort has a memory leak. :-( > > > The snort version+build is exactly the same in all three OS versions. > Libpcap is the same. > PCRE is the same between 10.3 and the test on 11.1. > ZLIB has changed from v1.2.8 to v1.2.11 in both cases. > > So in 10.3 everything is working fine. > In 10.4 and 11.1, with ZLIB 1.2.11, there is a memory leak. > > I suspect there's a problem in snort together with ZLIB 1.2.11. > > > > On a sensor which see lots of traffic, all of its 16 GB RAM is consumed by > snort in roughly 30 minutes. (swap get full and things crash) > > This problem is reproduceable all the time, on all upgraded boxes. > > > > Can you developers please take a look at this? > > Let me know if you need more information/testing from me. > I'm no programmer, but I can compile snort with debugging symbols and I > can run gdb commands if you provide them. > > > > Some more info: > > Arch: amd64 > 12 CPUs: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz > RAM: 16 GB > > I'm building snort from FreeBSD ports, using poudriere. > > Build options: > ---Begin OPTIONS List--- > ===> The following configuration options are available for > snort-2.9.11.1_1: > APPID=off: Build with application id support (EXPERIMENTAL) > DOCS=on: Build and/or install documentation > FILEINSPECT=off: Build with extended file inspection features > (EXPERIMENTAL) > GRE=off: GRE support > HA=off: Enable high-availability state sharing (EXPERIMENTAL) > IPV6=off: IPv6 in snort.conf > LRGPCAP=off: Pcaps larger than 2GB > NONETHER=off: Non-Ethernet Decoders > NORMALIZER=on: Normalizer > PERFPROFILE=on: Performance profiling > SOURCEFIRE=on: Sourcefire recommended build options > ====> Depend on 3rd party addons > BARNYARD=off: Depend on barnyard2 (supports also snortsam) > PULLEDPORK=off: Depend on pulledpork > ====> Developer options > DBGSNORT=off: Enable debugging symbols+core dumps > ===> Use 'make config' to modify these settings > ---End OPTIONS List--- > > Snort daq is running in pcap mode. > Snort is running in passive mode. > Snort is using search-method ac-split. > preprocessor http_inspect: global iis_unicode_map > /usr/local/etc/snort/unicode.map 1252 compress_depth 65535 > decompress_depth 65535 max_gzip_mem 1000000 memcap 603979776 > http_inspect: unlimited_decompress is enabled > http_inspect: inspect_gzip is enabled > > > PS: > libz is part of the FreeBSD base system, it is not a port, so I can't hold > it back when upgrading the OS. > > /Elof > > > > ---------- Forwarded message ---------- > From: [email protected] > To: snort-users mailinglist <[email protected]> > Date: Fri, 2 Mar 2018 16:35:24 +0100 (CET) > Subject: Memory leak in snort 2.9 and FreeBSD 11? > > > Critical issue. > > > After I upgraded a few FreeBSD 10.3 machines to 11.1, snort has begun > eating > memory until it crashes. > This seem to be happening on all upgraded machines, all the time. > > > I suspect there's a memory leak somewhere. > > > Example of 40 minutes after I start snort. I run: > > while true > do > ps faxuw | egrep "^USER|/[s]nort " > echo "---" > top | grep -B3 ^Swap > echo "---" > sleep 120 > done > > Here you see it start to consume RAM: > #### > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND > snort 7337 98.4 1.4 356096 232376 - Rs 14:40 0:01.35 snort > --- > Mem: 550M Active, 174M Inact, 1585M Wired, 13G Free > ARC: 711M Total, 153M MFU, 545M MRU, 1600K Anon, 4623K Header, 7465K Other > 593M Compressed, 1647M Uncompressed, 2.78:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND > snort 7337 54.3 7.2 3002112 1199900 - Rs 14:40 1:04.85 snort > --- > Mem: 1499M Active, 191M Inact, 1670M Wired, 12G Free > ARC: 763M Total, 178M MFU, 572M MRU, 1308K Anon, 4860K Header, 7441K Other > 646M Compressed, 1779M Uncompressed, 2.75:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND > snort 7337 54.3 12.9 5644032 2155388 - Ss 14:40 2:07.16 snort > --- > Mem: 2427M Active, 191M Inact, 1682M Wired, 11G Free > ARC: 777M Total, 178M MFU, 585M MRU, 1344K Anon, 4935K Header, 7513K Other > 661M Compressed, 1815M Uncompressed, 2.75:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND > snort 7337 54.5 18.7 8275712 3114844 - Rs 14:40 3:09.86 snort > --- > Mem: 3357M Active, 192M Inact, 1768M Wired, 10G Free > ARC: 821M Total, 194M MFU, 614M MRU, 556K Anon, 5195K Header, 7513K Other > 711M Compressed, 1942M Uncompressed, 2.73:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 54.2 24.3 10862336 4053456 - Rs 14:40 4:11.28 snort > --- > Mem: 4270M Active, 194M Inact, 1778M Wired, 9646M Free > ARC: 890M Total, 299M MFU, 578M MRU, 400K Anon, 5243K Header, 7442K Other > 726M Compressed, 1978M Uncompressed, 2.73:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 56.2 29.9 13461248 4998904 - Ss 14:40 5:13.96 snort > --- > Mem: 5188M Active, 195M Inact, 1798M Wired, 8708M Free > ARC: 826M Total, 261M MFU, 551M MRU, 528K Anon, 5300K Header, 7410K Other > 741M Compressed, 2015M Uncompressed, 2.72:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 53.1 35.5 16033536 5929068 - Rs 14:40 6:15.56 snort > --- > Mem: 6091M Active, 195M Inact, 1823M Wired, 7779M Free > ARC: 870M Total, 255M MFU, 602M MRU, 276K Anon, 5391K Header, 7521K Other > 755M Compressed, 2051M Uncompressed, 2.72:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 51.2 41.1 18605824 6867124 - Ss 14:40 7:16.95 snort > --- > Mem: 7002M Active, 195M Inact, 1848M Wired, 6843M Free > ARC: 885M Total, 221M MFU, 651M MRU, 288K Anon, 5454K Header, 7515K Other > 769M Compressed, 2087M Uncompressed, 2.71:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 54.0 46.8 21212928 7810464 - Rs 14:40 8:19.24 snort > --- > Mem: 7924M Active, 195M Inact, 1943M Wired, 5826M Free > ARC: 936M Total, 218M MFU, 703M MRU, 952K Anon, 5766K Header, 7829K Other > 824M Compressed, 2222M Uncompressed, 2.70:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 53.0 52.5 23834368 8762624 - Ss 14:40 9:21.14 snort > --- > Mem: 8849M Active, 195M Inact, 1954M Wired, 4891M Free > ARC: 951M Total, 218M MFU, 718M MRU, 920K Anon, 5827K Header, 7814K Other > 838M Compressed, 2259M Uncompressed, 2.69:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 50.9 58.2 26472192 9721948 - Rs 14:40 10:23.33 snort > --- > Mem: 9782M Active, 195M Inact, 1971M Wired, 3941M Free > ARC: 965M Total, 220M MFU, 731M MRU, 920K Anon, 5882K Header, 7822K Other > 853M Compressed, 2295M Uncompressed, 2.69:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 50.3 63.9 29105920 10676928 - Rs 14:40 11:26.69 > snort > --- > Mem: 10G Active, 204M Inact, 2116M Wired, 2852M Free > ARC: 1062M Total, 274M MFU, 774M MRU, 1052K Anon, 6252K Header, 7728K Other > 939M Compressed, 2510M Uncompressed, 2.67:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 55.5 69.6 31735552 11617828 - Rs 14:40 12:28.81 > snort > --- > Mem: 11G Active, 224M Inact, 2132M Wired, 1921M Free > ARC: 1092M Total, 300M MFU, 777M MRU, 1072K Anon, 6285K Header, 7591K Other > 954M Compressed, 2547M Uncompressed, 2.67:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 55.1 75.1 34324224 12545340 - Rs 14:40 13:30.78 > snort > --- > Mem: 12G Active, 230M Inact, 2222M Wired, 931M Free > ARC: 1117M Total, 325M MFU, 777M MRU, 1204K Anon, 6542K Header, 7580K Other > 1010M Compressed, 2686M Uncompressed, 2.66:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 55.2 80.4 36824832 13427160 - Ss 14:40 14:32.93 > snort > --- > Mem: 12G Active, 263M Inact, 1428M Laundry, 1929M Wired, 367M Free > ARC: 1156M Total, 325M MFU, 816M MRU, 1068K Anon, 6614K Header, 7613K Other > 1025M Compressed, 2722M Uncompressed, 2.66:1 Ratio > Swap: 4096M Total, 4096M Free > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 54.6 82.6 39397120 13800500 - Rs 14:40 15:35.14 > snort > --- > Mem: 12G Active, 110M Inact, 1784M Laundry, 1937M Wired, 191M Free > ARC: 1159M Total, 323M MFU, 822M MRU, 936K Anon, 6648K Header, 7488K Other > 1039M Compressed, 2758M Uncompressed, 2.65:1 Ratio > Swap: 4096M Total, 685M Used, 3411M Free, 16% Inuse > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 53.8 82.6 41801472 13800476 - Ss 14:40 16:35.38 > snort > --- > Mem: 11G Active, 107M Inact, 1846M Laundry, 1990M Wired, 182M Free > ARC: 1208M Total, 338M MFU, 854M MRU, 1552K Anon, 6866K Header, 7564K Other > 1087M Compressed, 2878M Uncompressed, 2.65:1 Ratio > Swap: 4096M Total, 1508M Used, 2587M Free, 36% Inuse > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 54.5 82.2 44414720 13735076 - Rs 14:40 17:38.62 > snort > --- > Mem: 12G Active, 34M Inact, 1766M Laundry, 2014M Wired, 292M Free > ARC: 1221M Total, 337M MFU, 868M MRU, 1436K Anon, 6944K Header, 7616K Other > 1102M Compressed, 2914M Uncompressed, 2.64:1 Ratio > Swap: 4096M Total, 2551M Used, 1544M Free, 62% Inuse > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 7337 54.6 82.1 46968576 13714884 - Ss 14:40 18:42.19 > snort > --- > Mem: 11G Active, 64M Inact, 1884M Laundry, 2058M Wired, 259M Free > ARC: 1252M Total, 352M MFU, 885M MRU, 672K Anon, 7150K Header, 7525K Other > 1153M Compressed, 3043M Uncompressed, 2.64:1 Ratio > Swap: 4096M Total, 3457M Used, 639M Free, 84% Inuse > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND > snort 7337 2.2 0.0 0 16 - R<Es 14:40 19:49.65 snort > --- > Mem: 12G Active, 5252K Inact, 786M Laundry, 1937M Wired, 461M Free > ARC: 1319M Total, 384M MFU, 920M MRU, 967K Anon, 7275K Header, 7534K Other > 1181M Compressed, 3112M Uncompressed, 2.63:1 Ratio > Swap: 4096M Total, 4096M Used, K Free, 100% Inuse > --- > > > > After maxing out at 82.2% or RAM for a copuple of minutes, the process is > automatically killed by the system: > > Mar 2 15:17:48 chobetsu-10 kernel: swap_pager: out of swap space > Mar 2 15:17:48 chobetsu-10 kernel: swap_pager_getswapspace(11): failed > Mar 2 15:20:18 chobetsu-10 kernel: pid 7337 (snort), uid 100, was killed: > out > of swap space > > > > > > ...the while-loop continues... > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND > <no snort process started> > --- > Mem: 272M Active, 12M Inact, 2079M Wired, 13G Free > ARC: 1376M Total, 516M MFU, 844M MRU, 1552K Anon, 7347K Header, 7524K Other > 1197M Compressed, 3150M Uncompressed, 2.63:1 Ratio > Swap: 4096M Total, 225M Used, 3871M Free, 5% Inuse > --- > > > > > I startup snort again. > ...the while-loop continues... > > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND > snort 14277 57.3 5.1 2055936 856740 - Rs 15:23 0:43.21 snort > --- > Mem: 1090M Active, 56M Inact, 2136M Wired, 12G Free > ARC: 1330M Total, 435M MFU, 879M MRU, 1432K Anon, 7487K Header, 7824K Other > 1213M Compressed, 3193M Uncompressed, 2.63:1 Ratio > Swap: 4096M Total, 190M Used, 3906M Free, 4% Inuse > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 14277 57.9 11.0 4714240 1829656 - Ss 15:23 1:49.20 snort > --- > Mem: 2046M Active, 97M Inact, 2366M Wired, 11G Free > ARC: 1401M Total, 438M MFU, 945M MRU, 1296K Anon, 8041K Header, 8700K Other > 1314M Compressed, 3438M Uncompressed, 2.62:1 Ratio > Swap: 4096M Total, 172M Used, 3924M Free, 4% Inuse > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 14277 54.3 16.8 7362304 2798096 - Ss 15:23 2:54.43 snort > --- > Mem: 2985M Active, 98M Inact, 2378M Wired, 10G Free > ARC: 1450M Total, 438M MFU, 994M MRU, 1468K Anon, 8120K Header, 8754K Other > 1329M Compressed, 3475M Uncompressed, 2.62:1 Ratio > Swap: 4096M Total, 172M Used, 3924M Free, 4% Inuse > --- > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > COMMAND > snort 14277 60.4 22.6 10053376 3776916 - Ss 15:23 4:00.53 snort > > ...and so on until pid 14277 gets out of swap. > > > > I'm running: > FreeBSD 11.1-RELEASE-p4 amd64 > Snort Version 2.9.11.1 (Build 268) > Using libpcap version 1.8.1 > Using PCRE version: 8.40 2017-01-11 > Using ZLIB version: 1.2.11 > > > > Has anyone else observed this? > > Any tips on how I can help debug this further? > > A SIGHUP don't reveal anything about what subsystem is eating memory. > > > > > I've tried setting two memcaps to a really low value, to see if the > process > stop increasing in size: > preprocessor stream5_global: ......... memcap 128257751 > preprocessor http_inspect: global .... memcap 85505167 > No luck. The snort process grows to >80% of system RAM and then dies. > > > > Some info about the snort conf: > > Running in IDS mode > Detection: Search-Method = AC-Full-Q > Split Any/Any group = enabled > Search-Method-Optimizations = enabled > Maximum pattern length = 20 > > +-----------------------[detection-filter-config]------------------------------ > memory-cap : 1048576 bytes > > +-----------------------[rate-filter-config]----------------------------------- > memory-cap : 1048576 bytes > > +-----------------------[event-filter-config]---------------------------------- > memory-cap : 1048576 bytes > Rule application order: > pass->activation->dynamic->drop->alert->log->sdrop->reject > pcap DAQ configured to passive. > chroot > Set gid to 100 > Set uid to 100 > > > > The same snort version was running just fine on FreeBSD 10.3 before the > upgrade. > > > /Elof > _______________________________________________ > Snort-devel mailing list > [email protected] > https://lists.snort.org/mailman/listinfo/snort-devel > > Please visit http://blog.snort.org for the latest news about Snort! > _______________________________________________ Snort-devel mailing list [email protected] https://lists.snort.org/mailman/listinfo/snort-devel Please visit http://blog.snort.org for the latest news about Snort!