Re: Segfault building Snort3 b245 with --enable-tcmalloc flag
Michael Altizer via Snort-devel <[email protected]> Mon, 18 Jun 2018 16:43:21 -0400
| Newsgroups | gmane.comp.security.ids.snort.devel |
|---|---|
| Message-ID | <[email protected]> |
Good to hear that switching to a newer version of gperftools solved the problem, although that's still a little troubling. I don't have exact numbers, but at least in one of the more contrived tests we were doing, it looked like a 7-12% reduction in processing time. On 06/15/2018 06:19 AM, Noah Dietrich wrote: > Thank you Michael for the deep-dive analysis of this issue. It looks > that installing gperftools 2.7 from github fixes the problem, although > I haven't done more than just run snort with the -V flag. The steps it > took for this are below. > Has anyone done an analysis of the performance benefits of using > tcmalloc? I'd assume that you're getting more speed for the trade-off > of increased memory usage, but it would be interesting if someone has > some results to look at. > > Thanks, > Noah > > > *# remove gperftools 2.5 from system:* > sudo apt-get remove libgoogle-perftools-dev > sudo apt-get autoremove > > *$ install gperftools 2.7 from github from source:* > cd ~ > wget > https://github.com/gperftools/gperftools/releases/download/gperftools-2.7/gperftools-2.7.tar.gz > tar xzvf gperftools-2.7.tar.gz > cd gperftools-2.7/ > ./configure > make > sudo make install > > *# get, build, and install Snort3* > cd ~/snort_src/ > git clone https://github.com/snort3/snort3.git > cd snort3/ > ./configure_cmake.sh --prefix=/opt/snort3a4 --enable-tcmalloc > cd build/ > make > sudo make install > > *# Verify libraries linked and that snort3 runs:* > noah@snort18x:~$ ldd /opt/snort3a4/bin/snort > ... > libtcmalloc.so.4 => /usr/local/lib/libtcmalloc.so.4 (0x00007fc7ce334000) > ... > > noah@snort18x~$ /opt/snort3a4/bin/snort -V > > ,,_ -*> Snort++ <*- > o" )~ Version 3.0.0 (Build 245) from 2.9.11 > '''' By Martin Roesch & The Snort Team > http://snort.org/contact#team > Copyright (C) 2014-2018 Cisco and/or its affiliates. All > rights reserved. > Copyright (C) 1998-2013 Sourcefire, Inc., et al. > Using DAQ version 2.2.2 > Using LuaJIT version 2.1.0-beta3 > Using OpenSSL 1.1.0g 2 Nov 2017 > Using libpcap version 1.8.1 > Using PCRE version 8.39 2016-06-14 > Using ZLIB version 1.2.11 > Using FlatBuffers 1.9.0 > Using Hyperscan version 4.7.0 2018-06-13 > Using LZMA version 5.2.2 > > > On Thu, Jun 14, 2018 at 6:16 AM, Michael Altizer via Snort-devel > <[email protected] <mailto:[email protected]>> wrote: > > I can reproduce it on my Ubuntu 18.04 system, but it doesn't look > like a bug in Snort. From combing through the backtrace and > stepping through the code, it looks like a messy bug in the > interaction between tcmalloc, libhwloc, and libltdl. libhwloc > uses libltdl to iterate over and load all of its plugins. > Somewhere in the middle of "canonicalizing" the paths of the > plugins it is evaluating, libltdl's loop gets horribly confused > and starts writing over itself forever until get gets a SEGV. In > stepping through it in a debugger, it appears that the malloc it > calls to create the new character array to write into returns an > address 16 bytes into the original path. I didn't dig into the > assembly, but it might actually be a compiler issue where it > optimized libltdl around assumptions using builtin malloc and then > it gets hosed by tcmalloc doing the actual allocation. > Alternatively, something in what it's doing is buggily stomping on > tcmalloc's internal metadata, allowing it to give out the "path" > memory again even while it's being used and read. Or, who knows, > maybe there's a bug in the gperftools 2.5 that Ubuntu is packing > (we tested with 2.7)? Not that it's a great solution, but I can > make the crash go away by uninstalling the hwloc-plugins package. > > On 06/13/2018 12:50 AM, Noah Dietrich wrote: >> Hello, >> >> thanks for tracking that down. I have verified that the segfault >> issue also occurs when building on Ubuntu 16 x64. >> >> Changing the command to: >> * ./configure_cmake.sh --prefix=/opt/snort --enable-tcmalloc >> --disable-docs* >> does not generate any errors and allows for successful build and >> installation. >> >> however, running snort3 when*-enable-tcmalloc *is included as >> above generates a segfault; >> noah@snort18x:~/snort_src/snort3/build$*/opt/snort/bin/snort -V* >> Segmentation fault (core dumped) >> >> Please let me know if any further info is needed. >> thanks >> Noah >> >> >> >> >> On Tue, Jun 12, 2018 at 8:32 PM, Y M via Snort-devel >> <[email protected] >> <mailto:[email protected]>> wrote: >> >> After looking at the errors, which I received too, the build >> fails at generating the text documentation at the target >> prefix. Using the --disable-text-docs alone did not stop the >> documentation generation. However, disabling all types of >> documentation generation resulted in a successful build. >> >> # ./configure_cmake.sh --prefix=/usr/local/snort >> --enable-tcmalloc --disable-text-docs --disable-html-docs >> --disable-pdf-docs --disable-docs >> >> ... >> [ 99%] Built target preprocessor_states >> Scanning dependencies of target snort2lua >> [100%] Building CXX object >> tools/snort2lua/CMakeFiles/snort2lua.dir/init_state.cc.o >> [100%] Building CXX object >> tools/snort2lua/CMakeFiles/snort2lua.dir/snort2lua.cc.o >> [100%] Linking CXX executable snort >> [100%] Linking CXX executable snort2lua >> [100%] Built target snort2lua >> [100%] Built target snort >> >> This doesn't particularly answers your question, but >> hopefully can help you move along. >> >> Thanks. >> YM >> >> ------------------------------------------------------------------------ >> *From:* Snort-devel <[email protected] >> <mailto:[email protected]>> on behalf of Y >> M via Snort-devel <[email protected] >> <mailto:[email protected]>> >> *Sent:* Tuesday, June 12, 2018 8:25 PM >> *To:* [email protected] >> <mailto:[email protected]> >> *Subject:* Re: [Snort-devel] Segfault building Snort3 b245 >> with --enable-tcmalloc flag >> Interesting. Snort build 245 with tcmalloc over CetnOS 7 and >> FreeBSD 11 was successful. There have been recent changes in >> Ubuntu 18 that are making several apps too unhappy. I will do >> some testing on my side and report back if I come up with >> anything useful. >> >> YM >> >> ------------------------------------------------------------------------ >> *From:* Snort-devel <[email protected] >> <mailto:[email protected]>> on behalf of >> Noah Dietrich <[email protected] >> <mailto:[email protected]>> >> *Sent:* Tuesday, June 12, 2018 5:26 PM >> *To:* [email protected] >> <mailto:[email protected]> >> *Subject:* [Snort-devel] Segfault building Snort3 b245 with >> --enable-tcmalloc flag >> Hello, >> >> when building Snort 3 245 with the --enable-tcmalloc flag, >> snort throws a segfault. >> >> *Environment: *git clone of the latest Snort3 (b245) on >> Ubuntu 18 x64 in a VMware virtual Machine. >> >> *Commands:* >> cd ~/snort_src >> git clone https://github.com/snort3/snort3.git >> <https://github.com/snort3/snort3.git> >> cd ./snort3 >> ./configure_cmake.sh --prefix=/usr/local --enable-tcmalloc >> cd build >> make >> >> *Error:* >> ... >> Scanning dependencies of target all_built_sources >> [ 97%] Documents: building commands.txt with >> Segmentation fault (core dumped) >> doc/CMakeFiles/all_built_sources.dir/build.make:80: recipe >> for target 'doc/commands.txt' failed >> make[2]: *** [doc/commands.txt] Error 139 >> make[2]: *** Deleting file 'doc/commands.txt' >> CMakeFiles/Makefile2:6091: recipe for target >> 'doc/CMakeFiles/all_built_sources.dir/all' failed >> make[1]: *** [doc/CMakeFiles/all_built_sources.dir/all] Error 2 >> Makefile:151: recipe for target 'all' failed >> make: *** [all] Error 2 >> >> >> >> When you remove the --enable-tcmalloc flag, snort compiles >> and runs without error: >> >> ,,_ -*> Snort++ <*- >> o" )~ Version 3.0.0 (Build 245) from 2.9.11 >> '''' By Martin Roesch & The Snort Team >> http://snort.org/contact#team >> Copyright (C) 2014-2018 Cisco and/or its >> affiliates. All rights reserved. >> Copyright (C) 1998-2013 Sourcefire, Inc., et al. >> Using DAQ version 2.2.2 >> Using LuaJIT version 2.1.0-beta3 >> Using OpenSSL 1.1.0g 2 Nov 2017 >> Using libpcap version 1.8.1 >> Using PCRE version 8.39 2016-06-14 >> Using ZLIB version 1.2.11 >> Using FlatBuffers 1.9.0 >> Using Hyperscan version 4.7.0 2018-06-12 >> Using LZMA version 5.2.2 >> >> >> please let me know if you need any other information. >> thank you, >> Noah >> >> >> >> _______________________________________________ >> Snort-devel mailing list >> [email protected] <mailto:[email protected]> >> https://lists.snort.org/mailman/listinfo/snort-devel >> <https://lists.snort.org/mailman/listinfo/snort-devel> >> >> Please visit http://blog.snort.org for the latest news about >> Snort! >> >> >> >> >> _______________________________________________ >> Snort-devel mailing list >> [email protected] <mailto:[email protected]> >> https://lists.snort.org/mailman/listinfo/snort-devel >> <https://lists.snort.org/mailman/listinfo/snort-devel> >> >> Please visithttp://blog.snort.org for the latest news about Snort! > > > > _______________________________________________ > Snort-devel mailing list > [email protected] <mailto:[email protected]> > https://lists.snort.org/mailman/listinfo/snort-devel > <https://lists.snort.org/mailman/listinfo/snort-devel> > > Please visit http://blog.snort.org for the latest news about Snort! > > _______________________________________________ Snort-devel mailing list [email protected] https://lists.snort.org/mailman/listinfo/snort-devel Please visit http://blog.snort.org for the latest news about Snort!