Re: Bus errors and segmentation faults after upgrade to 2.9.7.3 and daq 2.0.5
"Hui Cao (huica)" <[email protected]>
| Newsgroups | gmane.comp.security.ids.snort.devel |
|---|---|
| Message-ID | <D195E52E.2F59C%[email protected]> |
Try Assume snort pid is 1222 gdb /path/to/snort 1222 Best, Hui. On 6/4/15, 10:37 AM, "[email protected]" <[email protected]> wrote: > >An update: > >On a sensor where snort crashed with signal 6 three times, I downgraded >daq to 2.0.4_1 and rebooted the machine to rule out if the problem seem >to >be in 'snort' or 'daq'. > >With snort 2.9.7.3 and daq 2.0.4_1 I got signal 6 again. > > >This make me believe that there's something wrong in snort 2.9.7.3 and >not in daq 2.0.5. > > > >On this sensor I have now done the opposite, upgraded daq to 2.0.5 and >downgraded snort to 2.9.7.2 to see if I get any more signal 6. > >On another sensor, I'm running 2.9.7.3 (compiled with debug) and daq >2.0.5 >without chroot and uid/gid change, i.e. running as root, in order to >create a core file, if the problem happen again. >(if it doesn't happen on this sensor, I guess the problem lies somewhere >in the chrooting code in snort. I know it has been updated between >2.9.7.2 >and 2.9.7.3) > > > >Russ C also wrote: >> Elof - since this is happening frequently, you could try attaching the >> debugger to one of your Snort processes and wait wait for segfault. > >I know too little about debugging. :-/ Can you give me instructions or >point me to a guide that describes the steps I should take? > > > >/Elof > > >On Thu, 4 Jun 2015, [email protected] wrote: > >> >> Five different sensors have now had bus errors (signal 10), segmentation >> faults (signal 11) and even signal 6 (SIGABRT). >> >> My snort config uses both chroot and dropping user privileges, so even >>if >> I start out as root with ulimit unlimited, this doesn't seem to be in >>effect >> after the chroot/uid-change. >> >> So currently I have no core-file to debug. :-/ >> >> Anyone know how to set the ulimits for a chrooted and uid/gid-changed >> process in FreeBSD? >> >> /Elof >> >> >> On Thu, 4 Jun 2015, [email protected] wrote: >> >>> >>> Hi Hui! >>> >>> Yes, the dynamic engine/preproc files are updated as well. >>> >>> Last night the problem reocurred, so this seem to be reproduceable. >>>Good. >>> Then there's a good chance this problem can be sorted out. >>> >>> >>> A few minutes ago a signal 10 happened on another sensor (running >>> FreeBSD 10.1 amd64), so the problem must be in DAQ 2.0.5 or in Snort >>> 2.9.7.3 and not in the hardware nor in FreeBSD. >>> >>> >>> I will compile a debug-snort and try to generate core files. >>> I'll let you know the outcome next week. >>> >>> /Elof >>> >>> >>> On Wed, 3 Jun 2015, Hui cao wrote: >>> >>>> Hi Elof, >>>> >>>> Are snort and snort dynamic preprocessors are in sync? >>>> >>>> If so, can you help us get a backtrace from the crush? You need >>>> 1) build snort with ./configure --enable-debug >>>> 2) allowing core dump (ulimit -c unlimited) >>>> 3) run the snort >>>> 4) use "gdb snort core_file " and them type "bt" in the gdb command >>>>line >>>> >>>> Best, >>>> Hui. >>>> >>>> >>>> On 06/03/2015 05:51 AM, [email protected] wrote: >>>>> Hi all! >>>>> >>>>> This is just a report to inform that after I updated snort and DAQ >>>>>to the >>>>> latest versions, one of my sensors started throwing signal 10 (bus >>>>>error) >>>>> and signal 11 (segmentation fault). >>>>> >>>>> # uptime >>>>> 11:32AM up 1 day, 9:48, 1 user, load averages: 0.36, 0.37, 0.38 >>>>> # dmesg | grep snort >>>>> pid 1183 (snort), uid 100: exited on signal 11 >>>>> pid 16920 (snort), uid 100: exited on signal 11 >>>>> pid 17502 (snort), uid 100: exited on signal 11 >>>>> pid 18862 (snort), uid 100: exited on signal 11 >>>>> pid 20223 (snort), uid 100: exited on signal 11 >>>>> pid 20927 (snort), uid 100: exited on signal 11 >>>>> pid 1193 (snort), uid 100: exited on signal 11 >>>>> pid 2447 (snort), uid 100: exited on signal 11 >>>>> pid 3811 (snort), uid 100: exited on signal 10 >>>>> pid 7881 (snort), uid 100: exited on signal 11 >>>>> pid 9252 (snort), uid 100: exited on signal 10 >>>>> pid 25593 (snort), uid 100: exited on signal 11 >>>>> pid 26627 (snort), uid 100: exited on signal 11 >>>>> pid 56658 (snort), uid 100: exited on signal 11 >>>>> pid 57237 (snort), uid 100: exited on signal 10 >>>>> pid 58595 (snort), uid 100: exited on signal 11 >>>>> pid 68639 (snort), uid 100: exited on signal 11 >>>>> pid 70008 (snort), uid 100: exited on signal 11 >>>>> pid 71361 (snort), uid 100: exited on signal 10 >>>>> pid 72725 (snort), uid 100: exited on signal 11 >>>>> >>>>> 20 crashes in a day... >>>>> A reboot didn't help. >>>>> >>>>> This sensor has never behaved like this during its lifetime (1 year). >>>>> >>>>> >>>>> >>>>> >>>>> FreeBSD 9.3 amd64 >>>>> >>>>> ,,_ -*> Snort! <*- >>>>> o" )~ Version 2.9.7.3 (Build 217) >>>>> '''' By Martin Roesch & The Snort Team: >>>>> http://www.snort.org/contact#team >>>>> Copyright (C) 2014-2015 Cisco and/or its affiliates. >>>>>All rights >>>>> reserved. >>>>> Copyright (C) 1998-2013 Sourcefire, Inc., et al. >>>>> Using libpcap version 1.4.0 >>>>> Using PCRE version: 8.37 2015-04-28 >>>>> Using ZLIB version: 1.2.8 >>>>> >>>>> daq-2.0.5 >>>>> >>>>> >>>>> >>>>> Bus errors are quite unusual in general, so I'll keep looking at >>>>>this, >>>>> trying to see if it is e.g. paging errors. >>>>> It doesn't look like it though: >>>>> # swapinfo >>>>> Device 1K-blocks Used Avail Capacity >>>>> /dev/mirror/swap 4194300 0 4194300 0% >>>>> >>>>> The machine doesn't seem to be overheated either: >>>>> System Temp: 30 degrees C >>>>> Peripheral Temp: 40 degrees C >>>>> CPU Temp: Low >>>>> >>>>> >>>>> If you need me to do something special to debug this further, let me >>>>>know. >>>>> >>>>> >>>>> PS. It is only one sensor, out of 20, that behaves like this. So >>>>>perhaps >>>>> it is something in the mirrored traffic that make DAQ or snort point >>>>>at >>>>> illegal memory addresses and crash. >>>>> Or this particular machine is having hardware issues. However, it is >>>>> strange that those hw-issues should suddenly start right after I >>>>>updated >>>>> the software on the machine... >>>>> >>>>> When I write this, the current snort process has been alive for 5 >>>>>hours. >>>>> It's going to be interesting to see if the traffic tonight will >>>>>cause it >>>>> to crash many times again. >>>>> >>>>> /Elof >>>>> >>>>> >>>>>---------------------------------------------------------------------- >>>>>-------- >>>>> _______________________________________________ >>>>> Snort-devel mailing list >>>>> [email protected] >>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel >>>>> Archive: >>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel >>>>> >>>>> Please visit http://blog.snort.org for the latest news about Snort! >>>> >>>> >>>> >>>>----------------------------------------------------------------------- >>>>------- >>>> _______________________________________________ >>>> Snort-devel mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/snort-devel >>>> Archive: >>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel >>>> >>>> Please visit http://blog.snort.org for the latest news about Snort! >>>> >>> >>> >>>------------------------------------------------------------------------ >>>------ >>> _______________________________________________ >>> Snort-devel mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/snort-devel >>> Archive: >>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel >>> >>> Please visit http://blog.snort.org for the latest news about Snort! >>> >> >> >>------------------------------------------------------------------------- >>----- >> _______________________________________________ >> Snort-devel mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/snort-devel >> Archive: >> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel >> >> Please visit http://blog.snort.org for the latest news about Snort! >> ------------------------------------------------------------------------------ _______________________________________________ Snort-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/snort-devel Archive: http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel Please visit http://blog.snort.org for the latest news about Snort!