Re: Bus errors and segmentation faults after upgrade to 2.9.7.3 and daq 2.0.5
"Hui Cao (huica)" <[email protected]>
| Newsgroups | gmane.comp.security.ids.snort.devel |
|---|---|
| Message-ID | <D195E975.2F5AC%[email protected]> |
That¹s cool. All looks good to me. No need to do more things... Best, Hui On 6/4/15, 11:35 AM, "[email protected]" <[email protected]> wrote: > >Hi Hui. > >That much I know. It is the debugging steps I'm curious about. > >(I think you forgot one important first command: continue ) > > >Is this a good start: > >gdb /path/to/snort 1222 >(gdb) set logging file gdb-snort.txt >(gdb) set logging on >(gdb) continue > ><wait for it to crash> > >(gdb) backtrace full >(gdb) info registers >(gdb) x/16i $pc >(gdb) thread apply all backtrace >(gdb) quit > >Email the report. > > >Should I prepare more stuff before the 'continue'? >Like "handle SIG33 pass nostop noprint" or something? > >/Elof > > >On Thu, 4 Jun 2015, Hui Cao (huica) wrote: > >> Try >> >> Assume snort pid is 1222 >> >> gdb /path/to/snort 1222 >> >> Best, >> Hui. >> On 6/4/15, 10:37 AM, "[email protected]" <[email protected]> wrote: >> >>> >>> An update: >>> >>> On a sensor where snort crashed with signal 6 three times, I downgraded >>> daq to 2.0.4_1 and rebooted the machine to rule out if the problem seem >>> to >>> be in 'snort' or 'daq'. >>> >>> With snort 2.9.7.3 and daq 2.0.4_1 I got signal 6 again. >>> >>> >>> This make me believe that there's something wrong in snort 2.9.7.3 and >>> not in daq 2.0.5. >>> >>> >>> >>> On this sensor I have now done the opposite, upgraded daq to 2.0.5 and >>> downgraded snort to 2.9.7.2 to see if I get any more signal 6. >>> >>> On another sensor, I'm running 2.9.7.3 (compiled with debug) and daq >>> 2.0.5 >>> without chroot and uid/gid change, i.e. running as root, in order to >>> create a core file, if the problem happen again. >>> (if it doesn't happen on this sensor, I guess the problem lies >>>somewhere >>> in the chrooting code in snort. I know it has been updated between >>> 2.9.7.2 >>> and 2.9.7.3) >>> >>> >>> >>> Russ C also wrote: >>>> Elof - since this is happening frequently, you could try attaching the >>>> debugger to one of your Snort processes and wait wait for segfault. >>> >>> I know too little about debugging. :-/ Can you give me instructions or >>> point me to a guide that describes the steps I should take? >>> >>> >>> >>> /Elof >>> >>> >>> On Thu, 4 Jun 2015, [email protected] wrote: >>> >>>> >>>> Five different sensors have now had bus errors (signal 10), >>>>segmentation >>>> faults (signal 11) and even signal 6 (SIGABRT). >>>> >>>> My snort config uses both chroot and dropping user privileges, so even >>>> if >>>> I start out as root with ulimit unlimited, this doesn't seem to be in >>>> effect >>>> after the chroot/uid-change. >>>> >>>> So currently I have no core-file to debug. :-/ >>>> >>>> Anyone know how to set the ulimits for a chrooted and uid/gid-changed >>>> process in FreeBSD? >>>> >>>> /Elof >>>> >>>> >>>> On Thu, 4 Jun 2015, [email protected] wrote: >>>> >>>>> >>>>> Hi Hui! >>>>> >>>>> Yes, the dynamic engine/preproc files are updated as well. >>>>> >>>>> Last night the problem reocurred, so this seem to be reproduceable. >>>>> Good. >>>>> Then there's a good chance this problem can be sorted out. >>>>> >>>>> >>>>> A few minutes ago a signal 10 happened on another sensor (running >>>>> FreeBSD 10.1 amd64), so the problem must be in DAQ 2.0.5 or in Snort >>>>> 2.9.7.3 and not in the hardware nor in FreeBSD. >>>>> >>>>> >>>>> I will compile a debug-snort and try to generate core files. >>>>> I'll let you know the outcome next week. >>>>> >>>>> /Elof >>>>> >>>>> >>>>> On Wed, 3 Jun 2015, Hui cao wrote: >>>>> >>>>>> Hi Elof, >>>>>> >>>>>> Are snort and snort dynamic preprocessors are in sync? >>>>>> >>>>>> If so, can you help us get a backtrace from the crush? You need >>>>>> 1) build snort with ./configure --enable-debug >>>>>> 2) allowing core dump (ulimit -c unlimited) >>>>>> 3) run the snort >>>>>> 4) use "gdb snort core_file " and them type "bt" in the gdb command >>>>>> line >>>>>> >>>>>> Best, >>>>>> Hui. >>>>>> >>>>>> >>>>>> On 06/03/2015 05:51 AM, [email protected] wrote: >>>>>>> Hi all! >>>>>>> >>>>>>> This is just a report to inform that after I updated snort and DAQ >>>>>>> to the >>>>>>> latest versions, one of my sensors started throwing signal 10 (bus >>>>>>> error) >>>>>>> and signal 11 (segmentation fault). >>>>>>> >>>>>>> # uptime >>>>>>> 11:32AM up 1 day, 9:48, 1 user, load averages: 0.36, 0.37, 0.38 >>>>>>> # dmesg | grep snort >>>>>>> pid 1183 (snort), uid 100: exited on signal 11 >>>>>>> pid 16920 (snort), uid 100: exited on signal 11 >>>>>>> pid 17502 (snort), uid 100: exited on signal 11 >>>>>>> pid 18862 (snort), uid 100: exited on signal 11 >>>>>>> pid 20223 (snort), uid 100: exited on signal 11 >>>>>>> pid 20927 (snort), uid 100: exited on signal 11 >>>>>>> pid 1193 (snort), uid 100: exited on signal 11 >>>>>>> pid 2447 (snort), uid 100: exited on signal 11 >>>>>>> pid 3811 (snort), uid 100: exited on signal 10 >>>>>>> pid 7881 (snort), uid 100: exited on signal 11 >>>>>>> pid 9252 (snort), uid 100: exited on signal 10 >>>>>>> pid 25593 (snort), uid 100: exited on signal 11 >>>>>>> pid 26627 (snort), uid 100: exited on signal 11 >>>>>>> pid 56658 (snort), uid 100: exited on signal 11 >>>>>>> pid 57237 (snort), uid 100: exited on signal 10 >>>>>>> pid 58595 (snort), uid 100: exited on signal 11 >>>>>>> pid 68639 (snort), uid 100: exited on signal 11 >>>>>>> pid 70008 (snort), uid 100: exited on signal 11 >>>>>>> pid 71361 (snort), uid 100: exited on signal 10 >>>>>>> pid 72725 (snort), uid 100: exited on signal 11 >>>>>>> >>>>>>> 20 crashes in a day... >>>>>>> A reboot didn't help. >>>>>>> >>>>>>> This sensor has never behaved like this during its lifetime (1 >>>>>>>year). >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> FreeBSD 9.3 amd64 >>>>>>> >>>>>>> ,,_ -*> Snort! <*- >>>>>>> o" )~ Version 2.9.7.3 (Build 217) >>>>>>> '''' By Martin Roesch & The Snort Team: >>>>>>> http://www.snort.org/contact#team >>>>>>> Copyright (C) 2014-2015 Cisco and/or its affiliates. >>>>>>> All rights >>>>>>> reserved. >>>>>>> Copyright (C) 1998-2013 Sourcefire, Inc., et al. >>>>>>> Using libpcap version 1.4.0 >>>>>>> Using PCRE version: 8.37 2015-04-28 >>>>>>> Using ZLIB version: 1.2.8 >>>>>>> >>>>>>> daq-2.0.5 >>>>>>> >>>>>>> >>>>>>> >>>>>>> Bus errors are quite unusual in general, so I'll keep looking at >>>>>>> this, >>>>>>> trying to see if it is e.g. paging errors. >>>>>>> It doesn't look like it though: >>>>>>> # swapinfo >>>>>>> Device 1K-blocks Used Avail Capacity >>>>>>> /dev/mirror/swap 4194300 0 4194300 0% >>>>>>> >>>>>>> The machine doesn't seem to be overheated either: >>>>>>> System Temp: 30 degrees C >>>>>>> Peripheral Temp: 40 degrees C >>>>>>> CPU Temp: Low >>>>>>> >>>>>>> >>>>>>> If you need me to do something special to debug this further, let >>>>>>>me >>>>>>> know. >>>>>>> >>>>>>> >>>>>>> PS. It is only one sensor, out of 20, that behaves like this. So >>>>>>> perhaps >>>>>>> it is something in the mirrored traffic that make DAQ or snort >>>>>>>point >>>>>>> at >>>>>>> illegal memory addresses and crash. >>>>>>> Or this particular machine is having hardware issues. However, it >>>>>>>is >>>>>>> strange that those hw-issues should suddenly start right after I >>>>>>> updated >>>>>>> the software on the machine... >>>>>>> >>>>>>> When I write this, the current snort process has been alive for 5 >>>>>>> hours. >>>>>>> It's going to be interesting to see if the traffic tonight will >>>>>>> cause it >>>>>>> to crash many times again. >>>>>>> >>>>>>> /Elof >>>>>>> >>>>>>> >>>>>>> >>>>>>>-------------------------------------------------------------------- >>>>>>>-- >>>>>>> -------- >>>>>>> _______________________________________________ >>>>>>> Snort-devel mailing list >>>>>>> [email protected] >>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel >>>>>>> Archive: >>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel >>>>>>> >>>>>>> Please visit http://blog.snort.org for the latest news about Snort! >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>--------------------------------------------------------------------- >>>>>>-- >>>>>> ------- >>>>>> _______________________________________________ >>>>>> Snort-devel mailing list >>>>>> [email protected] >>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel >>>>>> Archive: >>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel >>>>>> >>>>>> Please visit http://blog.snort.org for the latest news about Snort! >>>>>> >>>>> >>>>> >>>>> >>>>>---------------------------------------------------------------------- >>>>>-- >>>>> ------ >>>>> _______________________________________________ >>>>> Snort-devel mailing list >>>>> [email protected] >>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel >>>>> Archive: >>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel >>>>> >>>>> Please visit http://blog.snort.org for the latest news about Snort! >>>>> >>>> >>>> >>>> >>>>----------------------------------------------------------------------- >>>>-- >>>> ----- >>>> _______________________________________________ >>>> Snort-devel mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/snort-devel >>>> Archive: >>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel >>>> >>>> Please visit http://blog.snort.org for the latest news about Snort! >>>> >> ------------------------------------------------------------------------------ _______________________________________________ Snort-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/snort-devel Archive: http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel Please visit http://blog.snort.org for the latest news about Snort!