Re: Bus errors and segmentation faults after upgrade to 2.9.7.3 and daq 2.0.5
"Hui Cao (huica)" <[email protected]>
| Newsgroups | gmane.comp.security.ids.snort.devel |
|---|---|
| Message-ID | <D195F608.2F5CD%[email protected]> |
Thanks! The issue happens on smtp preprocessor, but the so is not compiled with debug enabled. Can you recompile it with ―enable-debug ? Best, Hui. On 6/4/15, 12:10 PM, "[email protected]" <[email protected]> wrote: > >So I just had a signal 6... > >I assume I can't attach files to the mailing list, so here it is, >directly >in the mailbody. :-) > > > > > >gdb /usr/local/bin/snort 11057 > >GNU gdb 6.1.1 [FreeBSD] >Copyright 2004 Free Software Foundation, Inc. >GDB is free software, covered by the GNU General Public License, and you >are >welcome to change it and/or distribute copies of it under certain >conditions. >Type "show copying" to see the conditions. >There is absolutely no warranty for GDB. Type "show warranty" for >details. >This GDB was configured as "amd64-marcel-freebsd"... >Attaching to program: /usr/local/bin/snort, process 11057 >Reading symbols from /usr/local/lib/libdnet.so.1...done. >Loaded symbols for /usr/local/lib/libdnet.so.1 >Reading symbols from /usr/local/lib/libpcre.so.1...done. >Loaded symbols for /usr/local/lib/libpcre.so.1 >Reading symbols from /lib/libm.so.5...done. >Loaded symbols for /lib/libm.so.5 >Reading symbols from /lib/libcrypto.so.6...done. >Loaded symbols for /lib/libcrypto.so.6 >Reading symbols from /lib/libpcap.so.8...done. >Loaded symbols for /lib/libpcap.so.8 >Reading symbols from /usr/local/lib/libsfbpf.so.0...done. >Loaded symbols for /usr/local/lib/libsfbpf.so.0 >Reading symbols from /lib/libz.so.6...done. >Loaded symbols for /lib/libz.so.6 >Reading symbols from /usr/lib/liblzma.so.5...done. >Loaded symbols for /usr/lib/liblzma.so.5 >Reading symbols from /lib/libthr.so.3...done. >[New Thread 815a59400 (LWP 100459/snort)] >[New Thread 802407400 (LWP 100375/snort)] >Loaded symbols for /lib/libthr.so.3 >Reading symbols from /lib/libc.so.7...done. >Loaded symbols for /lib/libc.so.7 >Reading symbols from >/usr/local/lib/snort_dynamicengine/libsf_engine.so...done. >Loaded symbols for /usr/local/lib/snort_dynamicengine/libsf_engine.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so...done. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so...done. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so...done. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so...don >e. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so...done. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so...done. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so...done. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so...done. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so...do >ne. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so...done. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so...done. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so...done. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so...done. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so >Reading symbols from >/usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so...done. >Loaded symbols for >/usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so >Reading symbols from /libexec/ld-elf.so.1...done. >Loaded symbols for /libexec/ld-elf.so.1 >[Switching to Thread 815a59400 (LWP 100459/snort)] >0x0000000801faa40c in nanosleep () from /lib/libc.so.7 >(gdb) set logging file gdb-snort.txt >(gdb) set logging on >Copying output to gdb-snort.txt. >(gdb) continue >Continuing. > > > ><...it has just been a few minutes when I receive a SIGABRT> > > > >Program received signal SIGABRT, Aborted. >[Switching to Thread 802407400 (LWP 100375/snort)] >0x0000000801f2364c in thr_kill () from /lib/libc.so.7 > >(gdb) backtrace full >#0 0x0000000801f2364c in thr_kill () from /lib/libc.so.7 >No symbol table info available. >#1 0x0000000801fc7c4b in abort () from /lib/libc.so.7 >No symbol table info available. >#2 0x0000000801fab315 in __assert () from /lib/libc.so.7 >No symbol table info available. >#3 0x0000000805068395 in ?? () from >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so >No symbol table info available. >#4 0x0000000805068781 in ?? () from >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so >No symbol table info available. >#5 0x000000080506afd0 in ?? () from >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so >No symbol table info available. >#6 0x000000080506b85b in ?? () from >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so >No symbol table info available. >#7 0x000000080506c150 in ?? () from >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so >No symbol table info available. >#8 0x000000080506cb27 in ?? () from >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so >No symbol table info available. >#9 0x00000000004423f7 in DispatchPreprocessors (p=0x8033a3e00, >policy_id=0, policy=0x802faa000) at detect.c:136 > scb = (SessionControlBlock *) 0x8a1aad2f0 > ppn = (PreprocEvalFuncNode *) 0x8033ff0a0 > pps_enabled_foo = 1123336 > alerts_processed = true >#10 0x000000000044286d in Preprocess (p=0x8033a3e00) at detect.c:234 > retval = 0 > policy_id = 0 > policy = (SnortPolicy *) 0x802faa000 > pktcnt = 0 > snort_ticks_start = 34413886976 > snort_ticks_end = 34413888664 >#11 0x00000000004e44b8 in _flush_to_seq (tcpssn=0x80811ce50, >st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4, >dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4352 > tmp_do_detect = 1 > tmp_do_detect_content = 1 > snort_ticks_start = 37073416192 > snort_ticks_end = 37069258752 > start_seq = 846966387 > stop_seq = 1940818286 > footprint = 3644 > bytes_processed = 3644 > flushed_bytes = 3644 > pkth = {ts = {tv_sec = 100375, tv_usec = 0}, caplen = 0, pktlen = 0, >ingress_index = -1, egress_index = -1, ingress_group = -1, egress_group = >-1, flags = 0, opaque = 8, priv_ptr = 0x8a1800000, flow_id = 535241216, >address_space_id = 0} > enc_flags = 2147483648 > snort_ticks_start = 51544732022 > snort_ticks_end = 113187 >#12 0x00000000004e3e06 in flush_to_seq (tcpssn=0x80811ce50, >st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4, >dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4493 >No locals. >#13 0x00000000004e4f8e in flush_ackd (tcpssn=0x80811ce50, st=0x80811cfa8, >p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8, sp=39946, dp=6400, >dir=128) at snort_stream_tcp.c:4559 > bytes = 3644 >#14 0x00000000004e3bcb in flush_stream (tcpssn=0x80811ce50, >st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8, >sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4588 > fm = (FlushMgr *) 0x80811cfb4 >#15 0x00000000004e60b4 in FlushQueuedSegs (scb=0x8a1aad2f0, >tcpssn=0x80811ce50) at snort_stream_tcp.c:5074 > p = (Packet *) 0x8033a4900 > flushed = 1926 > tmp_pcap_hdr = {ts = {tv_sec = 1433431165, tv_usec = 321125}, caplen = >94, pktlen = 94, ingress_index = 5004089, egress_index = 0, ingress_group >= 38246208, egress_group = 8, flags = 4294960320, opaque = 32767, >priv_ptr = 0x4b4c81, flow_id = 0, address_space_id = 0} >#16 0x00000000004e61bd in TcpSessionCleanup (scb=0x8a1aad2f0, >freeApplicationData=1) at snort_stream_tcp.c:5115 > tcpssn = (TcpSession *) 0x80811ce50 >#17 0x00000000004e61ea in TcpSessionCleanupWithFreeApplicationData >(scb=0x8a1aad2f0) at snort_stream_tcp.c:5122 >No locals. >#18 0x00000000004e69e1 in StreamProcessTcp (p=0xee2a40, scb=0x8a1aad2f0, >s5TcpPolicy=0x812807000, skey=0x7fffffffe6c0) at snort_stream_tcp.c:5648 > sscc = {old_mem_in_use = 15788887, client_ip = {family = 2, bits = 32, >ip = {u6_addr8 = "\nm\027L", '\0' <repeats 11 times>, u6_addr16 = {27914, >19479, 0, 0, 0, 0, 0, 0}, u6_addr32 = {1276603658, 0, 0, 0}}}, server_ip >= {family = 2, bits = 32, ip = {u6_addr8 = "\nm\026\024", '\0' <repeats >11 times>, > u6_addr16 = {27914, 5142, 0, 0, 0, 0, 0, 0}, u6_addr32 = >{337014026, 0, 0, 0}}}, client_port = 39946, server_port = 6400, >lw_session_state = 200, lw_session_flags = 4284679, app_proto_id = 0} > tdb = {seq = 1940818286, ack = 2349672268, win = 64032, end_seq = >1940818325, ts = 0} > rc = 0 > status = 4512282 > snort_ticks_start = 34397587520 > snort_ticks_end = 140737488348864 >#19 0x00000000004b5a14 in StreamProcess (p=0xee2a40, context=0x0) at >spp_stream6.c:751 > key = {ip_l = {0, 0, 4216431, 0}, ip_h = {0, 2, 362856224, 8}, port_l = >59136, port_h = 65535, vlan_tag = 32767, protocol = 0 '\0', pad = 0 '\0', >mplsLabel = 5328944, addressSpaceId = 0, addressSpaceIdPad1 = 0} > scb = (SessionControlBlock *) 0x8a1aad2f0 > snort_ticks_start = 140737488348960 > snort_ticks_end = 34722594592 >#20 0x00000000004423f7 in DispatchPreprocessors (p=0xee2a40, policy_id=0, >policy=0x802faa000) at detect.c:136 > scb = (SessionControlBlock *) 0x8a1aad2f0 > ppn = (PreprocEvalFuncNode *) 0x815b61340 > pps_enabled_foo = 1123336 > alerts_processed = true >#21 0x000000000044286d in Preprocess (p=0xee2a40) at detect.c:234 > retval = 0 > policy_id = 0 > policy = (SnortPolicy *) 0x802faa000 > pktcnt = 0 > snort_ticks_start = 0 > snort_ticks_end = 6059431713369489410 >#22 0x00000000004351b3 in ProcessPacket (p=0xee2a40, >pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?, ft=0x0) at snort.c:1873 > verdict = DAQ_VERDICT_PASS > __func__ = "ProcessPacket" >#23 0x0000000000434ccd in PacketCallback (user=0x0, >pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?) at snort.c:1718 > inject = 0 > verdict = DAQ_VERDICT_PASS > snort_ticks_start = 34894979584 > snort_ticks_end = 34367935488 >#24 0x000000000052fe34 in pcap_process_loop () >No symbol table info available. >#25 0x0000000801429dcd in pcap_create_interface () from /lib/libpcap.so.8 >No symbol table info available. >#26 0x000000000053025f in pcap_daq_acquire () >No symbol table info available. >#27 0x000000000045a1b4 in DAQ_Acquire (max=0, callback=0x434b40 ><PacketCallback>, user=0x0) at sfdaq.c:541 > err = 0 >#28 0x0000000000437616 in PacketLoop () at snort.c:3268 > error = 0 > pkts_to_read = 0 >#29 0x00000000004337c7 in SnortMain (argc=6, argv=0x7fffffffebc0) at >snort.c:921 > tmp_ptr = 0x0 > intf = 0x8024c4540 "mon0" > daqInit = 1 >#30 0x000000000043364f in main (argc=6, argv=0x7fffffffebc0) at >snort.c:817 >No locals. >rax 0x0 0 >rbx 0x7fffffffddec 140737488346604 >rcx 0x801fc8fbc 34393067452 >rdx 0x0 0 >rsi 0x6 6 >rdi 0x18817 100375 >rbp 0x7fffffffde60 0x7fffffffde60 >rsp 0x7fffffffddd8 0x7fffffffddd8 >r8 0x0 0 >r9 0xfffffe0032ea54a8 -2198169037656 >r10 0x59 89 >r11 0x202 514 >r12 0x80811ce50 34495123024 >r13 0x8033a51b8 34413892024 >r14 0x82251deaa 34935529130 >r15 0x1ba24 113188 >rip 0x801f2364c 0x801f2364c <thr_kill+12> >eflags 0x206 518 >cs 0x43 67 >ss 0x3b 59 >ds 0x0 0 >es 0x0 0 >fs 0x0 0 >gs 0x0 0 >0x801f2364c <thr_kill+12>: jb 0x801f2364f <thr_kill+15> >0x801f2364e <thr_kill+14>: retq >0x801f2364f <thr_kill+15>: mov 0x2d6bea(%rip),%rcx # >0x8021fa240 <__nsdefaultsrc+5696> >0x801f23656 <thr_kill+22>: jmpq *%rcx >0x801f23658 <thr_kill+24>: nop >0x801f23659 <thr_kill+25>: nop >0x801f2365a <thr_kill+26>: nop >0x801f2365b <thr_kill+27>: nop >0x801f2365c <thr_kill+28>: nop >0x801f2365d <thr_kill+29>: nop >0x801f2365e <thr_kill+30>: nop >0x801f2365f <thr_kill+31>: nop >0x801f23660 <thr_self>: mov $0x1b0,%rax >0x801f23667 <thr_self+7>: mov %rcx,%r10 >0x801f2366a <thr_self+10>: syscall >0x801f2366c <thr_self+12>: jb 0x801f2366f <thr_self+15> > >Thread 2 (Thread 802407400 (LWP 100375/snort)): >#0 0x0000000801f2364c in thr_kill () from /lib/libc.so.7 >#1 0x0000000801fc7c4b in abort () from /lib/libc.so.7 >#2 0x0000000801fab315 in __assert () from /lib/libc.so.7 >#3 0x0000000805068395 in ?? () from >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so >#4 0x0000000805068781 in ?? () from >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so >#5 0x000000080506afd0 in ?? () from >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so >#6 0x000000080506b85b in ?? () from >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so >#7 0x000000080506c150 in ?? () from >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so >#8 0x000000080506cb27 in ?? () from >/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so >#9 0x00000000004423f7 in DispatchPreprocessors (p=0x8033a3e00, >policy_id=0, policy=0x802faa000) at detect.c:136 >#10 0x000000000044286d in Preprocess (p=0x8033a3e00) at detect.c:234 >#11 0x00000000004e44b8 in _flush_to_seq (tcpssn=0x80811ce50, >st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4, >dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4352 >#12 0x00000000004e3e06 in flush_to_seq (tcpssn=0x80811ce50, >st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4, >dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4493 >#13 0x00000000004e4f8e in flush_ackd (tcpssn=0x80811ce50, st=0x80811cfa8, >p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8, sp=39946, dp=6400, >dir=128) at snort_stream_tcp.c:4559 >#14 0x00000000004e3bcb in flush_stream (tcpssn=0x80811ce50, >st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8, >sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4588 >#15 0x00000000004e60b4 in FlushQueuedSegs (scb=0x8a1aad2f0, >tcpssn=0x80811ce50) at snort_stream_tcp.c:5074 >#16 0x00000000004e61bd in TcpSessionCleanup (scb=0x8a1aad2f0, >freeApplicationData=1) at snort_stream_tcp.c:5115 >#17 0x00000000004e61ea in TcpSessionCleanupWithFreeApplicationData >(scb=0x8a1aad2f0) at snort_stream_tcp.c:5122 >#18 0x00000000004e69e1 in StreamProcessTcp (p=0xee2a40, scb=0x8a1aad2f0, >s5TcpPolicy=0x812807000, skey=0x7fffffffe6c0) at snort_stream_tcp.c:5648 >#19 0x00000000004b5a14 in StreamProcess (p=0xee2a40, context=0x0) at >spp_stream6.c:751 >#20 0x00000000004423f7 in DispatchPreprocessors (p=0xee2a40, policy_id=0, >policy=0x802faa000) at detect.c:136 >#21 0x000000000044286d in Preprocess (p=0xee2a40) at detect.c:234 >#22 0x00000000004351b3 in ProcessPacket (p=0xee2a40, >pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?, ft=0x0) at snort.c:1873 >#23 0x0000000000434ccd in PacketCallback (user=0x0, >pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?) at snort.c:1718 >#24 0x000000000052fe34 in pcap_process_loop () >#25 0x0000000801429dcd in pcap_create_interface () from /lib/libpcap.so.8 >#26 0x000000000053025f in pcap_daq_acquire () >#27 0x000000000045a1b4 in DAQ_Acquire (max=0, callback=0x434b40 ><PacketCallback>, user=0x0) at sfdaq.c:541 >#28 0x0000000000437616 in PacketLoop () at snort.c:3268 >#29 0x00000000004337c7 in SnortMain (argc=6, argv=0x7fffffffebc0) at >snort.c:921 >#30 0x000000000043364f in main (argc=6, argv=0x7fffffffebc0) at >snort.c:817 > >Thread 1 (Thread 815a59400 (LWP 100459/snort)): >#0 0x0000000801faa40c in nanosleep () from /lib/libc.so.7 >#1 0x0000000801f15a58 in sleep () from /lib/libc.so.7 >#2 0x0000000801ca8078 in sleep () from /lib/libthr.so.3 >#3 0x000000000043b215 in ReloadConfigThread (data=0x0) at snort.c:5695 >#4 0x0000000801ca5dc4 in pthread_getprio () from /lib/libthr.so.3 >#5 0x0000000000000000 in ?? () >#0 0x0000000801f2364c in thr_kill () from /lib/libc.so.7 >The program is running. Quit anyway (and detach it)? (y or n) Detaching >from program: /usr/local/bin/snort, process 11057 > > > > >As gdb detached from snort, I got the signal 6 in my syslog: >2015-06-04 17:51:53 +02:00 foobar kernel: pid 11057 (snort), uid 100: >exited on signal 6 > > > > > >So, this time we got a signal 6 but during this sensor's 14 hour uptime >we've seen: >pid 1199 (snort), uid 100: exited on signal 10 >pid 4503 (snort), uid 100: exited on signal 10 >pid 5908 (snort), uid 100: exited on signal 10 >pid 11057 (snort), uid 100: exited on signal 6 > > > > > >I hope this gdb was helpful. >Let me know if it should be run again. > > > > > >This was all performed on a sensor running: > > ,,_ -*> Snort! <*- > o" )~ Version 2.9.7.3 (Build 217) > '''' By Martin Roesch & The Snort Team: >http://www.snort.org/contact#team > Copyright (C) 2014-2015 Cisco and/or its affiliates. All >rights reserved. > Copyright (C) 1998-2013 Sourcefire, Inc., et al. > Using libpcap version 1.4.0 > Using PCRE version: 8.37 2015-04-28 > Using ZLIB version: 1.2.8 > > >daq-2.0.5 > >FreeBSD 9.3-RELEASE-p13 > > >/Elof > > > > > >On Thu, 4 Jun 2015, Hui Cao (huica) wrote: > >> That¹s cool. All looks good to me. No need to do more things... >> >> Best, >> Hui >> >> On 6/4/15, 11:35 AM, "[email protected]" <[email protected]> wrote: >> >>> >>> Hi Hui. >>> >>> That much I know. It is the debugging steps I'm curious about. >>> >>> (I think you forgot one important first command: continue ) >>> >>> >>> Is this a good start: >>> >>> gdb /path/to/snort 1222 >>> (gdb) set logging file gdb-snort.txt >>> (gdb) set logging on >>> (gdb) continue >>> >>> <wait for it to crash> >>> >>> (gdb) backtrace full >>> (gdb) info registers >>> (gdb) x/16i $pc >>> (gdb) thread apply all backtrace >>> (gdb) quit >>> >>> Email the report. >>> >>> >>> Should I prepare more stuff before the 'continue'? >>> Like "handle SIG33 pass nostop noprint" or something? >>> >>> /Elof >>> >>> >>> On Thu, 4 Jun 2015, Hui Cao (huica) wrote: >>> >>>> Try >>>> >>>> Assume snort pid is 1222 >>>> >>>> gdb /path/to/snort 1222 >>>> >>>> Best, >>>> Hui. >>>> On 6/4/15, 10:37 AM, "[email protected]" <[email protected]> wrote: >>>> >>>>> >>>>> An update: >>>>> >>>>> On a sensor where snort crashed with signal 6 three times, I >>>>>downgraded >>>>> daq to 2.0.4_1 and rebooted the machine to rule out if the problem >>>>>seem >>>>> to >>>>> be in 'snort' or 'daq'. >>>>> >>>>> With snort 2.9.7.3 and daq 2.0.4_1 I got signal 6 again. >>>>> >>>>> >>>>> This make me believe that there's something wrong in snort 2.9.7.3 >>>>>and >>>>> not in daq 2.0.5. >>>>> >>>>> >>>>> >>>>> On this sensor I have now done the opposite, upgraded daq to 2.0.5 >>>>>and >>>>> downgraded snort to 2.9.7.2 to see if I get any more signal 6. >>>>> >>>>> On another sensor, I'm running 2.9.7.3 (compiled with debug) and daq >>>>> 2.0.5 >>>>> without chroot and uid/gid change, i.e. running as root, in order to >>>>> create a core file, if the problem happen again. >>>>> (if it doesn't happen on this sensor, I guess the problem lies >>>>> somewhere >>>>> in the chrooting code in snort. I know it has been updated between >>>>> 2.9.7.2 >>>>> and 2.9.7.3) >>>>> >>>>> >>>>> >>>>> Russ C also wrote: >>>>>> Elof - since this is happening frequently, you could try attaching >>>>>>the >>>>>> debugger to one of your Snort processes and wait wait for segfault. >>>>> >>>>> I know too little about debugging. :-/ Can you give me instructions >>>>>or >>>>> point me to a guide that describes the steps I should take? >>>>> >>>>> >>>>> >>>>> /Elof >>>>> >>>>> >>>>> On Thu, 4 Jun 2015, [email protected] wrote: >>>>> >>>>>> >>>>>> Five different sensors have now had bus errors (signal 10), >>>>>> segmentation >>>>>> faults (signal 11) and even signal 6 (SIGABRT). >>>>>> >>>>>> My snort config uses both chroot and dropping user privileges, so >>>>>>even >>>>>> if >>>>>> I start out as root with ulimit unlimited, this doesn't seem to be >>>>>>in >>>>>> effect >>>>>> after the chroot/uid-change. >>>>>> >>>>>> So currently I have no core-file to debug. :-/ >>>>>> >>>>>> Anyone know how to set the ulimits for a chrooted and >>>>>>uid/gid-changed >>>>>> process in FreeBSD? >>>>>> >>>>>> /Elof >>>>>> >>>>>> >>>>>> On Thu, 4 Jun 2015, [email protected] wrote: >>>>>> >>>>>>> >>>>>>> Hi Hui! >>>>>>> >>>>>>> Yes, the dynamic engine/preproc files are updated as well. >>>>>>> >>>>>>> Last night the problem reocurred, so this seem to be reproduceable. >>>>>>> Good. >>>>>>> Then there's a good chance this problem can be sorted out. >>>>>>> >>>>>>> >>>>>>> A few minutes ago a signal 10 happened on another sensor (running >>>>>>> FreeBSD 10.1 amd64), so the problem must be in DAQ 2.0.5 or in >>>>>>>Snort >>>>>>> 2.9.7.3 and not in the hardware nor in FreeBSD. >>>>>>> >>>>>>> >>>>>>> I will compile a debug-snort and try to generate core files. >>>>>>> I'll let you know the outcome next week. >>>>>>> >>>>>>> /Elof >>>>>>> >>>>>>> >>>>>>> On Wed, 3 Jun 2015, Hui cao wrote: >>>>>>> >>>>>>>> Hi Elof, >>>>>>>> >>>>>>>> Are snort and snort dynamic preprocessors are in sync? >>>>>>>> >>>>>>>> If so, can you help us get a backtrace from the crush? You need >>>>>>>> 1) build snort with ./configure --enable-debug >>>>>>>> 2) allowing core dump (ulimit -c unlimited) >>>>>>>> 3) run the snort >>>>>>>> 4) use "gdb snort core_file " and them type "bt" in the gdb >>>>>>>>command >>>>>>>> line >>>>>>>> >>>>>>>> Best, >>>>>>>> Hui. >>>>>>>> >>>>>>>> >>>>>>>> On 06/03/2015 05:51 AM, [email protected] wrote: >>>>>>>>> Hi all! >>>>>>>>> >>>>>>>>> This is just a report to inform that after I updated snort and >>>>>>>>>DAQ >>>>>>>>> to the >>>>>>>>> latest versions, one of my sensors started throwing signal 10 >>>>>>>>>(bus >>>>>>>>> error) >>>>>>>>> and signal 11 (segmentation fault). >>>>>>>>> >>>>>>>>> # uptime >>>>>>>>> 11:32AM up 1 day, 9:48, 1 user, load averages: 0.36, 0.37, 0.38 >>>>>>>>> # dmesg | grep snort >>>>>>>>> pid 1183 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 16920 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 17502 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 18862 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 20223 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 20927 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 1193 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 2447 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 3811 (snort), uid 100: exited on signal 10 >>>>>>>>> pid 7881 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 9252 (snort), uid 100: exited on signal 10 >>>>>>>>> pid 25593 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 26627 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 56658 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 57237 (snort), uid 100: exited on signal 10 >>>>>>>>> pid 58595 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 68639 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 70008 (snort), uid 100: exited on signal 11 >>>>>>>>> pid 71361 (snort), uid 100: exited on signal 10 >>>>>>>>> pid 72725 (snort), uid 100: exited on signal 11 >>>>>>>>> >>>>>>>>> 20 crashes in a day... >>>>>>>>> A reboot didn't help. >>>>>>>>> >>>>>>>>> This sensor has never behaved like this during its lifetime (1 >>>>>>>>> year). >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> FreeBSD 9.3 amd64 >>>>>>>>> >>>>>>>>> ,,_ -*> Snort! <*- >>>>>>>>> o" )~ Version 2.9.7.3 (Build 217) >>>>>>>>> '''' By Martin Roesch & The Snort Team: >>>>>>>>> http://www.snort.org/contact#team >>>>>>>>> Copyright (C) 2014-2015 Cisco and/or its affiliates. >>>>>>>>> All rights >>>>>>>>> reserved. >>>>>>>>> Copyright (C) 1998-2013 Sourcefire, Inc., et al. >>>>>>>>> Using libpcap version 1.4.0 >>>>>>>>> Using PCRE version: 8.37 2015-04-28 >>>>>>>>> Using ZLIB version: 1.2.8 >>>>>>>>> >>>>>>>>> daq-2.0.5 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Bus errors are quite unusual in general, so I'll keep looking at >>>>>>>>> this, >>>>>>>>> trying to see if it is e.g. paging errors. >>>>>>>>> It doesn't look like it though: >>>>>>>>> # swapinfo >>>>>>>>> Device 1K-blocks Used Avail Capacity >>>>>>>>> /dev/mirror/swap 4194300 0 4194300 0% >>>>>>>>> >>>>>>>>> The machine doesn't seem to be overheated either: >>>>>>>>> System Temp: 30 degrees C >>>>>>>>> Peripheral Temp: 40 degrees C >>>>>>>>> CPU Temp: Low >>>>>>>>> >>>>>>>>> >>>>>>>>> If you need me to do something special to debug this further, let >>>>>>>>> me >>>>>>>>> know. >>>>>>>>> >>>>>>>>> >>>>>>>>> PS. It is only one sensor, out of 20, that behaves like this. So >>>>>>>>> perhaps >>>>>>>>> it is something in the mirrored traffic that make DAQ or snort >>>>>>>>> point >>>>>>>>> at >>>>>>>>> illegal memory addresses and crash. >>>>>>>>> Or this particular machine is having hardware issues. However, it >>>>>>>>> is >>>>>>>>> strange that those hw-issues should suddenly start right after I >>>>>>>>> updated >>>>>>>>> the software on the machine... >>>>>>>>> >>>>>>>>> When I write this, the current snort process has been alive for 5 >>>>>>>>> hours. >>>>>>>>> It's going to be interesting to see if the traffic tonight will >>>>>>>>> cause it >>>>>>>>> to crash many times again. >>>>>>>>> >>>>>>>>> /Elof >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>------------------------------------------------------------------ >>>>>>>>>-- >>>>>>>>> -- >>>>>>>>> -------- >>>>>>>>> _______________________________________________ >>>>>>>>> Snort-devel mailing list >>>>>>>>> [email protected] >>>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel >>>>>>>>> Archive: >>>>>>>>> >>>>>>>>>http://sourceforge.net/mailarchive/forum.php?forum_name=snort-deve >>>>>>>>>l >>>>>>>>> >>>>>>>>> Please visit http://blog.snort.org for the latest news about >>>>>>>>>Snort! >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>------------------------------------------------------------------- >>>>>>>>-- >>>>>>>> -- >>>>>>>> ------- >>>>>>>> _______________________________________________ >>>>>>>> Snort-devel mailing list >>>>>>>> [email protected] >>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel >>>>>>>> Archive: >>>>>>>> >>>>>>>>http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel >>>>>>>> >>>>>>>> Please visit http://blog.snort.org for the latest news about >>>>>>>>Snort! >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>-------------------------------------------------------------------- >>>>>>>-- >>>>>>> -- >>>>>>> ------ >>>>>>> _______________________________________________ >>>>>>> Snort-devel mailing list >>>>>>> [email protected] >>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel >>>>>>> Archive: >>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel >>>>>>> >>>>>>> Please visit http://blog.snort.org for the latest news about Snort! >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>--------------------------------------------------------------------- >>>>>>-- >>>>>> -- >>>>>> ----- >>>>>> _______________________________________________ >>>>>> Snort-devel mailing list >>>>>> [email protected] >>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel >>>>>> Archive: >>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel >>>>>> >>>>>> Please visit http://blog.snort.org for the latest news about Snort! >>>>>> >>>> >> >> >> >>------------------------------------------------------------------------- >>----- >> _______________________________________________ >> Snort-devel mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/snort-devel >> Archive: >> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel >> >> Please visit http://blog.snort.org for the latest news about Snort! ------------------------------------------------------------------------------ _______________________________________________ Snort-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/snort-devel Archive: http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel Please visit http://blog.snort.org for the latest news about Snort!