Re: [erlang-questions] beam core file R17
Matthew Evans <[email protected]> Fri, 3 Jun 2016 13:24:12 -0400
| Newsgroups | gmane.comp.lang.erlang.bugs,gmane.comp.lang.erlang.general |
|---|---|
| Message-ID | <[email protected]> |
--===============9043578613314440583== Content-Type: multipart/alternative; boundary="_5681fa72-082e-43cf-ad28-2ce04ebb0766_" --_5681fa72-082e-43cf-ad28-2ce04ebb0766_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable For what it's worth here's a gdb with symbols: =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= (gdb) bt=0A= #0 sweep_off_heap (p=3D0x7faeb8fc5488=2C fullsweep=3D-1295963664) at beam/= erl_gc.c:2353=0A= #1 0x00000000004b77d1 in do_minor (p=3D0x7faeb8fc5488=2C new_sz=3D<optimiz= ed out>=2C objv=3D<optimized out>=2C nobj=3D<optimized out>) at beam/erl_gc= .c:1166=0A= #2 0x00000000004b8479 in minor_collection (recl=3D<optimized out>=2C nobj= =3D<optimized out>=2C objv=3D<optimized out>=2C need=3D<optimized out>=2C p= =3D<optimized out>) at beam/erl_gc.c:876=0A= #3 erts_garbage_collect (p=3D0x7faeb8fc5488=2C need=3D<optimized out>=2C o= bjv=3D<optimized out>=2C nobj=3D<optimized out>) at beam/erl_gc.c:450=0A= #4 0x00000000004e1374 in process_main () at beam/beam_emu.c:1858=0A= #5 0x000000000048071d in sched_thread_func (vesdp=3D<optimized out>) at be= am/erl_process.c:7719=0A= #6 0x0000000000549f89 in thr_wrapper (vtwd=3D<optimized out>) at pthread/e= thread.c:106=0A= #7 0x00007faeba32ba30 in ?? ()=0A= #8 0x0000000000000000 in ?? () From: [email protected] To: [email protected] Date: Sun=2C 29 May 2016 21:32:58 -0400 CC: [email protected]=3B [email protected] Subject: Re: [erlang-questions] [erlang-bugs] beam core file R17 =0A= =0A= =0A= Thanks=2C Fortunately our latest release has moved all the NIF logic to a separate CN= ODE based process. I am leaning towards the NIF as the cause since the VM i= tself has proven to be very stable. Date: Sun=2C 29 May 2016 21:23:15 +0200 Subject: Re: [erlang-bugs] beam core file R17 From: [email protected] To: [email protected] CC: [email protected] sweep_off_heap is run when collecting refc binaries (among other things)=2C= so if you by mistake have decremented the reference count of a binary in a= nif too much=2C this error will happen when the GC is inspecting the binar= y. I'd recommend looking for something like that in any nifs you have. On Sun=2C May 29=2C 2016 at 6:04 PM=2C Matthew Evans <mattevans123@hotmail.= com> wrote: Thanks=2C this software does use nifs =0A= =0A= Sent from my iPhone =0A= =0A= > On May 29=2C 2016=2C at 11:56 AM=2C Mikael Pettersson <[email protected]= om> wrote: =0A= > =0A= > Matthew Evans writes: =0A= >> Hi=2C =0A= >> This core was found on a live system (R17): =0A= >> =0A= >> =0A= >> =0A= >> =0A= >> =0A= >> =0A= >> =0A= >> =0A= >> 10:16:38:# erl =0A= >> Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:8:8] [async-threads:10] = [hipe] [kernel-poll:false] =0A= >> =0A= >> =0A= >> Eshell V6.2 (abort with ^G) =0A= >> 1> =0A= >> =0A= >> ........ =0A= >> =0A= >> =0A= >> =0A= >> =0A= >> =0A= >> =0A= >> =0A= >> =0A= >> May 28 22:18:56 [info ] plexxi kernel: [1235119.885465] beam.smp[2267]= general protection ip:4b698a sp:7faeb6a7d650 error:0 in beam.smp[400000+1a= c000] =0A= >> warning: Can't read pathname for load map: Input/output error.[Thread de= bugging using libthread_db enabled]Using host libthread_db library "/lib/li= bthread_db.so.1".Core was generated by `/usr/lib/erlang/erts-6.2/bin/beam.s= mp -K true -A 24 -P 350000 -- -root /usr/lib'.Program terminated with signa= l 11=2C Segmentation fault.#0 0x00000000004b698a in sweep_off_heap ()(gdb)= bt#0 0x00000000004b698a in sweep_off_heap ()#1 0x00000000004b77d1 in do_= minor ()#2 0x00000000004b8479 in erts_garbage_collect ()#3 0x00000000004e= 1374 in process_main ()#4 0x000000000048071d in sched_thread_func ()#5 0x= 0000000000549f89 in thr_wrapper ()#6 0x00007faeba32ba30 in start_thread ()= from /lib/libpthread.so.0#7 0x00007faeb9e8a53d in clone () from /lib/libc= .so.6 =0A= > =0A= > It looks like your beam.smp binary lacks debugging information=2C so we o= nly know the general =0A= > area where it crashed (sweep_off_heap() as called from do_minor()). Cras= hes here would usually =0A= > be due to memory corruption=2C which could be caused by: =0A= > - a bug in the VM =0A= > - a bug in a NIF =0A= > - a bug in HiPE =0A= > - a bug in the C compiler used to compile the VM (I've seen that happen a= t least 3 times) =0A= > - a HW error (though you'd then also find e.g. machine check events logge= d) =0A= > =0A= > If you want to debug this=2C you should first ensure that your beam.smp g= ets built and installed =0A= > with full debugging information (just attach gdb=2C bt=2C and list to ver= ify). You should also try =0A= > without NIFs or native code=2C if those are used and you can configure th= em not to be. =0A= _______________________________________________ =0A= erlang-bugs mailing list =0A= [email protected] =0A= http://erlang.org/mailman/listinfo/erlang-bugs =0A= =0A= _______________________________________________=0A= erlang-questions mailing list=0A= [email protected]=0A= http://erlang.org/mailman/listinfo/erlang-questions = --_5681fa72-082e-43cf-ad28-2ce04ebb0766_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <style><!-- .hmmessage P { margin:0px=3B padding:0px } body.hmmessage { font-size: 12pt=3B font-family:Calibri } --></style></head> <body class=3D'hmmessage'><div dir=3D'ltr'>For what it's worth here's a gdb= with symbols:<div><br></div><div>=0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= <p class=3D"p1"><span class=3D"s1">(gdb) bt</span></p>=0A= <p class=3D"p1"><span class=3D"s1">#0 =3B sweep_off_heap (p=3D0x7faeb8f= c5488=2C fullsweep=3D-1295963664) at beam/erl_gc.c:2353</span></p>=0A= <p class=3D"p1"><span class=3D"s1">#1 =3B 0x00000000004b77d1 in do_mino= r (p=3D0x7faeb8fc5488=2C new_sz=3D<=3Boptimized out>=3B=2C objv=3D<= =3Boptimized out>=3B=2C nobj=3D<=3Boptimized out>=3B) at beam/erl_gc.= c:1166</span></p>=0A= <p class=3D"p1"><span class=3D"s1">#2 =3B 0x00000000004b8479 in minor_c= ollection (recl=3D<=3Boptimized out>=3B=2C nobj=3D<=3Boptimized out&g= t=3B=2C objv=3D<=3Boptimized out>=3B=2C need=3D<=3Boptimized out>= =3B=2C p=3D<=3Boptimized out>=3B) at beam/erl_gc.c:876</span></p>=0A= <p class=3D"p1"><span class=3D"s1">#3 =3B erts_garbage_collect (p=3D0x7= faeb8fc5488=2C need=3D<=3Boptimized out>=3B=2C objv=3D<=3Boptimized o= ut>=3B=2C nobj=3D<=3Boptimized out>=3B) at beam/erl_gc.c:450</span></= p>=0A= <p class=3D"p1"><span class=3D"s1">#4 =3B 0x00000000004e1374 in process= _main () at beam/beam_emu.c:1858</span></p>=0A= <p class=3D"p1"><span class=3D"s1">#5 =3B 0x000000000048071d in sched_t= hread_func (vesdp=3D<=3Boptimized out>=3B) at beam/erl_process.c:7719</= span></p>=0A= <p class=3D"p1"><span class=3D"s1">#6 =3B 0x0000000000549f89 in thr_wra= pper (vtwd=3D<=3Boptimized out>=3B) at pthread/ethread.c:106</span></p>= =0A= <p class=3D"p1"><span class=3D"s1">#7 =3B 0x00007faeba32ba30 in ?? ()</= span></p>=0A= <p class=3D"p1"><span class=3D"s1">#8 =3B 0x0000000000000000 in ?? ()</= span></p><p class=3D"p1"><span class=3D"s1"><br></span></p><br><div><hr id= =3D"stopSpelling">From: [email protected]<br>To: [email protected]= <br>Date: Sun=2C 29 May 2016 21:32:58 -0400<br>CC: erlang-questions@erlang.= org=3B [email protected]<br>Subject: Re: [erlang-questions] [erlang-bu= gs] beam core file R17<br><br>=0A= =0A= <style><!--=0A= .ExternalClass .ecxhmmessage P {=0A= padding:0px=3B=0A= }=0A= =0A= .ExternalClass body.ecxhmmessage {=0A= font-size:12pt=3B=0A= font-family:Calibri=3B=0A= }=0A= =0A= --></style>=0A= <div dir=3D"ltr"><br>Thanks=2C<div><br></div><div>Fortunately our latest re= lease has moved all the NIF logic to a separate CNODE based process. I am l= eaning towards the NIF as the cause since the VM itself has proven to be ve= ry stable.</div><div><br></div><div><br><div><hr id=3D"ecxstopSpelling">Dat= e: Sun=2C 29 May 2016 21:23:15 +0200<br>Subject: Re: [erlang-bugs] beam cor= e file R17<br>From: [email protected]<br>To: [email protected]<br>= CC: [email protected]<br><br><div dir=3D"ltr">sweep_off_heap is r= un when collecting refc binaries (among other things)=2C so if you by mista= ke have decremented the reference count of a binary in a nif too much=2C th= is error will happen when the GC is inspecting the binary. I'd recommend lo= oking for something like that in any nifs you have.<br><div class=3D"ecxgma= il_extra"><br><div class=3D"ecxgmail_quote">On Sun=2C May 29=2C 2016 at 6:0= 4 PM=2C Matthew Evans <span dir=3D"ltr"><=3B<a href=3D"mailto:mattevans12= [email protected]" target=3D"_blank">[email protected]</a>>=3B</span> = wrote:<br><blockquote class=3D"ecxgmail_quote" style=3D"border-left:1px #cc= c solid=3Bpadding-left:1ex=3B">Thanks=2C this software does use nifs<br>=0A= <br>=0A= Sent from my iPhone<br>=0A= <div class=3D"ecxHOEnZb"><div class=3D"h5"><br>=0A= >=3B On May 29=2C 2016=2C at 11:56 AM=2C Mikael Pettersson <=3B<a href= =3D"mailto:[email protected]">[email protected]</a>>=3B wrote:<br>= =0A= >=3B<br>=0A= >=3B Matthew Evans writes:<br>=0A= >=3B>=3B Hi=2C<br>=0A= >=3B>=3B This core was found on a live system (R17):<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B 10:16:38:# erl<br>=0A= >=3B>=3B Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:8:8] [async-th= reads:10] [hipe] [kernel-poll:false]<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B Eshell V6.2 =3B (abort with ^G)<br>=0A= >=3B>=3B 1>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B ........<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B<br>=0A= >=3B>=3B May 28 22:18:56 [info =3B  =3B] plexxi kernel: [123511= 9.885465] beam.smp[2267] general protection ip:4b698a sp:7faeb6a7d650 error= :0 in beam.smp[400000+1ac000]<br>=0A= >=3B>=3B warning: Can't read pathname for load map: Input/output error.= [Thread debugging using libthread_db enabled]Using host libthread_db librar= y "/lib/libthread_db.so.1".Core was generated by `/usr/lib/erlang/erts-6.2/= bin/beam.smp -K true -A 24 -P 350000 -- -root /usr/lib'.Program terminated = with signal 11=2C Segmentation fault.#0 =3B 0x00000000004b698a in sweep= _off_heap ()(gdb) bt#0 =3B 0x00000000004b698a in sweep_off_heap ()#1&nb= sp=3B 0x00000000004b77d1 in do_minor ()#2 =3B 0x00000000004b8479 in ert= s_garbage_collect ()#3 =3B 0x00000000004e1374 in process_main ()#4 = =3B 0x000000000048071d in sched_thread_func ()#5 =3B 0x0000000000549f89= in thr_wrapper ()#6 =3B 0x00007faeba32ba30 in start_thread () from /li= b/libpthread.so.0#7 =3B 0x00007faeb9e8a53d in clone () from /lib/libc.s= o.6<br>=0A= >=3B<br>=0A= >=3B It looks like your beam.smp binary lacks debugging information=2C so= we only know the general<br>=0A= >=3B area where it crashed (sweep_off_heap() as called from do_minor()).&= nbsp=3B Crashes here would usually<br>=0A= >=3B be due to memory corruption=2C which could be caused by:<br>=0A= >=3B - a bug in the VM<br>=0A= >=3B - a bug in a NIF<br>=0A= >=3B - a bug in HiPE<br>=0A= >=3B - a bug in the C compiler used to compile the VM (I've seen that hap= pen at least 3 times)<br>=0A= >=3B - a HW error (though you'd then also find e.g. machine check events = logged)<br>=0A= >=3B<br>=0A= >=3B If you want to debug this=2C you should first ensure that your beam.= smp gets built and installed<br>=0A= >=3B with full debugging information (just attach gdb=2C bt=2C and list t= o verify). =3B You should also try<br>=0A= >=3B without NIFs or native code=2C if those are used and you can configu= re them not to be.<br>=0A= _______________________________________________<br>=0A= erlang-bugs mailing list<br>=0A= <a href=3D"mailto:[email protected]">[email protected]</a><br>=0A= <a href=3D"http://erlang.org/mailman/listinfo/erlang-bugs" rel=3D"noreferre= r" target=3D"_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a><br>= =0A= </div></div></blockquote></div><br></div></div></div></div> </di= v>=0A= <br>_______________________________________________=0A= erlang-questions mailing list=0A= [email protected]=0A= http://erlang.org/mailman/listinfo/erlang-questions</div></div> = </div></body> </html>= --_5681fa72-082e-43cf-ad28-2ce04ebb0766_-- --===============9043578613314440583== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ erlang-bugs mailing list [email protected] http://erlang.org/mailman/listinfo/erlang-bugs --===============9043578613314440583==--