Re: When is it safe to call PyStackless_Schedule
Robert Babiak <[email protected]> Wed, 10 Sep 2014 07:17:13 -0600
| Newsgroups | gmane.comp.python.stackless |
|---|---|
| Message-ID | <CAEmPv6H4AwCtOCRSpLD1e0Fos19CaE=YAtvUBFaxXt_LxWsA_A@mail.gmail.com> |
--===============5805335975723935215== Content-Type: multipart/alternative; boundary=001a1134eeb8c1a51c0502b5de2b --001a1134eeb8c1a51c0502b5de2b Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks Kristj=C3=A1n you nailed it. Adding the GIL around it solved all the crashes I had gotten with both calling schedule or my BeNice. It is working slick now. Is there a good resource for reading about when the GIL is held and when it is not? On Wed, Sep 10, 2014 at 3:03 AM, Kristj=C3=A1n Valur J=C3=B3nsson <sweskman= @gmail.com> wrote: > PyGILState_STATE PyGILState_Ensure() / PyGILState_Release() > K > > On 10 September 2014 09:02, Kristj=C3=A1n Valur J=C3=B3nsson <sweskman@gm= ail.com> > wrote: > >> I suspect that the raw input hook is called without the GIL held. >> You need to add a call to >> PyGIL_Ensure / PyGIL_Release >> >> On 10 September 2014 05:02, Richard Tew <[email protected]> wrote: >> >>> Hard switching immediately switches by changing the stack contents. >>> And soft switching has the function you call return an unwind token >>> which should be propagated back through the call chain to the >>> scheduler. If you ignore this, and throw it away, it has caused crash >>> problems in the past. Of course, with the approach you've taken, >>> there's no way it can be returned out of that event and back through >>> the call chain. You can try and call the versions of the api which do >>> not soft switch, these might be documented in the documentation. >>> >>> Cheers, >>> Richard. >>> >>> On 9/10/14, Robert Babiak <[email protected]> wrote: >>> > oh the actual break message >>> > >>> > Process 53817 stopped >>> > >>> > * thread #5: tid =3D 0x1c027f, 0x0000000100ea30a3 >>> Python`PyStackless_Schedule >>> > + 51, stop reason =3D EXC_BAD_ACCESS (code=3D1, address=3D0xd8) >>> > >>> > frame #0: 0x0000000100ea30a3 Python`PyStackless_Schedule + 51 >>> > >>> > Python`PyStackless_Schedule + 51: >>> > >>> > -> 0x100ea30a3: movq 0xd8(%r13), %rbx >>> > >>> > 0x100ea30aa: movq 0x10(%rbx), %r12 >>> > >>> > 0x100ea30ae: movq $0x0, -0x30(%rbp) >>> > >>> > 0x100ea30b6: cmpq $0x0, 0xd0(%r13) >>> > >>> > On Tue, Sep 9, 2014 at 9:57 PM, Robert Babiak <[email protected]> >>> wrote: >>> > >>> >> I took the Receive out and replaced it with a simple call to >>> >> PyStackless_Schedule. >>> >> >>> >> static int EventHook(void) >>> >> { >>> >> PyStackless_Schedule(Py_None, 0); >>> >> return 0; >>> >> } >>> >> ... >>> >> Py_SetProgramName(argv[0]); >>> >> Py_InitializeEx(0); >>> >> if (!Py_IsInitialized()) >>> >> { >>> >> std::cerr << "Python initialization failed" << std::endl= ; >>> >> return; >>> >> } >>> >> PySys_SetArgv(argc, argv); >>> >> PyOS_InputHook =3D EventHook; >>> >> ... >>> >> // stacklessMain returns when program is shutdown. >>> >> PyStackless_CallMethod_Main(oMainModule.ptr(), >>> "stacklessMain", >>> >> 0); >>> >> ... >>> >> >>> >> This is the lldb stacktrace >>> >> >>> >> frame #0: 0x0000000100ea30a3 Python`PyStackless_Schedule + 51 >>> >> >>> >> frame #1: 0x0000000100155fd9 main`EventHook() + 25 at >>> >> pythonInterperter.cpp:472 >>> >> >>> >> frame #2: 0x0000000103dd5e25 readline.so`call_readline + 293 >>> >> >>> >> frame #3: 0x0000000100dfa3c3 Python`PyOS_Readline + 179 >>> >> >>> >> frame #4: 0x0000000100e92882 Python`builtin_raw_input + 434 >>> >> >>> >> frame #5: 0x0000000100e95d88 Python`PyEval_EvalFrame_value + 383= 2 >>> >> >>> >> frame #6: 0x0000000100e94d38 Python`PyEval_EvalFrameEx_slp + 376 >>> >> >>> >> frame #7: 0x0000000100e9744c Python`PyEval_EvalFrame_value + 966= 0 >>> >> >>> >> frame #8: 0x0000000100e94d38 Python`PyEval_EvalFrameEx_slp + 376 >>> >> >>> >> frame #9: 0x0000000100e9744c Python`PyEval_EvalFrame_value + 966= 0 >>> >> >>> >> frame #10: 0x0000000100e94d38 Python`PyEval_EvalFrameEx_slp + 37= 6 >>> >> >>> >> frame #11: 0x0000000100e9e9e1 Python`slp_frame_dispatch_top + 49 >>> >> >>> >> frame #12: 0x0000000100ea2772 Python`slp_run_tasklet + 226 >>> >> >>> >> frame #13: 0x0000000100e9e0fc Python`slp_eval_frame + 348 >>> >> >>> >> frame #14: 0x0000000100e9e083 Python`slp_eval_frame + 227 >>> >> >>> >> frame #15: 0x0000000100e9e083 Python`slp_eval_frame + 227 >>> >> >>> >> ..... >>> >> >>> >> frame #719: 0x0000000100e9e083 Python`slp_eval_frame + 227 >>> >> >>> >> frame #720: 0x0000000100e9e083 Python`slp_eval_frame + 227 >>> >> >>> >> frame #721: 0x0000000100ea3801 Python`PyStackless_CallMethod_Mai= n >>> + >>> >> 433 >>> >> >>> >> frame #722: 0x00000001001588e3 >>> >> main`PythonThread::operator(this=3D0x0000000101506940)() + 1187 at >>> >> pythonInterperter.cpp:555 >>> >> >>> >> frame #723: 0x000000010015761c >>> >> >>> main`boost::detail::thread_data<PythonThread>::run(this=3D0x00000001015= 06740) >>> >> + 28 at thread.hpp:117 >>> >> >>> >> frame #724: 0x00000001010ff24a >>> libboost_thread-mt.dylib`thread_proxy >>> >> + >>> >> 186 >>> >> >>> >> frame #725: 0x00007fff863fb899 >>> libsystem_pthread.dylib`_pthread_body >>> >> + >>> >> 138 >>> >> >>> >> frame #726: 0x00007fff863fb72a >>> libsystem_pthread.dylib`_pthread_start >>> >> + 137 >>> >> >>> >> On Tue, Sep 9, 2014 at 9:17 PM, Richard Tew <[email protected]= > >>> >> wrote: >>> >> >>> >>> If you have a crash when you call receive, then you likely have >>> larger >>> >>> problems than when the right time to call schedule is. >>> >>> >>> >>> I don't think it's possible to divine what is going wrong, and expe= ct >>> >>> you need to provide a reproduction case. >>> >>> >>> >>> Cheers >>> >>> >>> >>> On 9/10/14, Robert Babiak <[email protected]> wrote: >>> >>> > Dear wise stackless list, I need some of your wisdom. >>> >>> > >>> >>> > I am trying to make raw_input play nice with the stackless >>> scheduler >>> >>> > and >>> >>> > not block with my embedded stackless. >>> >>> > >>> >>> > I set the PyOS_InputHook to get what amounts to an idle event but >>> when >>> >>> > I >>> >>> > call PyStackless_Schedule(PY_None, 0) it crashes (no stackless >>> >>> symbols...) >>> >>> > >>> >>> > Is there something I need to check to know if it is safe to call >>> the >>> >>> > scheduler from C? >>> >>> > >>> >>> > In general when is it safe to call PyStackless_Schedule? >>> >>> > >>> >>> > I tried to call my BeNice to block and get woken up later, but th= is >>> >>> > also >>> >>> > crashes when trying to receive on my BeNice channel. >>> >>> > >>> >>> > - Thanks Rob. >>> >>> > -- >>> >>> > Life: Bah, I will worry about it when it is over. >>> >>> > >>> >>> >>> >>> _______________________________________________ >>> >>> Stackless mailing list >>> >>> [email protected] >>> >>> http://www.stackless.com/mailman/listinfo/stackless >>> >>> >>> >> >>> >> >>> >> >>> >> -- >>> >> Life: Bah, I will worry about it when it is over. >>> >> >>> > >>> > >>> > >>> > -- >>> > Life: Bah, I will worry about it when it is over. >>> > >>> >>> _______________________________________________ >>> Stackless mailing list >>> [email protected] >>> http://www.stackless.com/mailman/listinfo/stackless >>> >> >> > > _______________________________________________ > Stackless mailing list > [email protected] > http://www.stackless.com/mailman/listinfo/stackless > --=20 Life: Bah, I will worry about it when it is over. --001a1134eeb8c1a51c0502b5de2b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Thanks Kristj=C3=A1n you nailed it. Adding the GIL around = it solved all the crashes I had gotten with both calling schedule or my BeN= ice. It is working slick now.=C2=A0<div><br></div><div>Is there a good reso= urce for reading about when the GIL is held and when it is not?</div><div><= br></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On= Wed, Sep 10, 2014 at 3:03 AM, Kristj=C3=A1n Valur J=C3=B3nsson <span dir= =3D"ltr"><<a href=3D"mailto:[email protected]" target=3D"_blank">sweskm= [email protected]</a>></span> wrote:<br><blockquote class=3D"gmail_quote" sty= le=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div d= ir=3D"ltr"><dt>PyGILState_STATE <tt>PyGILState_Ensure</tt><big>(</big><big>= ) / PyGILState_Release()</big></dt><dt><big><br></big></dt><dt><big>K</big>= </dt></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On 10 = September 2014 09:02, Kristj=C3=A1n Valur J=C3=B3nsson <span dir=3D"ltr">&l= t;<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]= m</a>></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margi= n:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">= I suspect that the raw input hook is called without the GIL held.<div>You n= eed to add a call to=C2=A0</div><div>PyGIL_Ensure / PyGIL_Release</div></di= v><div><div class=3D"h5"><div><div><div class=3D"gmail_extra"><br><div clas= s=3D"gmail_quote">On 10 September 2014 05:02, Richard Tew <span dir=3D"ltr"= ><<a href=3D"mailto:[email protected]" target=3D"_blank">richard.m= [email protected]</a>></span> wrote:<br><blockquote class=3D"gmail_quote" s= tyle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hard= switching immediately switches by changing the stack contents.<br> And soft switching has the function you call return an unwind token<br> which should be propagated back through the call chain to the<br> scheduler.=C2=A0 If you ignore this, and throw it away, it has caused crash= <br> problems in the past.=C2=A0 Of course, with the approach you've taken,<= br> there's no way it can be returned out of that event and back through<br= > the call chain.=C2=A0 You can try and call the versions of the api which do= <br> not soft switch, these might be documented in the documentation.<br> <br> Cheers,<br> Richard.<br> <div><div><br> On 9/10/14, Robert Babiak <<a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a>> wrote:<br> > oh the actual break message<br> ><br> > Process 53817 stopped<br> ><br> > * thread #5: tid =3D 0x1c027f, 0x0000000100ea30a3 Python`PyStackless_S= chedule<br> > + 51, stop reason =3D EXC_BAD_ACCESS (code=3D1, address=3D0xd8)<br> ><br> >=C2=A0 =C2=A0 =C2=A0frame #0: 0x0000000100ea30a3 Python`PyStackless_Sch= edule + 51<br> ><br> > Python`PyStackless_Schedule + 51:<br> ><br> > -> 0x100ea30a3:=C2=A0 movq=C2=A0 =C2=A00xd8(%r13), %rbx<br> ><br> >=C2=A0 =C2=A0 0x100ea30aa:=C2=A0 movq=C2=A0 =C2=A00x10(%rbx), %r12<br> ><br> >=C2=A0 =C2=A0 0x100ea30ae:=C2=A0 movq=C2=A0 =C2=A0$0x0, -0x30(%rbp)<br> ><br> >=C2=A0 =C2=A0 0x100ea30b6:=C2=A0 cmpq=C2=A0 =C2=A0$0x0, 0xd0(%r13)<br> ><br> > On Tue, Sep 9, 2014 at 9:57 PM, Robert Babiak <<a href=3D"mailto:rb= [email protected]" target=3D"_blank">[email protected]</a>> wrote:<br> ><br> >> I took the Receive out and replaced it with a simple call to<br> >> PyStackless_Schedule.<br> >><br> >> static int EventHook(void)<br> >> {<br> >>=C2=A0 =C2=A0 =C2=A0PyStackless_Schedule(Py_None, 0);<br> >>=C2=A0 =C2=A0 =C2=A0return 0;<br> >> }<br> >> ...<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Py_SetProgramName(argv[0]);<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Py_InitializeEx(0);<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (!Py_IsInitialized())<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0std::cerr << = "Python initialization failed" << std::endl;<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return;<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0}<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PySys_SetArgv(argc, argv);<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PyOS_InputHook =3D EventHook;<br> >> ...<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0// stacklessMain returns when pro= gram is shutdown.<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PyStackless_CallMethod_Main(oMain= Module.ptr(), "stacklessMain",<br> >> 0);<br> >> ...<br> >><br> >> This is the lldb stacktrace<br> >><br> >> frame #0: 0x0000000100ea30a3 Python`PyStackless_Schedule + 51<br> >><br> >>=C2=A0 frame #1: 0x0000000100155fd9 main`EventHook() + 25 at<br> >> pythonInterperter.cpp:472<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #2: 0x0000000103dd5e25 readline.so`call_r= eadline + 293<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #3: 0x0000000100dfa3c3 Python`PyOS_Readli= ne + 179<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #4: 0x0000000100e92882 Python`builtin_raw= _input + 434<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #5: 0x0000000100e95d88 Python`PyEval_Eval= Frame_value + 3832<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #6: 0x0000000100e94d38 Python`PyEval_Eval= FrameEx_slp + 376<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #7: 0x0000000100e9744c Python`PyEval_Eval= Frame_value + 9660<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #8: 0x0000000100e94d38 Python`PyEval_Eval= FrameEx_slp + 376<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #9: 0x0000000100e9744c Python`PyEval_Eval= Frame_value + 9660<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #10: 0x0000000100e94d38 Python`PyEval_Eva= lFrameEx_slp + 376<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #11: 0x0000000100e9e9e1 Python`slp_frame_= dispatch_top + 49<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #12: 0x0000000100ea2772 Python`slp_run_ta= sklet + 226<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #13: 0x0000000100e9e0fc Python`slp_eval_f= rame + 348<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #14: 0x0000000100e9e083 Python`slp_eval_f= rame + 227<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #15: 0x0000000100e9e083 Python`slp_eval_f= rame + 227<br> >><br> >> .....<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #719: 0x0000000100e9e083 Python`slp_eval_= frame + 227<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #720: 0x0000000100e9e083 Python`slp_eval_= frame + 227<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #721: 0x0000000100ea3801 Python`PyStackle= ss_CallMethod_Main +<br> >> 433<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #722: 0x00000001001588e3<br> >> main`PythonThread::operator(this=3D0x0000000101506940)() + 1187 at= <br> >> pythonInterperter.cpp:555<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #723: 0x000000010015761c<br> >> main`boost::detail::thread_data<PythonThread>::run(this=3D0x= 0000000101506740)<br> >> + 28 at thread.hpp:117<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #724: 0x00000001010ff24a libboost_thread-= mt.dylib`thread_proxy<br> >> +<br> >> 186<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #725: 0x00007fff863fb899 libsystem_pthrea= d.dylib`_pthread_body<br> >> +<br> >> 138<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #726: 0x00007fff863fb72a libsystem_pthrea= d.dylib`_pthread_start<br> >> + 137<br> >><br> >> On Tue, Sep 9, 2014 at 9:17 PM, Richard Tew <<a href=3D"mailto:= [email protected]" target=3D"_blank">[email protected]</a>><= br> >> wrote:<br> >><br> >>> If you have a crash when you call receive, then you likely hav= e larger<br> >>> problems than when the right time to call schedule is.<br> >>><br> >>> I don't think it's possible to divine what is going wr= ong, and expect<br> >>> you need to provide a reproduction case.<br> >>><br> >>> Cheers<br> >>><br> >>> On 9/10/14, Robert Babiak <<a href=3D"mailto:rbabiak@gmail.= com" target=3D"_blank">[email protected]</a>> wrote:<br> >>> > Dear wise stackless list, I need some of your wisdom.<br> >>> ><br> >>> > I am trying to make raw_input play nice with the stackles= s scheduler<br> >>> > and<br> >>> > not block with my embedded stackless.<br> >>> ><br> >>> > I set the PyOS_InputHook to get what amounts to an idle e= vent but when<br> >>> > I<br> >>> > call PyStackless_Schedule(PY_None, 0) it crashes (no stac= kless<br> >>> symbols...)<br> >>> ><br> >>> > Is there something I need to check to know if it is safe = to call the<br> >>> > scheduler from C?<br> >>> ><br> >>> > In general when is it safe to call=C2=A0 PyStackless_Sche= dule?<br> >>> ><br> >>> > I tried to call my BeNice to block and get woken up later= , but this<br> >>> > also<br> >>> > crashes when trying to receive on my BeNice channel.<br> >>> ><br> >>> > - Thanks Rob.<br> >>> > --<br> >>> > Life: Bah, I will worry about it when it is over.<br> >>> ><br> >>><br> >>> _______________________________________________<br> >>> Stackless mailing list<br> >>> <a href=3D"mailto:[email protected]" target=3D"_blank">S= [email protected]</a><br> >>> <a href=3D"http://www.stackless.com/mailman/listinfo/stackless= " target=3D"_blank">http://www.stackless.com/mailman/listinfo/stackless</a>= <br> >>><br> >><br> >><br> >><br> >> --<br> >> Life: Bah, I will worry about it when it is over.<br> >><br> ><br> ><br> ><br> > --<br> > Life: Bah, I will worry about it when it is over.<br> ><br> <br> _______________________________________________<br> Stackless mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">Stackless@stac= kless.com</a><br> <a href=3D"http://www.stackless.com/mailman/listinfo/stackless" target=3D"_= blank">http://www.stackless.com/mailman/listinfo/stackless</a><br> </div></div></blockquote></div><br></div> </div></div></div></div></blockquote></div><br></div> <br>_______________________________________________<br> Stackless mailing list<br> <a href=3D"mailto:[email protected]">[email protected]</a><br> <a href=3D"http://www.stackless.com/mailman/listinfo/stackless" target=3D"_= blank">http://www.stackless.com/mailman/listinfo/stackless</a><br></blockqu= ote></div><br><br clear=3D"all"><div><br></div>-- <br>Life: Bah, I will wor= ry about it when it is over. </div> --001a1134eeb8c1a51c0502b5de2b-- --===============5805335975723935215== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Stackless mailing list [email protected] http://www.stackless.com/mailman/listinfo/stackless --===============5805335975723935215==--