Re: When is it safe to call PyStackless_Schedule
Robert Babiak <[email protected]> Wed, 10 Sep 2014 07:20:36 -0600
| Newsgroups | gmane.comp.python.stackless |
|---|---|
| Message-ID | <CAEmPv6HWP2+H4OnXSyQ_fT0dx+NZn-3eHJ5eWiOPuoJei18aDQ@mail.gmail.com> |
--===============4366623348249355656== Content-Type: multipart/alternative; boundary=001a11c29f3ee5a37e0502b5eae8 --001a11c29f3ee5a37e0502b5eae8 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Would It be possible to add the raw_input to the stackless lib so it will not block waiting on input or is that not possible since it is a C module built into python/stackless? On Wed, Sep 10, 2014 at 7:17 AM, Robert Babiak <[email protected]> wrote: > Thanks Kristj=C3=A1n you nailed it. Adding the GIL around it solved all t= he > crashes I had gotten with both calling schedule or my BeNice. It is worki= ng > slick now. > > Is there a good resource for reading about when the GIL is held and when > it is not? > > > On Wed, Sep 10, 2014 at 3:03 AM, Kristj=C3=A1n Valur J=C3=B3nsson < > [email protected]> wrote: > >> PyGILState_STATE PyGILState_Ensure() / PyGILState_Release() >> K >> >> On 10 September 2014 09:02, Kristj=C3=A1n Valur J=C3=B3nsson <sweskman@g= mail.com> >> wrote: >> >>> I suspect that the raw input hook is called without the GIL held. >>> You need to add a call to >>> PyGIL_Ensure / PyGIL_Release >>> >>> On 10 September 2014 05:02, Richard Tew <[email protected]> wrote= : >>> >>>> Hard switching immediately switches by changing the stack contents. >>>> And soft switching has the function you call return an unwind token >>>> which should be propagated back through the call chain to the >>>> scheduler. If you ignore this, and throw it away, it has caused crash >>>> problems in the past. Of course, with the approach you've taken, >>>> there's no way it can be returned out of that event and back through >>>> the call chain. You can try and call the versions of the api which do >>>> not soft switch, these might be documented in the documentation. >>>> >>>> Cheers, >>>> Richard. >>>> >>>> On 9/10/14, Robert Babiak <[email protected]> wrote: >>>> > oh the actual break message >>>> > >>>> > Process 53817 stopped >>>> > >>>> > * thread #5: tid =3D 0x1c027f, 0x0000000100ea30a3 >>>> Python`PyStackless_Schedule >>>> > + 51, stop reason =3D EXC_BAD_ACCESS (code=3D1, address=3D0xd8) >>>> > >>>> > frame #0: 0x0000000100ea30a3 Python`PyStackless_Schedule + 51 >>>> > >>>> > Python`PyStackless_Schedule + 51: >>>> > >>>> > -> 0x100ea30a3: movq 0xd8(%r13), %rbx >>>> > >>>> > 0x100ea30aa: movq 0x10(%rbx), %r12 >>>> > >>>> > 0x100ea30ae: movq $0x0, -0x30(%rbp) >>>> > >>>> > 0x100ea30b6: cmpq $0x0, 0xd0(%r13) >>>> > >>>> > On Tue, Sep 9, 2014 at 9:57 PM, Robert Babiak <[email protected]> >>>> wrote: >>>> > >>>> >> I took the Receive out and replaced it with a simple call to >>>> >> PyStackless_Schedule. >>>> >> >>>> >> static int EventHook(void) >>>> >> { >>>> >> PyStackless_Schedule(Py_None, 0); >>>> >> return 0; >>>> >> } >>>> >> ... >>>> >> Py_SetProgramName(argv[0]); >>>> >> Py_InitializeEx(0); >>>> >> if (!Py_IsInitialized()) >>>> >> { >>>> >> std::cerr << "Python initialization failed" << std::end= l; >>>> >> return; >>>> >> } >>>> >> PySys_SetArgv(argc, argv); >>>> >> PyOS_InputHook =3D EventHook; >>>> >> ... >>>> >> // stacklessMain returns when program is shutdown. >>>> >> PyStackless_CallMethod_Main(oMainModule.ptr(), >>>> "stacklessMain", >>>> >> 0); >>>> >> ... >>>> >> >>>> >> This is the lldb stacktrace >>>> >> >>>> >> frame #0: 0x0000000100ea30a3 Python`PyStackless_Schedule + 51 >>>> >> >>>> >> frame #1: 0x0000000100155fd9 main`EventHook() + 25 at >>>> >> pythonInterperter.cpp:472 >>>> >> >>>> >> frame #2: 0x0000000103dd5e25 readline.so`call_readline + 293 >>>> >> >>>> >> frame #3: 0x0000000100dfa3c3 Python`PyOS_Readline + 179 >>>> >> >>>> >> frame #4: 0x0000000100e92882 Python`builtin_raw_input + 434 >>>> >> >>>> >> frame #5: 0x0000000100e95d88 Python`PyEval_EvalFrame_value + 38= 32 >>>> >> >>>> >> frame #6: 0x0000000100e94d38 Python`PyEval_EvalFrameEx_slp + 37= 6 >>>> >> >>>> >> frame #7: 0x0000000100e9744c Python`PyEval_EvalFrame_value + 96= 60 >>>> >> >>>> >> frame #8: 0x0000000100e94d38 Python`PyEval_EvalFrameEx_slp + 37= 6 >>>> >> >>>> >> frame #9: 0x0000000100e9744c Python`PyEval_EvalFrame_value + 96= 60 >>>> >> >>>> >> frame #10: 0x0000000100e94d38 Python`PyEval_EvalFrameEx_slp + 3= 76 >>>> >> >>>> >> frame #11: 0x0000000100e9e9e1 Python`slp_frame_dispatch_top + 4= 9 >>>> >> >>>> >> frame #12: 0x0000000100ea2772 Python`slp_run_tasklet + 226 >>>> >> >>>> >> frame #13: 0x0000000100e9e0fc Python`slp_eval_frame + 348 >>>> >> >>>> >> frame #14: 0x0000000100e9e083 Python`slp_eval_frame + 227 >>>> >> >>>> >> frame #15: 0x0000000100e9e083 Python`slp_eval_frame + 227 >>>> >> >>>> >> ..... >>>> >> >>>> >> frame #719: 0x0000000100e9e083 Python`slp_eval_frame + 227 >>>> >> >>>> >> frame #720: 0x0000000100e9e083 Python`slp_eval_frame + 227 >>>> >> >>>> >> frame #721: 0x0000000100ea3801 >>>> Python`PyStackless_CallMethod_Main + >>>> >> 433 >>>> >> >>>> >> frame #722: 0x00000001001588e3 >>>> >> main`PythonThread::operator(this=3D0x0000000101506940)() + 1187 at >>>> >> pythonInterperter.cpp:555 >>>> >> >>>> >> frame #723: 0x000000010015761c >>>> >> >>>> main`boost::detail::thread_data<PythonThread>::run(this=3D0x0000000101= 506740) >>>> >> + 28 at thread.hpp:117 >>>> >> >>>> >> frame #724: 0x00000001010ff24a >>>> libboost_thread-mt.dylib`thread_proxy >>>> >> + >>>> >> 186 >>>> >> >>>> >> frame #725: 0x00007fff863fb899 >>>> libsystem_pthread.dylib`_pthread_body >>>> >> + >>>> >> 138 >>>> >> >>>> >> frame #726: 0x00007fff863fb72a >>>> libsystem_pthread.dylib`_pthread_start >>>> >> + 137 >>>> >> >>>> >> On Tue, Sep 9, 2014 at 9:17 PM, Richard Tew <[email protected]= m >>>> > >>>> >> wrote: >>>> >> >>>> >>> If you have a crash when you call receive, then you likely have >>>> larger >>>> >>> problems than when the right time to call schedule is. >>>> >>> >>>> >>> I don't think it's possible to divine what is going wrong, and >>>> expect >>>> >>> you need to provide a reproduction case. >>>> >>> >>>> >>> Cheers >>>> >>> >>>> >>> On 9/10/14, Robert Babiak <[email protected]> wrote: >>>> >>> > Dear wise stackless list, I need some of your wisdom. >>>> >>> > >>>> >>> > I am trying to make raw_input play nice with the stackless >>>> scheduler >>>> >>> > and >>>> >>> > not block with my embedded stackless. >>>> >>> > >>>> >>> > I set the PyOS_InputHook to get what amounts to an idle event bu= t >>>> when >>>> >>> > I >>>> >>> > call PyStackless_Schedule(PY_None, 0) it crashes (no stackless >>>> >>> symbols...) >>>> >>> > >>>> >>> > Is there something I need to check to know if it is safe to call >>>> the >>>> >>> > scheduler from C? >>>> >>> > >>>> >>> > In general when is it safe to call PyStackless_Schedule? >>>> >>> > >>>> >>> > I tried to call my BeNice to block and get woken up later, but >>>> this >>>> >>> > also >>>> >>> > crashes when trying to receive on my BeNice channel. >>>> >>> > >>>> >>> > - Thanks Rob. >>>> >>> > -- >>>> >>> > Life: Bah, I will worry about it when it is over. >>>> >>> > >>>> >>> >>>> >>> _______________________________________________ >>>> >>> Stackless mailing list >>>> >>> [email protected] >>>> >>> http://www.stackless.com/mailman/listinfo/stackless >>>> >>> >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> Life: Bah, I will worry about it when it is over. >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > Life: Bah, I will worry about it when it is over. >>>> > >>>> >>>> _______________________________________________ >>>> Stackless mailing list >>>> [email protected] >>>> http://www.stackless.com/mailman/listinfo/stackless >>>> >>> >>> >> >> _______________________________________________ >> Stackless mailing list >> [email protected] >> http://www.stackless.com/mailman/listinfo/stackless >> > > > > -- > Life: Bah, I will worry about it when it is over. > --=20 Life: Bah, I will worry about it when it is over. --001a11c29f3ee5a37e0502b5eae8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Would It be possible to add the raw_input to the stackless= lib so it will not block waiting on input or is that not possible since it= is a C module built into python/stackless?<div><br></div></div><div class= =3D"gmail_extra"><br><div class=3D"gmail_quote">On Wed, Sep 10, 2014 at 7:1= 7 AM, Robert Babiak <span dir=3D"ltr"><<a href=3D"mailto:[email protected]= om" target=3D"_blank">[email protected]</a>></span> wrote:<br><blockquot= e class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc sol= id;padding-left:1ex"><div dir=3D"ltr">Thanks Kristj=C3=A1n you nailed it. A= dding the GIL around it solved all the crashes I had gotten with both calli= ng schedule or my BeNice. It is working slick now.=C2=A0<div><br></div><div= >Is there a good resource for reading about when the GIL is held and when i= t is not?</div><div><br></div></div><div class=3D"HOEnZb"><div class=3D"h5"= ><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Wed, Sep 10, = 2014 at 3:03 AM, Kristj=C3=A1n Valur J=C3=B3nsson <span dir=3D"ltr"><<a = href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>= ></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0= 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><dt>P= yGILState_STATE <tt>PyGILState_Ensure</tt><big>(</big><big>) / PyGILState_R= elease()</big></dt><dt><big><br></big></dt><dt><big>K</big></dt></div><div = class=3D"gmail_extra"><br><div class=3D"gmail_quote">On 10 September 2014 0= 9:02, Kristj=C3=A1n Valur J=C3=B3nsson <span dir=3D"ltr"><<a href=3D"mai= lto:[email protected]" target=3D"_blank">[email protected]</a>></span>= wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bor= der-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">I suspect that t= he raw input hook is called without the GIL held.<div>You need to add a cal= l to=C2=A0</div><div>PyGIL_Ensure / PyGIL_Release</div></div><div><div><div= ><div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On 10 Septe= mber 2014 05:02, Richard Tew <span dir=3D"ltr"><<a href=3D"mailto:richar= [email protected]" target=3D"_blank">[email protected]</a>></span>= wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bor= der-left:1px #ccc solid;padding-left:1ex">Hard switching immediately switch= es by changing the stack contents.<br> And soft switching has the function you call return an unwind token<br> which should be propagated back through the call chain to the<br> scheduler.=C2=A0 If you ignore this, and throw it away, it has caused crash= <br> problems in the past.=C2=A0 Of course, with the approach you've taken,<= br> there's no way it can be returned out of that event and back through<br= > the call chain.=C2=A0 You can try and call the versions of the api which do= <br> not soft switch, these might be documented in the documentation.<br> <br> Cheers,<br> Richard.<br> <div><div><br> On 9/10/14, Robert Babiak <<a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a>> wrote:<br> > oh the actual break message<br> ><br> > Process 53817 stopped<br> ><br> > * thread #5: tid =3D 0x1c027f, 0x0000000100ea30a3 Python`PyStackless_S= chedule<br> > + 51, stop reason =3D EXC_BAD_ACCESS (code=3D1, address=3D0xd8)<br> ><br> >=C2=A0 =C2=A0 =C2=A0frame #0: 0x0000000100ea30a3 Python`PyStackless_Sch= edule + 51<br> ><br> > Python`PyStackless_Schedule + 51:<br> ><br> > -> 0x100ea30a3:=C2=A0 movq=C2=A0 =C2=A00xd8(%r13), %rbx<br> ><br> >=C2=A0 =C2=A0 0x100ea30aa:=C2=A0 movq=C2=A0 =C2=A00x10(%rbx), %r12<br> ><br> >=C2=A0 =C2=A0 0x100ea30ae:=C2=A0 movq=C2=A0 =C2=A0$0x0, -0x30(%rbp)<br> ><br> >=C2=A0 =C2=A0 0x100ea30b6:=C2=A0 cmpq=C2=A0 =C2=A0$0x0, 0xd0(%r13)<br> ><br> > On Tue, Sep 9, 2014 at 9:57 PM, Robert Babiak <<a href=3D"mailto:rb= [email protected]" target=3D"_blank">[email protected]</a>> wrote:<br> ><br> >> I took the Receive out and replaced it with a simple call to<br> >> PyStackless_Schedule.<br> >><br> >> static int EventHook(void)<br> >> {<br> >>=C2=A0 =C2=A0 =C2=A0PyStackless_Schedule(Py_None, 0);<br> >>=C2=A0 =C2=A0 =C2=A0return 0;<br> >> }<br> >> ...<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Py_SetProgramName(argv[0]);<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Py_InitializeEx(0);<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (!Py_IsInitialized())<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0std::cerr << = "Python initialization failed" << std::endl;<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return;<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0}<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PySys_SetArgv(argc, argv);<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PyOS_InputHook =3D EventHook;<br> >> ...<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0// stacklessMain returns when pro= gram is shutdown.<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PyStackless_CallMethod_Main(oMain= Module.ptr(), "stacklessMain",<br> >> 0);<br> >> ...<br> >><br> >> This is the lldb stacktrace<br> >><br> >> frame #0: 0x0000000100ea30a3 Python`PyStackless_Schedule + 51<br> >><br> >>=C2=A0 frame #1: 0x0000000100155fd9 main`EventHook() + 25 at<br> >> pythonInterperter.cpp:472<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #2: 0x0000000103dd5e25 readline.so`call_r= eadline + 293<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #3: 0x0000000100dfa3c3 Python`PyOS_Readli= ne + 179<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #4: 0x0000000100e92882 Python`builtin_raw= _input + 434<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #5: 0x0000000100e95d88 Python`PyEval_Eval= Frame_value + 3832<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #6: 0x0000000100e94d38 Python`PyEval_Eval= FrameEx_slp + 376<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #7: 0x0000000100e9744c Python`PyEval_Eval= Frame_value + 9660<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #8: 0x0000000100e94d38 Python`PyEval_Eval= FrameEx_slp + 376<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #9: 0x0000000100e9744c Python`PyEval_Eval= Frame_value + 9660<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #10: 0x0000000100e94d38 Python`PyEval_Eva= lFrameEx_slp + 376<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #11: 0x0000000100e9e9e1 Python`slp_frame_= dispatch_top + 49<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #12: 0x0000000100ea2772 Python`slp_run_ta= sklet + 226<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #13: 0x0000000100e9e0fc Python`slp_eval_f= rame + 348<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #14: 0x0000000100e9e083 Python`slp_eval_f= rame + 227<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #15: 0x0000000100e9e083 Python`slp_eval_f= rame + 227<br> >><br> >> .....<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #719: 0x0000000100e9e083 Python`slp_eval_= frame + 227<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #720: 0x0000000100e9e083 Python`slp_eval_= frame + 227<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #721: 0x0000000100ea3801 Python`PyStackle= ss_CallMethod_Main +<br> >> 433<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #722: 0x00000001001588e3<br> >> main`PythonThread::operator(this=3D0x0000000101506940)() + 1187 at= <br> >> pythonInterperter.cpp:555<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #723: 0x000000010015761c<br> >> main`boost::detail::thread_data<PythonThread>::run(this=3D0x= 0000000101506740)<br> >> + 28 at thread.hpp:117<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #724: 0x00000001010ff24a libboost_thread-= mt.dylib`thread_proxy<br> >> +<br> >> 186<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #725: 0x00007fff863fb899 libsystem_pthrea= d.dylib`_pthread_body<br> >> +<br> >> 138<br> >><br> >>=C2=A0 =C2=A0 =C2=A0frame #726: 0x00007fff863fb72a libsystem_pthrea= d.dylib`_pthread_start<br> >> + 137<br> >><br> >> On Tue, Sep 9, 2014 at 9:17 PM, Richard Tew <<a href=3D"mailto:= [email protected]" target=3D"_blank">[email protected]</a>><= br> >> wrote:<br> >><br> >>> If you have a crash when you call receive, then you likely hav= e larger<br> >>> problems than when the right time to call schedule is.<br> >>><br> >>> I don't think it's possible to divine what is going wr= ong, and expect<br> >>> you need to provide a reproduction case.<br> >>><br> >>> Cheers<br> >>><br> >>> On 9/10/14, Robert Babiak <<a href=3D"mailto:rbabiak@gmail.= com" target=3D"_blank">[email protected]</a>> wrote:<br> >>> > Dear wise stackless list, I need some of your wisdom.<br> >>> ><br> >>> > I am trying to make raw_input play nice with the stackles= s scheduler<br> >>> > and<br> >>> > not block with my embedded stackless.<br> >>> ><br> >>> > I set the PyOS_InputHook to get what amounts to an idle e= vent but when<br> >>> > I<br> >>> > call PyStackless_Schedule(PY_None, 0) it crashes (no stac= kless<br> >>> symbols...)<br> >>> ><br> >>> > Is there something I need to check to know if it is safe = to call the<br> >>> > scheduler from C?<br> >>> ><br> >>> > In general when is it safe to call=C2=A0 PyStackless_Sche= dule?<br> >>> ><br> >>> > I tried to call my BeNice to block and get woken up later= , but this<br> >>> > also<br> >>> > crashes when trying to receive on my BeNice channel.<br> >>> ><br> >>> > - Thanks Rob.<br> >>> > --<br> >>> > Life: Bah, I will worry about it when it is over.<br> >>> ><br> >>><br> >>> _______________________________________________<br> >>> Stackless mailing list<br> >>> <a href=3D"mailto:[email protected]" target=3D"_blank">S= [email protected]</a><br> >>> <a href=3D"http://www.stackless.com/mailman/listinfo/stackless= " target=3D"_blank">http://www.stackless.com/mailman/listinfo/stackless</a>= <br> >>><br> >><br> >><br> >><br> >> --<br> >> Life: Bah, I will worry about it when it is over.<br> >><br> ><br> ><br> ><br> > --<br> > Life: Bah, I will worry about it when it is over.<br> ><br> <br> _______________________________________________<br> Stackless mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">Stackless@stac= kless.com</a><br> <a href=3D"http://www.stackless.com/mailman/listinfo/stackless" target=3D"_= blank">http://www.stackless.com/mailman/listinfo/stackless</a><br> </div></div></blockquote></div><br></div> </div></div></div></div></blockquote></div><br></div> <br>_______________________________________________<br> Stackless mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">Stackless@stac= kless.com</a><br> <a href=3D"http://www.stackless.com/mailman/listinfo/stackless" target=3D"_= blank">http://www.stackless.com/mailman/listinfo/stackless</a><br></blockqu= ote></div><br><br clear=3D"all"><div><br></div>-- <br>Life: Bah, I will wor= ry about it when it is over. </div> </div></div></blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>= Life: Bah, I will worry about it when it is over. </div> --001a11c29f3ee5a37e0502b5eae8-- --===============4366623348249355656== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Stackless mailing list [email protected] http://www.stackless.com/mailman/listinfo/stackless --===============4366623348249355656==--