Re: Implementing socket-io for Stackless (Kristj?n Valur J?nsson)
Andrew Francis <andrewfr_ice-/[email protected]> Fri, 6 Mar 2015 13:17:59 +0000 (UTC)
| Newsgroups | gmane.comp.python.stackless |
|---|---|
| Message-ID | <[email protected]> |
--===============8075351101247986243==
Content-Type: multipart/alternative;
boundary="----=_Part_5896306_330823641.1425647879112"
Content-Length: 18901
------=_Part_5896306_330823641.1425647879112
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Hi Kristjan:
Thanks for the reply. I will take another look at stackless-io when it beco=
mes available. For now, I guess my concerns=C2=A0revolve around starting. I=
am starting to =C2=A0look at the socket-io protocol. I am wondering if the=
re are any websocket Stackless Pythonimplementations. I should look at how =
gEvent does things.
Done right, I think Stackless can provide a more developer-friendly impleme=
ntation of Socket-io than Node.js can.
Cheers,Andrew=20
On Thursday, March 5, 2015 6:00 AM, "[email protected]" =
<[email protected]> wrote:
=20
Send Stackless mailing list submissions to
=C2=A0=C2=A0=C2=A0 [email protected]
To subscribe or unsubscribe via the World Wide Web, visit
=C2=A0=C2=A0=C2=A0 http://www.stackless.com/mailman/listinfo/stackless
or, via email, send a message with subject or body 'help' to
=C2=A0=C2=A0=C2=A0 [email protected]
You can reach the person managing the list at
=C2=A0=C2=A0=C2=A0 [email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Stackless digest..."
Today's Topics:
=C2=A0 1. Re: Implementing socket-io for Stackless (Kristj?n Valur J?nsson)
----------------------------------------------------------------------
Message: 1
Date: Thu, 5 Mar 2015 10:00:22 +0000
From: Kristj?n Valur J?nsson <[email protected]>
To: Andrew Francis <andrewfr_ice-/[email protected]>, =C2=A0=C2=A0=C2=A0 The Stackle=
ss Python
=C2=A0=C2=A0=C2=A0 Mailing List <[email protected]>
Subject: Re: [Stackless] Implementing socket-io for Stackless
Message-ID:
=C2=A0=C2=A0=C2=A0 <CAP8kY6bsTAvyhY+aDPsbzdt0dcimo1kAhC3=3DEkptfs=3D_r8GU=
[email protected]>
Content-Type: text/plain; charset=3D"utf-8"
Ah,
When I left CCP I got permission to take StacklessIO and make it open
source.=C2=A0 In my last months there, I had been working on a Mac OSX vers=
ion
of it.
This would port very easily to linux, with some minor work (mac OSX has a
built in thread pool library that linux doesn?t, and threads are necessary
for some operations that don't have async versions)
Anyway,=C2=A0 I was looking through my source for this stuff and couldn't f=
ind
it, I need to get it from there...
In the end, it is all about how you turn your asynchronous api that the OS
provides you with, into a syncronous one for stackless.=C2=A0 The approach =
is
different based on the type of API.
For windows, the api is completion based.=C2=A0 You get a notification when
something is done.
On linux, it is retry-based, you are invited to invite a non-blocking
operation when it is deemed likely to succeed.
It turns out that the linux approach is much easier to implement.=C2=A0 The
windows version was full of all kinds of boilerplate and stuff, mostly
because the "outstanding" requests need so much luggage with them while in
flight (return buffers, etc).
The macos version of stacklessio performed pretty well.=C2=A0 I will make t=
his
public, but the outline I employed in stacklessio was this:
1) there was a central event queue serviced by the main thread.=C2=A0 This =
was
implemented in C and we wold deposit there callbacks that the main thread
should run.
2) a socket request would first try to, e.g. recv().=C2=A0 if it got
EWOULDBLOCK, it would create a "read request", mark the socket for being
watched by someone using select (or equivalent) and then go to sleep, using
a channel-receive on the read request.
3) A worker thread would monitor the sockets marked for IO and their
associated "request" objects.=C2=A0 If the socket is marked as readable, it
would then try reading from it.=C2=A0 If it succeeded, it would mark that
request complete by putting it into the IO event queue, and waking up the
main thread if it was sleepeing, waiting for IO.
4) the main thread would wake up, look at the io queue, see a complete
request there, call its callback, the callback would call channel-send on
the object, and the original stackless reader would wake up.
There are a lot of corner cases and details.
The whole thing was implemented in C++ to maintain object lifetimes and for
other niceties.
The important thing for any stackless application, or an IO driven
application like this, is that there be only one Event queue, that the main
thread services.=C2=A0 It has to bee application wide.
This is why I was so optimistic about Guido's Tulip proposal, that it would
give us an application wide event loop that different frameworks could all
use cooperatively.
Unfortunately, it turned out to be yet another proprietary socket io
mechanism....
K
On 27 February 2015 at 16:16, Andrew Francis <andrewfr_ice-/[email protected]> wrote=
:
> Hi Folks:
>
> Pycon 2015 is coming soon. Lately I have been doing work with node.js. I
> am not fond of node.js. I have looked at the socket-io protocol. I am
> wondering about two things:
>
> 1) Is there any advice about implementing socket-io?
> 2) What does a Stackless Python friend socket-io implementation look like=
?
> I have looked at Go socket-io implementations and they look like Javascri=
pt.
>
> Cheers,
> Andrew
>
> _______________________________________________
> Stackless mailing list
> [email protected]
> http://www.stackless.com/mailman/listinfo/stackless
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20150305/18f=
03f15/attachment-0001.html>
------------------------------
_______________________________________________
Stackless mailing list
[email protected]
http://www.stackless.com/mailman/listinfo/stackless
End of Stackless Digest, Vol 133, Issue 1
*****************************************
------=_Part_5896306_330823641.1425647879112
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"color:#000; background-color:#fff; font-family:He=
lveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;fo=
nt-size:12px"><div id=3D"yiv4459084090"><div id=3D"yui_3_16_0_1_14256468775=
33_7773"><div style=3D"color:#000;background-color:#fff;font-family:Helveti=
caNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-si=
ze:12px;" id=3D"yui_3_16_0_1_1425646877533_7772"><div dir=3D"ltr" id=3D"yiv=
4459084090yui_3_16_0_1_1425646877533_3814"><span>Hi Kristjan:</span></div><=
div dir=3D"ltr" id=3D"yiv4459084090yui_3_16_0_1_1425646877533_3814"><span><=
br clear=3D"none"></span></div><div dir=3D"ltr" id=3D"yiv4459084090yui_3_16=
_0_1_1425646877533_3814"><span id=3D"yui_3_16_0_1_1425646877533_7780">Thank=
s for the reply. I will take another look at stackless-io when it becomes a=
vailable. For now, I guess my concerns </span></div><div dir=3D"ltr" i=
d=3D"yiv4459084090yui_3_16_0_1_1425646877533_3814"><span id=3D"yui_3_16_0_1=
_1425646877533_7999">revolve around starting. I am starting to look a=
t the socket-io protocol. I am wondering if there are any websocket Stackle=
ss Python</span></div><div dir=3D"ltr" id=3D"yiv4459084090yui_3_16_0_1_1425=
646877533_3814"><span id=3D"yui_3_16_0_1_1425646877533_8353">implementation=
s. I should look at how gEvent does things.</span></div><div dir=3D"ltr" id=
=3D"yiv4459084090yui_3_16_0_1_1425646877533_3814"><span><br></span></div><d=
iv dir=3D"ltr" id=3D"yiv4459084090yui_3_16_0_1_1425646877533_3814"><span id=
=3D"yui_3_16_0_1_1425646877533_8422">Done right, I think Stackless can prov=
ide a more developer-friendly implementation of Socket-io than Node.js can.=
</span></div><div dir=3D"ltr" id=3D"yiv4459084090yui_3_16_0_1_1425646877533=
_3814"><span><br></span></div><div dir=3D"ltr" id=3D"yiv4459084090yui_3_16_=
0_1_1425646877533_3814">Cheers,</div><div dir=3D"ltr" id=3D"yiv4459084090yu=
i_3_16_0_1_1425646877533_3814">Andrew</div> <div class=3D"yiv4459084090qtdS=
eparateBR" id=3D"yui_3_16_0_1_1425646877533_7782"><br clear=3D"none"><br cl=
ear=3D"none"></div><div class=3D"yiv4459084090yqt5947475630" id=3D"yiv44590=
84090yqt55403"><div class=3D"yiv4459084090yahoo_quoted" style=3D"display: b=
lock;" id=3D"yui_3_16_0_1_1425646877533_8429"> <div style=3D"font-family:He=
lveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;fo=
nt-size:12px;" id=3D"yui_3_16_0_1_1425646877533_8428"> <div style=3D"font-f=
amily:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-=
serif;font-size:16px;" id=3D"yui_3_16_0_1_1425646877533_8427"> <div dir=3D"=
ltr" id=3D"yui_3_16_0_1_1425646877533_8426"> <font size=3D"2" face=3D"Arial=
" id=3D"yui_3_16_0_1_1425646877533_8425"> On Thursday, March 5, 2015 6:00 A=
M, "[email protected]" <[email protected]>=
; wrote:<br clear=3D"none"> </font> </div> <br clear=3D"none"><br clear=3D=
"none"> <div class=3D"yiv4459084090y_msg_container" id=3D"yui_3_16_0_1_1425=
646877533_8433">Send Stackless mailing list submissions to<br clear=3D"none=
"> <a rel=3D"nofollow" shape=3D"rect" ymailto=3D"mailto:s=
[email protected]" target=3D"_blank" href=3D"mailto:stackless@stackles=
s.com" id=3D"yui_3_16_0_1_1425646877533_8432">[email protected]</a><b=
r clear=3D"none"><br clear=3D"none">To subscribe or unsubscribe via the Wor=
ld Wide Web, visit<br clear=3D"none"> <a rel=3D"nofollow"=
shape=3D"rect" target=3D"_blank" href=3D"http://www.stackless.com/mailman/=
listinfo/stackless">http://www.stackless.com/mailman/listinfo/stackless</a>=
<br clear=3D"none">or, via email, send a message with subject or body 'help=
' to<br clear=3D"none"> <a rel=3D"nofollow" shape=3D"rect=
" ymailto=3D"mailto:[email protected]" target=3D"_blank" href=
=3D"mailto:[email protected]" id=3D"yui_3_16_0_1_142564687753=
3_8436">[email protected]</a><br clear=3D"none"><br clear=3D"=
none">You can reach the person managing the list at<br clear=3D"none"> =
; <a rel=3D"nofollow" shape=3D"rect" ymailto=3D"mailto:stackles=
[email protected]" target=3D"_blank" href=3D"mailto:stackless-owner@sta=
ckless.com" id=3D"yui_3_16_0_1_1425646877533_8438">stackless-owner@stackles=
s.com</a><br clear=3D"none"><br clear=3D"none">When replying, please edit y=
our Subject line so it is more specific<br clear=3D"none">than "Re: Content=
s of Stackless digest..."<br clear=3D"none"><br clear=3D"none"><br clear=3D=
"none">Today's Topics:<br clear=3D"none"><br clear=3D"none"> 1. Re: =
Implementing socket-io for Stackless (Kristj?n Valur J?nsson)<br clear=3D"n=
one"><br clear=3D"none"><br clear=3D"none">--------------------------------=
--------------------------------------<br clear=3D"none"><br clear=3D"none"=
>Message: 1<br clear=3D"none">Date: Thu, 5 Mar 2015 10:00:22 +0000<br clear=
=3D"none">From: Kristj?n Valur J?nsson <<a rel=3D"nofollow" shape=3D"rec=
t" ymailto=3D"mailto:[email protected]" target=3D"_blank" href=3D"mailto:s=
[email protected]">[email protected]</a>><br clear=3D"none">To: Andrew =
Francis <<a rel=3D"nofollow" shape=3D"rect" ymailto=3D"mailto:andrewfr_i=
ce-/[email protected]" target=3D"_blank" href=3D"mailto:andrewfr_ice-/[email protected]">andr=
ewfr_ice-/[email protected]</a>>, The Stackless Python<br clea=
r=3D"none"> Mailing List <<a rel=3D"nofollow" shape=3D=
"rect" ymailto=3D"mailto:[email protected]" target=3D"_blank" href=3D=
"mailto:[email protected]">[email protected]</a>><br clear=
=3D"none">Subject: Re: [Stackless] Implementing socket-io for Stackless<br =
clear=3D"none">Message-ID:<br clear=3D"none"> <CAP8kY6=
bsTAvyhY+aDPsbzdt0dcimo1kAhC3=3DEkptfs=3D_r8GU=3D<a rel=3D"nofollow" shape=
=3D"rect" ymailto=3D"mailto:[email protected]" target=3D"_blank" href=3D"mai=
lto:[email protected]">[email protected]</a>><br clear=3D"none">Content-Ty=
pe: text/plain; charset=3D"utf-8"<br clear=3D"none"><br clear=3D"none">Ah,<=
br clear=3D"none">When I left CCP I got permission to take StacklessIO and =
make it open<br clear=3D"none">source. In my last months there, I had=
been working on a Mac OSX version<br clear=3D"none">of it.<br clear=3D"non=
e">This would port very easily to linux, with some minor work (mac OSX has =
a<br clear=3D"none">built in thread pool library that linux doesn?t, and th=
reads are necessary<br clear=3D"none">for some operations that don't have a=
sync versions)<br clear=3D"none">Anyway, I was looking through my sou=
rce for this stuff and couldn't find<br clear=3D"none">it, I need to get it=
from there...<br clear=3D"none"><br clear=3D"none">In the end, it is all a=
bout how you turn your asynchronous api that the OS<br clear=3D"none">provi=
des you with, into a syncronous one for stackless. The approach is<br=
clear=3D"none">different based on the type of API.<br clear=3D"none">For w=
indows, the api is completion based. You get a notification when<br c=
lear=3D"none">something is done.<br clear=3D"none">On linux, it is retry-ba=
sed, you are invited to invite a non-blocking<br clear=3D"none">operation w=
hen it is deemed likely to succeed.<br clear=3D"none"><br clear=3D"none">It=
turns out that the linux approach is much easier to implement. The<b=
r clear=3D"none">windows version was full of all kinds of boilerplate and s=
tuff, mostly<br clear=3D"none">because the "outstanding" requests need so m=
uch luggage with them while in<br clear=3D"none">flight (return buffers, et=
c).<br clear=3D"none"><br clear=3D"none">The macos version of stacklessio p=
erformed pretty well. I will make this<br clear=3D"none">public, but =
the outline I employed in stacklessio was this:<br clear=3D"none"><br clear=
=3D"none">1) there was a central event queue serviced by the main thread.&n=
bsp; This was<br clear=3D"none">implemented in C and we wold deposit there =
callbacks that the main thread<br clear=3D"none">should run.<br clear=3D"no=
ne">2) a socket request would first try to, e.g. recv(). if it got<br=
clear=3D"none">EWOULDBLOCK, it would create a "read request", mark the soc=
ket for being<br clear=3D"none">watched by someone using select (or equival=
ent) and then go to sleep, using<br clear=3D"none">a channel-receive on the=
read request.<br clear=3D"none"><br clear=3D"none">3) A worker thread woul=
d monitor the sockets marked for IO and their<br clear=3D"none">associated =
"request" objects. If the socket is marked as readable, it<br clear=
=3D"none">would then try reading from it. If it succeeded, it would m=
ark that<br clear=3D"none">request complete by putting it into the IO event=
queue, and waking up the<br clear=3D"none">main thread if it was sleepeing=
, waiting for IO.<br clear=3D"none"><br clear=3D"none">4) the main thread w=
ould wake up, look at the io queue, see a complete<br clear=3D"none">reques=
t there, call its callback, the callback would call channel-send on<br clea=
r=3D"none">the object, and the original stackless reader would wake up.<br =
clear=3D"none"><br clear=3D"none"><br clear=3D"none">There are a lot of cor=
ner cases and details.<br clear=3D"none">The whole thing was implemented in=
C++ to maintain object lifetimes and for<br clear=3D"none">other niceties.=
<br clear=3D"none"><br clear=3D"none">The important thing for any stackless=
application, or an IO driven<br clear=3D"none">application like this, is t=
hat there be only one Event queue, that the main<br clear=3D"none">thread s=
ervices. It has to bee application wide.<br clear=3D"none">This is wh=
y I was so optimistic about Guido's Tulip proposal, that it would<br clear=
=3D"none">give us an application wide event loop that different frameworks =
could all<br clear=3D"none">use cooperatively.<br clear=3D"none">Unfortunat=
ely, it turned out to be yet another proprietary socket io<br clear=3D"none=
">mechanism....<br clear=3D"none"><br clear=3D"none">K<br clear=3D"none"><b=
r clear=3D"none">On 27 February 2015 at 16:16, Andrew Francis <<a rel=3D=
"nofollow" shape=3D"rect" ymailto=3D"mailto:andrewfr_ice-/[email protected]" target=
=3D"_blank" href=3D"mailto:andrewfr_ice-/[email protected]">andrewfr_ice-/[email protected]</=
a>> wrote:<br clear=3D"none"><br clear=3D"none">> Hi Folks:<br clear=
=3D"none">><br clear=3D"none">> Pycon 2015 is coming soon. Lately I h=
ave been doing work with node.js. I<br clear=3D"none">> am not fond of n=
ode.js. I have looked at the socket-io protocol. I am<br clear=3D"none">>=
; wondering about two things:<br clear=3D"none">><br clear=3D"none">>=
1) Is there any advice about implementing socket-io?<br clear=3D"none">>=
; 2) What does a Stackless Python friend socket-io implementation look like=
?<br clear=3D"none">> I have looked at Go socket-io implementations and =
they look like Javascript.<br clear=3D"none">><br clear=3D"none">> Ch=
eers,<br clear=3D"none">> Andrew<br clear=3D"none">><br clear=3D"none=
">> _______________________________________________<br clear=3D"none">&g=
t; Stackless mailing list<br clear=3D"none">> <a rel=3D"nofollow" shape=
=3D"rect" ymailto=3D"mailto:[email protected]" target=3D"_blank" href=
=3D"mailto:[email protected]">[email protected]</a><br clear=3D=
"none">> <a rel=3D"nofollow" shape=3D"rect" target=3D"_blank" href=3D"ht=
tp://www.stackless.com/mailman/listinfo/stackless">http://www.stackless.com=
/mailman/listinfo/stackless</a><br clear=3D"none">><br clear=3D"none">--=
------------ next part --------------<br clear=3D"none">An HTML attachment =
was scrubbed...<br clear=3D"none">URL: <<a rel=3D"nofollow" shape=3D"rec=
t" target=3D"_blank" href=3D"http://www.stackless.com/pipermail/stackless/a=
ttachments/20150305/18f03f15/attachment-0001.html">http://www.stackless.com=
/pipermail/stackless/attachments/20150305/18f03f15/attachment-0001.html</a>=
><br clear=3D"none"><br clear=3D"none">------------------------------<br=
clear=3D"none"><br clear=3D"none">________________________________________=
_______<br clear=3D"none">Stackless mailing list<br clear=3D"none"><a rel=
=3D"nofollow" shape=3D"rect" ymailto=3D"mailto:[email protected]" tar=
get=3D"_blank" href=3D"mailto:[email protected]">Stackless@stackless.=
com</a><br clear=3D"none"><a rel=3D"nofollow" shape=3D"rect" target=3D"_bla=
nk" href=3D"http://www.stackless.com/mailman/listinfo/stackless">http://www=
.stackless.com/mailman/listinfo/stackless</a><br clear=3D"none"><br clear=
=3D"none">End of Stackless Digest, Vol 133, Issue 1<br clear=3D"none">*****=
************************************<br clear=3D"none"><br clear=3D"none"><=
br clear=3D"none"></div> </div> </div> </div></div> </div></div></div></d=
iv></body></html>
------=_Part_5896306_330823641.1425647879112--
--===============8075351101247986243==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Stackless mailing list
[email protected]
http://www.stackless.com/mailman/listinfo/stackless
--===============8075351101247986243==--