Re: async
Andrew Z <[email protected]> Thu, 26 Oct 2017 10:05:02 -0400
| Newsgroups | gmane.comp.python.db.cx-oracle |
|---|---|
| Message-ID | <CAPrcKXL0X0UK4XhSA67nL2GVF2NwsAMpk0XZXiQnbUmYvE45jQ@mail.gmail.com> |
--===============2119520385261374087== Content-Type: multipart/alternative; boundary="001a114bbc20a09f3f055c73a827" --001a114bbc20a09f3f055c73a827 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Norman, True. That is exactly how i use it. If you think about it, the "executer" is for those routines that can not be execited in the same thread ( blocking ). In the sense, running part of a program using asyncio single thread model and then spawning tons of threads for DB, is inconsistent from architectural perspective. We may ask - why bother with asyncio, if i have to use threads for a db. Or even, use other "asyncio compatible " driver + db - postgres, mysql, mongo those drivers are "asyncio ready" - https://aiopg.readthedocs.io/en/stable/ http://motor.readthedocs.io/en/stable/tutorial-asyncio.html https://github.com/aio-libs/aiomysql I also found these articles/discussions useful presenting different view points: http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ https://groups.google.com/forum/m/#!topic/zodb/ON3Q3E8HWtM https://news.ycombinator.com/item?id=3D9053286 On Oct 26, 2017 06:39, "Norman Denayer" <[email protected]> wrote: Hello Walter, (We can use cx_Oracle with these frameworks now, but during any calls to the database the code can do nothing else except wait for the answer from the database). Actually, cx_Oracle use the macros Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS to let other threads run during driver calls and database work. So even without an async version of cx_Oracle, you may execute your calls in a =E2=80=9C*concurrent.futures.ThreadPoolExecutor*=E2=80=9D Of course it=E2=80=99s not perfect but it allows async code still call non-= async code in a non-blocking way. See: https://docs.python.org/3/library/asyncio-eventloop.html#executor https://docs.python.org/3/library/concurrent.futures. html#concurrent.futures.ThreadPoolExecutor Norman On 26 Oct 2017, at 10:47, Walter D=C3=B6rwald <[email protected]> wrote= : On 24 Oct 2017, at 19:57, Anthony Tuininga wrote: Hi Andrew, Apologies for the delay in response. I had a busy month. :-) There are no current plans to make cx_Oracle "compatible" with asyncio, but that could change -- especially if you provided some indication of what you meant by that request! It might be useful to create an issue on the cx_Oracle issues page providing some more details. That will give others a chance to comment a well, since not everyone is on the mailing list. I guess this would mean that for each method that talks to the database there should be an async version, i.e. instead of def print_foo(): cursor.execute("select * from foo") for row in cursor: print(row) we could do: async def print_foo(): await cursor.execute("select * from foo") async for row in cursor: print(row) Then we could use cx_Oracle with other async frameworks (aiohttp, sanic and probably Tornado and Twisted and a bunch others) and take advantage of cx_Oracle's new async features. (We can use cx_Oracle with these frameworks now, but during any calls to the database the code can do nothing else except wait for the answer from the database). Details for async/await are in PEP 492 at https://www.python.org/dev/ peps/pep-0492/ Anthony On Mon, Sep 25, 2017 at 10:22 PM, Andrew Zyman <[email protected]> wrote: Hello, wonder if there are any plans to make the module "compatible" with asyncio ? Thank you AZ Servus, Walter ------------------------------------------------------------ ------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ cx-oracle-users mailing list cx-oracle-users-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org https://lists.sourceforge.net/lists/listinfo/cx-oracle-users ------------------------------------------------------------ ------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ cx-oracle-users mailing list cx-oracle-users-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org https://lists.sourceforge.net/lists/listinfo/cx-oracle-users --001a114bbc20a09f3f055c73a827 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"auto">Norman,=C2=A0<div dir=3D"auto">True. That is exactly how = i use it. If you think about it, the "executer" is for those rout= ines that can not be execited in the same thread ( blocking ). In the sense= , running part of a program using asyncio single thread model and then spaw= ning tons of threads for DB, is inconsistent from architectural perspective= . We may ask - why bother with asyncio, if i have to use threads for a db. = Or even, use other "asyncio compatible " driver + db - postgres, = mysql, mongo those drivers are "asyncio ready" -</div><div dir=3D= "auto"><br></div><div dir=3D"auto"><a href=3D"https://aiopg.readthedocs.io/= en/stable/">https://aiopg.readthedocs.io/en/stable/</a></div><div dir=3D"au= to"><a href=3D"http://motor.readthedocs.io/en/stable/tutorial-asyncio.html"= >http://motor.readthedocs.io/en/stable/tutorial-asyncio.html</a></div><div = dir=3D"auto"><a href=3D"https://github.com/aio-libs/aiomysql">https://githu= b.com/aio-libs/aiomysql</a><br></div><div dir=3D"auto"><br></div><div dir= =3D"auto">I also found these articles/discussions useful presenting differe= nt view points:</div><div dir=3D"auto"><a href=3D"http://techspot.zzzeek.or= g/2015/02/15/asynchronous-python-and-databases/" target=3D"_blank">http://t= echspot.zzzeek.org/<wbr>2015/02/15/asynchronous-<wbr>python-and-databases/<= /a></div><div dir=3D"auto"><a href=3D"https://groups.google.com/forum/m/#!t= opic/zodb/ON3Q3E8HWtM">https://groups.google.com/forum/m/#!topic/zodb/ON3Q3= E8HWtM</a></div><div dir=3D"auto"><a href=3D"https://news.ycombinator.com/i= tem?id=3D9053286">https://news.ycombinator.com/item?id=3D9053286</a><br></d= iv></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Oct 2= 6, 2017 06:39, "Norman Denayer" <<a href=3D"mailto:denayer.nor= [email protected]">[email protected]</a>> wrote:<br type=3D"attributi= on"><blockquote class=3D"quote" style=3D"margin:0 0 0 .8ex;border-left:1px = #ccc solid;padding-left:1ex"><div style=3D"word-wrap:break-word">Hello Walt= er,<div class=3D"quoted-text"><div><br></div><blockquote type=3D"cite">(We = can use cx_Oracle with these frameworks now, but during any calls to the da= tabase the code can do nothing else except wait for the answer from the dat= abase).<br></blockquote><div>=C2=A0</div></div><div>Actually, cx_Oracle use= the macros=C2=A0<span style=3D"color:rgb(36,41,46);font-family:SFMono-Regu= lar,Consolas,'Liberation Mono',Menlo,Courier,monospace;white-space:= pre-wrap;background-color:rgb(255,255,255)">Py_BEGIN_ALLOW_THREADS and </sp= an><span style=3D"color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,&= #39;Liberation Mono',Menlo,Courier,monospace;white-space:pre-wrap;backg= round-color:rgb(255,255,255)">Py_END_ALLOW_THREADS t</span><span style=3D"c= olor:rgb(36,41,46);white-space:pre-wrap;background-color:rgb(255,255,255)">= o let other threads run during driver calls and database work.</span></div>= <div><span style=3D"color:rgb(36,41,46);white-space:pre-wrap;background-col= or:rgb(255,255,255)">So even without an async version of cx_Oracle, you may= execute your calls in a </span><font color=3D"#24292e"><span style=3D"whit= e-space:pre-wrap">=E2=80=9C<u>concurrent.futures.<wbr>ThreadPoolExecutor</u= >=E2=80=9D</span> </font></div><div><font color=3D"#24292e"><span style=3D"= white-space:pre-wrap"><br></span></font></div><div><font color=3D"#24292e">= <span style=3D"white-space:pre-wrap">Of course it=E2=80=99s not perfect but= it allows async code still call non-async code in a non-blocking way.</spa= n></font></div><div><font color=3D"#24292e"><span style=3D"white-space:pre-= wrap"> </span></font></div><div><font color=3D"#24292e" face=3D"SFMono-Regu= lar, Consolas, Liberation Mono, Menlo, Courier, monospace"><span style=3D"w= hite-space:pre-wrap">See:</span></font></div><div><span style=3D"background= -color:rgb(255,255,255);white-space:pre-wrap"><font color=3D"#24292e" face= =3D"SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><= a href=3D"https://docs.python.org/3/library/asyncio-eventloop.html#executor= " target=3D"_blank">https://docs.python.org/3/<wbr>library/asyncio-eventloo= p.<wbr>html#executor</a></font></span></div><div><span style=3D"background-= color:rgb(255,255,255);white-space:pre-wrap"><font color=3D"#24292e" face= =3D"SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><= a href=3D"https://docs.python.org/3/library/concurrent.futures.html#concurr= ent.futures.ThreadPoolExecutor" target=3D"_blank">https://docs.python.org/3= /<wbr>library/concurrent.futures.<wbr>html#concurrent.futures.<wbr>ThreadPo= olExecutor</a></font></span></div><font color=3D"#888888"><div><span style= =3D"background-color:rgb(255,255,255);white-space:pre-wrap"><font color=3D"= #24292e" face=3D"SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier,= monospace"><br></font></span></div><div><span style=3D"background-color:rg= b(255,255,255);white-space:pre-wrap"><font color=3D"#24292e" face=3D"SFMono= -Regular, Consolas, Liberation Mono, Menlo, Courier, monospace">Norman</fon= t></span></div></font><div class=3D"elided-text"><div><br><div><blockquote = type=3D"cite"><div>On 26 Oct 2017, at 10:47, Walter D=C3=B6rwald <<a hre= f=3D"mailto:[email protected]" target=3D"_blank">[email protected]<= /a>> wrote:</div><br class=3D"m_1231080149851261391Apple-interchange-new= line"><div><div>On 24 Oct 2017, at 19:57, Anthony Tuininga wrote:<br><br><b= lockquote type=3D"cite">Hi Andrew,<br><br>Apologies for the delay in respon= se. I had a busy month. :-)<br><br>There are no current plans to make cx_Or= acle "compatible" with asyncio, but<br>that could change -- espec= ially if you provided some indication of what you<br>meant by that request!= It might be useful to create an issue on the<br>cx_Oracle issues page prov= iding some more details. That will give others a<br>chance to comment a wel= l, since not everyone is on the mailing list.<br></blockquote><br>I guess t= his would mean that for each method that talks to the database there should= be an async version, i.e. instead of<br><br> =C2=A0=C2=A0=C2=A0def print_f= oo():<br> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0cursor.execute("se= lect * from foo")<br> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0for ro= w in cursor:<br><span class=3D"m_1231080149851261391Apple-tab-span" style= =3D"white-space:pre-wrap"> </span> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0print(row)<br><br>we could do:<br><br> =C2=A0=C2=A0=C2=A0async def print= _foo():<br> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0await cursor.execute(= "select * from foo")<br> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0async for row in cursor:<br> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0print(row)<br><br>Then we could use cx_Oracle with = other async frameworks (aiohttp, sanic and probably Tornado and Twisted and= a bunch others) and take advantage of cx_Oracle's new async features.<= br><br>(We can use cx_Oracle with these frameworks now, but during any call= s to the database the code can do nothing else except wait for the answer f= rom the database).<br><br>Details for async/await are in PEP 492 at <a href= =3D"https://www.python.org/dev/peps/pep-0492/" target=3D"_blank">https://ww= w.python.org/dev/<wbr>peps/pep-0492/</a><br><br><blockquote type=3D"cite">A= nthony<br><br>On Mon, Sep 25, 2017 at 10:22 PM, Andrew Zyman <<a href=3D= "mailto:[email protected]" target=3D"_blank">[email protected]</a>> wrot= e:<br><br><blockquote type=3D"cite">Hello,<br> wonder if there are any plan= s to make the module "compatible" with<br>asyncio ?<br>Thank you<= br>AZ<br></blockquote></blockquote><br>Servus,<br> =C2=A0=C2=A0Walter<br><b= r>------------------------------<wbr>------------------------------<wbr>---= ---------------<br>Check out the vibrant tech community on one of the world= 's most<br>engaging tech sites, <a href=3D"http://Slashdot.org" target= =3D"_blank">Slashdot.org</a>! <a href=3D"http://sdm.link/slashdot" target= =3D"_blank">http://sdm.link/slashdot</a><br>______________________________<= wbr>_________________<br>cx-oracle-users mailing list<br><a href=3D"mailto:= cx-oracle-users-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org" target=3D"_blank">cx-oracle-users@li= sts.<wbr>sourceforge.net</a><br><a href=3D"https://lists.sourceforge.net/li= sts/listinfo/cx-oracle-users" target=3D"_blank">https://lists.sourceforge.n= et/<wbr>lists/listinfo/cx-oracle-users</a><br></div></div></blockquote></di= v><br></div></div></div><br>------------------------------<wbr>------------= ------------------<wbr>------------------<br> Check out the vibrant tech community on one of the world's most<br> engaging tech sites, Slashdot.org! <a href=3D"http://sdm.link/slashdot" rel= =3D"noreferrer" target=3D"_blank">http://sdm.link/slashdot</a><br>_________= _____________________<wbr>_________________<br> cx-oracle-users mailing list<br> <a href=3D"mailto:cx-oracle-users-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org">cx-oracle-users@li= sts.<wbr>sourceforge.net</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/cx-oracle-users" re= l=3D"noreferrer" target=3D"_blank">https://lists.sourceforge.net/<wbr>lists= /listinfo/cx-oracle-users</a><br> <br></blockquote></div><br></div> --001a114bbc20a09f3f055c73a827-- --===============2119520385261374087== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot --===============2119520385261374087== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ cx-oracle-users mailing list cx-oracle-users-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org https://lists.sourceforge.net/lists/listinfo/cx-oracle-users --===============2119520385261374087==--