Re: PEP 547: Could we implement a usable "get_code()" for extension modules?
Brett Cannon <[email protected]> Sat, 13 Jan 2018 19:49:10 +0000
| Newsgroups | gmane.comp.python.import |
|---|---|
| Message-ID | <CAP1=2W4MqQr+b=TfC7TDpbpx=34nY7QD+W=_3VeOLuSAJZNcFQ@mail.gmail.com> |
--===============5167978709715652514== Content-Type: multipart/alternative; boundary="001a114d3e2a6c3d460562adad75" --001a114d3e2a6c3d460562adad75 Content-Type: text/plain; charset="UTF-8" Awesome! Thanks for looking into it. On Sat, Jan 13, 2018, 07:49 Petr Viktorin, <[email protected]> wrote: > On 01/12/2018 06:52 PM, Brett Cannon wrote: > > So obviously implementing get_code() for the extension module loader > > would be great. :) So the question becomes how? > > Marcel took a quick look at it already. It seems it's quite a simple > addition, and it makes tests developed for PEP 547 pass. Hopefully we > can have a PR early next week :) > > > > On Thu, 11 Jan 2018 at 22:57 Nick Coghlan <[email protected] > > <mailto:[email protected]>> wrote: > > > > (cc'ed a couple of folks that I expect will be interested in this > > question, but may not be subscribed to import-sig) > > > > The current version of PEP 547 (supporting the -m switch for > extension > > modules) works by defining a new optional "exec_in_module" API for > > loaders to implement, and then updating runpy._run_module_as_main to > > call it. > > > > However, reviewing Mario Corchero's patches for > > https://bugs.python.org/issue9325 (adding "-m" switch support to > > assorted modules) has highlighted a potential challenge with that > > approach: it turns out the most useful private API in runpy for > > emulating the -m switch is "mod_name, mod_spec, code = > > runpy._get_module_details(module_name)". > > > > That means that if we can figure out a way to have > > ExtensionFileLoader.get_code() emit a Python code object that > > delegates to Py_mod_exec, then we'd be well on our way to supporting > > "python -m <extension module>" without making *any changes to runpy* > > (or the other modules that are gaining "-m" equivalents). > > > > If we did decide to go down that path, the main way I could see it > > working without any new features in the C interface is to structure > > things such that the extension module would still run in its own > > namespace, with the interface adaptation code returned from > get_code() > > (after compilation) looking something like: > > > > ns = globals() > > if ns is not locals(): > > raise RuntimeError("Cannot execute extension module > > {<interpolated_name>} with separate local namespace") > > module = _imp.create_dynamic(<interpolated_spec_details>) > > module.__dict__.update(ns) > > _imp.exec_dynamic(module) > > ns.update(module.__dict__) > > > > The biggest advantages of this approach are that it would still work > > for Cython (and other) modules that defined Py_mod_create, and it > > would implicitly interoperate (at least to some degree) with anything > > that relied on the "get code and exec it" model of interacting with > > Python modules. > > > > Alternatively, we could instead push the decision on how to handle > > this case down to extension module authors as follows: > > > > 1. Define a new Py_mod_exec_in_namespace slot that accepts a target > > namespace as its parameter instead of a pre-existing module > > 2. Add a new "_imp.exec_dynamic_in_namespace(spec, namespace)" API > > 3. When Py_mod_exec_in_namespace is defined, make the adapter code > > look something like: > > > > ns = globals() > > if ns is not locals(): > > import collections > > ns = collections.ChainMap(locals(), ns) > > _imp.exec_dynamic_in_namespace(<interpolated_spec_details>, ns) > > > > (There are several ways the functionality could be split up between > > the generated code and the _imp module, this is just an example that > > suggests the idea is technically feasible) > > > > The nice thing about including the new slot in the design is that it > > gives extension modules a way to avoid the overhead of copying > > attributes in and out, as would be needed if relying solely on the > PEP > > 489 APIs. > > > > Cheers, > > Nick. > > > > P.S. Given these changes we could technically define "get_source()" > on > > extension modules as well, but that doesn't seem especially useful. > > > _______________________________________________ > Import-SIG mailing list > [email protected] > https://mail.python.org/mailman/listinfo/import-sig > --001a114d3e2a6c3d460562adad75 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <p dir=3D"ltr">Awesome! Thanks for looking into it.</p> <br><div class=3D"gmail_quote"><div dir=3D"ltr">On Sat, Jan 13, 2018, 07:49= Petr Viktorin, <<a href=3D"mailto:[email protected]">[email protected]<= /a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0= 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 01/12/2018 06:52 = PM, Brett Cannon wrote:<br> > So obviously implementing get_code() for the extension module loader<b= r> > would be great. :) So the question becomes how?<br> <br> Marcel took a quick look at it already. It seems it's quite a simple<br= > addition, and it makes tests developed for PEP 547 pass. Hopefully we<br> can have a PR early next week :)<br> <br> <br> > On Thu, 11 Jan 2018 at 22:57 Nick Coghlan <<a href=3D"mailto:ncoghl= [email protected]" target=3D"_blank">[email protected]</a><br> > <mailto:<a href=3D"mailto:[email protected]" target=3D"_blank">nco= [email protected]</a>>> wrote:<br> ><br> >=C2=A0 =C2=A0 =C2=A0(cc'ed a couple of folks that I expect will be = interested in this<br> >=C2=A0 =C2=A0 =C2=A0question, but may not be subscribed to import-sig)<= br> ><br> >=C2=A0 =C2=A0 =C2=A0The current version of PEP 547 (supporting the -m s= witch for extension<br> >=C2=A0 =C2=A0 =C2=A0modules) works by defining a new optional "exe= c_in_module" API for<br> >=C2=A0 =C2=A0 =C2=A0loaders to implement, and then updating runpy._run_= module_as_main to<br> >=C2=A0 =C2=A0 =C2=A0call it.<br> ><br> >=C2=A0 =C2=A0 =C2=A0However, reviewing Mario Corchero's patches for= <br> >=C2=A0 =C2=A0 =C2=A0<a href=3D"https://bugs.python.org/issue9325" rel= =3D"noreferrer" target=3D"_blank">https://bugs.python.org/issue9325</a> (ad= ding "-m" switch support to<br> >=C2=A0 =C2=A0 =C2=A0assorted modules) has highlighted a potential chall= enge with that<br> >=C2=A0 =C2=A0 =C2=A0approach: it turns out the most useful private API = in runpy for<br> >=C2=A0 =C2=A0 =C2=A0emulating the -m switch is "mod_name, mod_spec= , code =3D<br> >=C2=A0 =C2=A0 =C2=A0runpy._get_module_details(module_name)".<br> ><br> >=C2=A0 =C2=A0 =C2=A0That means that if we can figure out a way to have<= br> >=C2=A0 =C2=A0 =C2=A0ExtensionFileLoader.get_code() emit a Python code o= bject that<br> >=C2=A0 =C2=A0 =C2=A0delegates to Py_mod_exec, then we'd be well on = our way to supporting<br> >=C2=A0 =C2=A0 =C2=A0"python -m <extension module>" with= out making *any changes to runpy*<br> >=C2=A0 =C2=A0 =C2=A0(or the other modules that are gaining "-m&quo= t; equivalents).<br> ><br> >=C2=A0 =C2=A0 =C2=A0If we did decide to go down that path, the main way= I could see it<br> >=C2=A0 =C2=A0 =C2=A0working without any new features in the C interface= is to structure<br> >=C2=A0 =C2=A0 =C2=A0things such that the extension module would still r= un in its own<br> >=C2=A0 =C2=A0 =C2=A0namespace, with the interface adaptation code retur= ned from get_code()<br> >=C2=A0 =C2=A0 =C2=A0(after compilation) looking something like:<br> ><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ns =3D globals()<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ns is not locals():<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 raise RuntimeError(&qu= ot;Cannot execute extension module<br> >=C2=A0 =C2=A0 =C2=A0{<interpolated_name>} with separate local nam= espace")<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 module =3D _imp.create_dynamic(<i= nterpolated_spec_details>)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 module.__dict__.update(ns)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 _imp.exec_dynamic(module)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ns.update(module.__dict__)<br> ><br> >=C2=A0 =C2=A0 =C2=A0The biggest advantages of this approach are that it= would still work<br> >=C2=A0 =C2=A0 =C2=A0for Cython (and other) modules that defined Py_mod_= create, and it<br> >=C2=A0 =C2=A0 =C2=A0would implicitly interoperate (at least to some deg= ree) with anything<br> >=C2=A0 =C2=A0 =C2=A0that relied on the "get code and exec it"= model of interacting with<br> >=C2=A0 =C2=A0 =C2=A0Python modules.<br> ><br> >=C2=A0 =C2=A0 =C2=A0Alternatively, we could instead push the decision o= n how to handle<br> >=C2=A0 =C2=A0 =C2=A0this case down to extension module authors as follo= ws:<br> ><br> >=C2=A0 =C2=A0 =C2=A01. Define a new Py_mod_exec_in_namespace slot that = accepts a target<br> >=C2=A0 =C2=A0 =C2=A0namespace as its parameter instead of a pre-existin= g module<br> >=C2=A0 =C2=A0 =C2=A02. Add a new "_imp.exec_dynamic_in_namespace(s= pec, namespace)" API<br> >=C2=A0 =C2=A0 =C2=A03. When Py_mod_exec_in_namespace is defined, make t= he adapter code<br> >=C2=A0 =C2=A0 =C2=A0look something like:<br> ><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ns =3D globals()<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ns is not locals():<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 import collections<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ns =3D collections.Cha= inMap(locals(), ns)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 _imp.exec_dynamic_in_namespace(<i= nterpolated_spec_details>, ns)<br> ><br> >=C2=A0 =C2=A0 =C2=A0(There are several ways the functionality could be = split up between<br> >=C2=A0 =C2=A0 =C2=A0the generated code and the _imp module, this is jus= t an example that<br> >=C2=A0 =C2=A0 =C2=A0suggests the idea is technically feasible)<br> ><br> >=C2=A0 =C2=A0 =C2=A0The nice thing about including the new slot in the = design is that it<br> >=C2=A0 =C2=A0 =C2=A0gives extension modules a way to avoid the overhead= of copying<br> >=C2=A0 =C2=A0 =C2=A0attributes in and out, as would be needed if relyin= g solely on the PEP<br> >=C2=A0 =C2=A0 =C2=A0489 APIs.<br> ><br> >=C2=A0 =C2=A0 =C2=A0Cheers,<br> >=C2=A0 =C2=A0 =C2=A0Nick.<br> ><br> >=C2=A0 =C2=A0 =C2=A0P.S. Given these changes we could technically defin= e "get_source()" on<br> >=C2=A0 =C2=A0 =C2=A0extension modules as well, but that doesn't see= m especially useful.<br> ><br> _______________________________________________<br> Import-SIG mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">Import-SIG@pytho= n.org</a><br> <a href=3D"https://mail.python.org/mailman/listinfo/import-sig" rel=3D"nore= ferrer" target=3D"_blank">https://mail.python.org/mailman/listinfo/import-s= ig</a><br> </blockquote></div> --001a114d3e2a6c3d460562adad75-- --===============5167978709715652514== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Import-SIG mailing list [email protected] https://mail.python.org/mailman/listinfo/import-sig --===============5167978709715652514==--