Re: PEP 547: Could we implement a usable "get_code()" for extension modules?
Brett Cannon <[email protected]> Tue, 16 Jan 2018 17:08:57 +0000
| Newsgroups | gmane.comp.python.import |
|---|---|
| Message-ID | <CAP1=2W7aRfWZHMGCrNW_sVm+YFnozt0tJcAjpzBoTjwqTJdL2A@mail.gmail.com> |
--===============1600739487883680985== Content-Type: multipart/alternative; boundary="f403045e7304f569c70562e7c9f1" --f403045e7304f569c70562e7c9f1 Content-Type: text/plain; charset="UTF-8" On Tue, 16 Jan 2018 at 05:43 Marcel Plch <[email protected]> wrote: > I took a look at the get_code() and it works basically just as is in > Nick's mail. Spec and name are accessible through passed globals. > Unlike the current proposal for -m switch for extension modules [PR > 1761], this approach does not require multiphase initialization. That > means that every module can be run with this switch just as it is > right now, including the standard library: > > $ python -im math > >>> print(e) > 2.718281828459045 > > Should I open a bug for this, or reuse [bpo-30403]? If you want, although a new issue is also totally justified. > Or does this need a PEP? > I don't think so since you're just implementing a pre-existing API and this doesn't require changing any pre-existing semantics. -Brett > > You can see the required changes here: > https://github.com/Traceur759/cpython/pull/6/files > > [PR 1761]: https://github.com/python/cpython/pull/1761 > [bpo-30403]: https://bugs.python.org/issue30403 > > On Sat, Jan 13, 2018 at 8:49 PM, Brett Cannon <[email protected]> wrote: > > Awesome! Thanks for looking into it. > > > > > > On Sat, Jan 13, 2018, 07:49 Petr Viktorin, <[email protected]> wrote: > >> > >> On 01/12/2018 06:52 PM, Brett Cannon wrote: > >> > So obviously implementing get_code() for the extension module loader > >> > would be great. :) So the question becomes how? > >> > >> Marcel took a quick look at it already. It seems it's quite a simple > >> addition, and it makes tests developed for PEP 547 pass. Hopefully we > >> can have a PR early next week :) > >> > >> > >> > On Thu, 11 Jan 2018 at 22:57 Nick Coghlan <[email protected] > >> > <mailto:[email protected]>> wrote: > >> > > >> > (cc'ed a couple of folks that I expect will be interested in this > >> > question, but may not be subscribed to import-sig) > >> > > >> > The current version of PEP 547 (supporting the -m switch for > >> > extension > >> > modules) works by defining a new optional "exec_in_module" API for > >> > loaders to implement, and then updating runpy._run_module_as_main > to > >> > call it. > >> > > >> > However, reviewing Mario Corchero's patches for > >> > https://bugs.python.org/issue9325 (adding "-m" switch support to > >> > assorted modules) has highlighted a potential challenge with that > >> > approach: it turns out the most useful private API in runpy for > >> > emulating the -m switch is "mod_name, mod_spec, code = > >> > runpy._get_module_details(module_name)". > >> > > >> > That means that if we can figure out a way to have > >> > ExtensionFileLoader.get_code() emit a Python code object that > >> > delegates to Py_mod_exec, then we'd be well on our way to > supporting > >> > "python -m <extension module>" without making *any changes to > runpy* > >> > (or the other modules that are gaining "-m" equivalents). > >> > > >> > If we did decide to go down that path, the main way I could see it > >> > working without any new features in the C interface is to > structure > >> > things such that the extension module would still run in its own > >> > namespace, with the interface adaptation code returned from > >> > get_code() > >> > (after compilation) looking something like: > >> > > >> > ns = globals() > >> > if ns is not locals(): > >> > raise RuntimeError("Cannot execute extension module > >> > {<interpolated_name>} with separate local namespace") > >> > module = _imp.create_dynamic(<interpolated_spec_details>) > >> > module.__dict__.update(ns) > >> > _imp.exec_dynamic(module) > >> > ns.update(module.__dict__) > >> > > >> > The biggest advantages of this approach are that it would still > work > >> > for Cython (and other) modules that defined Py_mod_create, and it > >> > would implicitly interoperate (at least to some degree) with > >> > anything > >> > that relied on the "get code and exec it" model of interacting > with > >> > Python modules. > >> > > >> > Alternatively, we could instead push the decision on how to handle > >> > this case down to extension module authors as follows: > >> > > >> > 1. Define a new Py_mod_exec_in_namespace slot that accepts a > target > >> > namespace as its parameter instead of a pre-existing module > >> > 2. Add a new "_imp.exec_dynamic_in_namespace(spec, namespace)" API > >> > 3. When Py_mod_exec_in_namespace is defined, make the adapter code > >> > look something like: > >> > > >> > ns = globals() > >> > if ns is not locals(): > >> > import collections > >> > ns = collections.ChainMap(locals(), ns) > >> > _imp.exec_dynamic_in_namespace(<interpolated_spec_details>, > ns) > >> > > >> > (There are several ways the functionality could be split up > between > >> > the generated code and the _imp module, this is just an example > that > >> > suggests the idea is technically feasible) > >> > > >> > The nice thing about including the new slot in the design is that > it > >> > gives extension modules a way to avoid the overhead of copying > >> > attributes in and out, as would be needed if relying solely on the > >> > PEP > >> > 489 APIs. > >> > > >> > Cheers, > >> > Nick. > >> > > >> > P.S. Given these changes we could technically define > "get_source()" > >> > on > >> > extension modules as well, but that doesn't seem especially > useful. > >> > > >> _______________________________________________ > >> Import-SIG mailing list > >> [email protected] > >> https://mail.python.org/mailman/listinfo/import-sig > > > > > > _______________________________________________ > > Import-SIG mailing list > > [email protected] > > https://mail.python.org/mailman/listinfo/import-sig > > > --f403045e7304f569c70562e7c9f1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><br><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Tue= , 16 Jan 2018 at 05:43 Marcel Plch <<a href=3D"mailto:gmarcel.plch@gmail= .com">[email protected]</a>> wrote:<br></div><blockquote class=3D"g= mail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-l= eft:1ex">I took a look at the get_code() and it works basically just as is = in<br> Nick's mail. Spec and name are accessible through passed globals.<br> Unlike the current proposal for -m switch for extension modules [PR<br> 1761], this approach does not require multiphase initialization. That<br> means that every module can be run with this switch just as it is<br> right now, including the standard library:<br> <br> =C2=A0 =C2=A0 $ python -im math<br> =C2=A0 =C2=A0 >>> print(e)<br> =C2=A0 =C2=A0 2.718281828459045<br> <br> Should I open a bug for this, or reuse [bpo-30403]?</blockquote><div><br></= div><div>If you want, although a new issue is also totally justified.<br></= div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 = 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Or does this need a PE= P?<br></blockquote><div><br></div><div>I don't think so since you'r= e just implementing a pre-existing API and this doesn't require changin= g any pre-existing semantics.</div><div><br></div><div>-Brett<br></div><div= >=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;b= order-left:1px #ccc solid;padding-left:1ex"> <br> You can see the required changes here:<br> <a href=3D"https://github.com/Traceur759/cpython/pull/6/files" rel=3D"noref= errer" target=3D"_blank">https://github.com/Traceur759/cpython/pull/6/files= </a><br> <br> [PR 1761]: <a href=3D"https://github.com/python/cpython/pull/1761" rel=3D"n= oreferrer" target=3D"_blank">https://github.com/python/cpython/pull/1761</a= ><br> [bpo-30403]: <a href=3D"https://bugs.python.org/issue30403" rel=3D"noreferr= er" target=3D"_blank">https://bugs.python.org/issue30403</a><br> <br> On Sat, Jan 13, 2018 at 8:49 PM, Brett Cannon <<a href=3D"mailto:brett@p= ython.org" target=3D"_blank">[email protected]</a>> wrote:<br> > Awesome! Thanks for looking into it.<br> ><br> ><br> > On Sat, Jan 13, 2018, 07:49 Petr Viktorin, <<a href=3D"mailto:encuk= [email protected]" target=3D"_blank">[email protected]</a>> wrote:<br> >><br> >> On 01/12/2018 06:52 PM, Brett Cannon wrote:<br> >> > So obviously implementing get_code() for the extension module= loader<br> >> > would be great. :) So the question becomes how?<br> >><br> >> Marcel took a quick look at it already. It seems it's quite a = simple<br> >> addition, and it makes tests developed for PEP 547 pass. Hopefully= we<br> >> can have a PR early next week :)<br> >><br> >><br> >> > On Thu, 11 Jan 2018 at 22:57 Nick Coghlan <<a href=3D"mail= to:[email protected]" target=3D"_blank">[email protected]</a><br> >> > <mailto:<a href=3D"mailto:[email protected]" target=3D"_b= lank">[email protected]</a>>> wrote:<br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A0(cc'ed a couple of folks that I expect= will be interested in this<br> >> >=C2=A0 =C2=A0 =C2=A0question, but may not be subscribed to imp= ort-sig)<br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A0The current version of PEP 547 (supporting= the -m switch for<br> >> > extension<br> >> >=C2=A0 =C2=A0 =C2=A0modules) works by defining a new optional = "exec_in_module" API for<br> >> >=C2=A0 =C2=A0 =C2=A0loaders to implement, and then updating ru= npy._run_module_as_main to<br> >> >=C2=A0 =C2=A0 =C2=A0call it.<br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A0However, reviewing Mario Corchero's pa= tches for<br> >> >=C2=A0 =C2=A0 =C2=A0<a href=3D"https://bugs.python.org/issue93= 25" rel=3D"noreferrer" target=3D"_blank">https://bugs.python.org/issue9325<= /a> (adding "-m" switch support to<br> >> >=C2=A0 =C2=A0 =C2=A0assorted modules) has highlighted a potent= ial challenge with that<br> >> >=C2=A0 =C2=A0 =C2=A0approach: it turns out the most useful pri= vate API in runpy for<br> >> >=C2=A0 =C2=A0 =C2=A0emulating the -m switch is "mod_name,= mod_spec, code =3D<br> >> >=C2=A0 =C2=A0 =C2=A0runpy._get_module_details(module_name)&quo= t;.<br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A0That means that if we can figure out a way= to have<br> >> >=C2=A0 =C2=A0 =C2=A0ExtensionFileLoader.get_code() emit a Pyth= on code object that<br> >> >=C2=A0 =C2=A0 =C2=A0delegates to Py_mod_exec, then we'd be= well on our way to supporting<br> >> >=C2=A0 =C2=A0 =C2=A0"python -m <extension module>&q= uot; without making *any changes to runpy*<br> >> >=C2=A0 =C2=A0 =C2=A0(or the other modules that are gaining &qu= ot;-m" equivalents).<br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A0If we did decide to go down that path, the= main way I could see it<br> >> >=C2=A0 =C2=A0 =C2=A0working without any new features in the C = interface is to structure<br> >> >=C2=A0 =C2=A0 =C2=A0things such that the extension module woul= d still run in its own<br> >> >=C2=A0 =C2=A0 =C2=A0namespace, with the interface adaptation c= ode returned from<br> >> > get_code()<br> >> >=C2=A0 =C2=A0 =C2=A0(after compilation) looking something like= :<br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ns =3D globals()<br> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ns is not locals():<br> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 raise Runtime= Error("Cannot execute extension module<br> >> >=C2=A0 =C2=A0 =C2=A0{<interpolated_name>} with separate = local namespace")<br> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 module =3D _imp.create_dyna= mic(<interpolated_spec_details>)<br> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 module.__dict__.update(ns)<= br> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 _imp.exec_dynamic(module)<b= r> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ns.update(module.__dict__)<= br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A0The biggest advantages of this approach ar= e that it would still work<br> >> >=C2=A0 =C2=A0 =C2=A0for Cython (and other) modules that define= d Py_mod_create, and it<br> >> >=C2=A0 =C2=A0 =C2=A0would implicitly interoperate (at least to= some degree) with<br> >> > anything<br> >> >=C2=A0 =C2=A0 =C2=A0that relied on the "get code and exec= it" model of interacting with<br> >> >=C2=A0 =C2=A0 =C2=A0Python modules.<br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A0Alternatively, we could instead push the d= ecision on how to handle<br> >> >=C2=A0 =C2=A0 =C2=A0this case down to extension module authors= as follows:<br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A01. Define a new Py_mod_exec_in_namespace s= lot that accepts a target<br> >> >=C2=A0 =C2=A0 =C2=A0namespace as its parameter instead of a pr= e-existing module<br> >> >=C2=A0 =C2=A0 =C2=A02. Add a new "_imp.exec_dynamic_in_na= mespace(spec, namespace)" API<br> >> >=C2=A0 =C2=A0 =C2=A03. When Py_mod_exec_in_namespace is define= d, make the adapter code<br> >> >=C2=A0 =C2=A0 =C2=A0look something like:<br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ns =3D globals()<br> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ns is not locals():<br> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 import collec= tions<br> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ns =3D collec= tions.ChainMap(locals(), ns)<br> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 _imp.exec_dynamic_in_namesp= ace(<interpolated_spec_details>, ns)<br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A0(There are several ways the functionality = could be split up between<br> >> >=C2=A0 =C2=A0 =C2=A0the generated code and the _imp module, th= is is just an example that<br> >> >=C2=A0 =C2=A0 =C2=A0suggests the idea is technically feasible)= <br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A0The nice thing about including the new slo= t in the design is that it<br> >> >=C2=A0 =C2=A0 =C2=A0gives extension modules a way to avoid the= overhead of copying<br> >> >=C2=A0 =C2=A0 =C2=A0attributes in and out, as would be needed = if relying solely on the<br> >> > PEP<br> >> >=C2=A0 =C2=A0 =C2=A0489 APIs.<br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A0Cheers,<br> >> >=C2=A0 =C2=A0 =C2=A0Nick.<br> >> ><br> >> >=C2=A0 =C2=A0 =C2=A0P.S. Given these changes we could technica= lly define "get_source()"<br> >> > on<br> >> >=C2=A0 =C2=A0 =C2=A0extension modules as well, but that doesn&= #39;t seem especially useful.<br> >> ><br> >> _______________________________________________<br> >> Import-SIG mailing list<br> >> <a href=3D"mailto:[email protected]" target=3D"_blank">Import-= [email protected]</a><br> >> <a href=3D"https://mail.python.org/mailman/listinfo/import-sig" re= l=3D"noreferrer" target=3D"_blank">https://mail.python.org/mailman/listinfo= /import-sig</a><br> ><br> ><br> > _______________________________________________<br> > Import-SIG mailing list<br> > <a href=3D"mailto:[email protected]" target=3D"_blank">Import-SIG@= python.org</a><br> > <a href=3D"https://mail.python.org/mailman/listinfo/import-sig" rel=3D= "noreferrer" target=3D"_blank">https://mail.python.org/mailman/listinfo/imp= ort-sig</a><br> ><br> </blockquote></div></div> --f403045e7304f569c70562e7c9f1-- --===============1600739487883680985== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Import-SIG mailing list [email protected] https://mail.python.org/mailman/listinfo/import-sig --===============1600739487883680985==--