Re: PEP 547: Could we implement a usable "get_code()" for extension modules?

--===============8474104652976345864==
Content-Type: multipart/alternative; boundary="001a11407a12cfd095056297ee2e"

--001a11407a12cfd095056297ee2e
Content-Type: text/plain; charset="UTF-8"

So obviously implementing get_code() for the extension module loader would
be great. :) So the question becomes how?

On Thu, 11 Jan 2018 at 22:57 Nick Coghlan <[email protected]> wrote:

> (cc'ed a couple of folks that I expect will be interested in this
> question, but may not be subscribed to import-sig)
>
> The current version of PEP 547 (supporting the -m switch for extension
> modules) works by defining a new optional "exec_in_module" API for
> loaders to implement, and then updating runpy._run_module_as_main to
> call it.
>
> However, reviewing Mario Corchero's patches for
> https://bugs.python.org/issue9325 (adding "-m" switch support to
> assorted modules) has highlighted a potential challenge with that
> approach: it turns out the most useful private API in runpy for
> emulating the -m switch is "mod_name, mod_spec, code =
> runpy._get_module_details(module_name)".
>
> That means that if we can figure out a way to have
> ExtensionFileLoader.get_code() emit a Python code object that
> delegates to Py_mod_exec, then we'd be well on our way to supporting
> "python -m <extension module>" without making *any changes to runpy*
> (or the other modules that are gaining "-m" equivalents).
>
> If we did decide to go down that path, the main way I could see it
> working without any new features in the C interface is to structure
> things such that the extension module would still run in its own
> namespace, with the interface adaptation code returned from get_code()
> (after compilation) looking something like:
>
>     ns = globals()
>     if ns is not locals():
>         raise RuntimeError("Cannot execute extension module
> {<interpolated_name>} with separate local namespace")
>     module = _imp.create_dynamic(<interpolated_spec_details>)
>     module.__dict__.update(ns)
>     _imp.exec_dynamic(module)
>     ns.update(module.__dict__)
>
> The biggest advantages of this approach are that it would still work
> for Cython (and other) modules that defined Py_mod_create, and it
> would implicitly interoperate (at least to some degree) with anything
> that relied on the "get code and exec it" model of interacting with
> Python modules.
>
> Alternatively, we could instead push the decision on how to handle
> this case down to extension module authors as follows:
>
> 1. Define a new Py_mod_exec_in_namespace slot that accepts a target
> namespace as its parameter instead of a pre-existing module
> 2. Add a new "_imp.exec_dynamic_in_namespace(spec, namespace)" API
> 3. When Py_mod_exec_in_namespace is defined, make the adapter code
> look something like:
>
>     ns = globals()
>     if ns is not locals():
>         import collections
>         ns = collections.ChainMap(locals(), ns)
>     _imp.exec_dynamic_in_namespace(<interpolated_spec_details>, ns)
>
> (There are several ways the functionality could be split up between
> the generated code and the _imp module, this is just an example that
> suggests the idea is technically feasible)
>
> The nice thing about including the new slot in the design is that it
> gives extension modules a way to avoid the overhead of copying
> attributes in and out, as would be needed if relying solely on the PEP
> 489 APIs.
>
> Cheers,
> Nick.
>
> P.S. Given these changes we could technically define "get_source()" on
> extension modules as well, but that doesn't seem especially useful.
>
> --
> Nick Coghlan   |   [email protected]   |   Brisbane, Australia
> _______________________________________________
> Import-SIG mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/import-sig
>

--001a11407a12cfd095056297ee2e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">So obviously implementing get_code() for the extension mod=
ule loader would be great. :) So the question becomes how?<br></div><br><di=
v class=3D"gmail_quote"><div dir=3D"ltr">On Thu, 11 Jan 2018 at 22:57 Nick =
Coghlan &lt;<a href=3D"mailto:[email protected]">[email protected]</a>&gt=
; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .=
8ex;border-left:1px #ccc solid;padding-left:1ex">(cc&#39;ed a couple of fol=
ks that I expect will be interested in this<br>
question, but may not be subscribed to import-sig)<br>
<br>
The current version of PEP 547 (supporting the -m switch for extension<br>
modules) works by defining a new optional &quot;exec_in_module&quot; API fo=
r<br>
loaders to implement, and then updating runpy._run_module_as_main to<br>
call it.<br>
<br>
However, reviewing Mario Corchero&#39;s patches for<br>
<a href=3D"https://bugs.python.org/issue9325" rel=3D"noreferrer" target=3D"=
_blank">https://bugs.python.org/issue9325</a> (adding &quot;-m&quot; switch=
 support to<br>
assorted modules) has highlighted a potential challenge with that<br>
approach: it turns out the most useful private API in runpy for<br>
emulating the -m switch is &quot;mod_name, mod_spec, code =3D<br>
runpy._get_module_details(module_name)&quot;.<br>
<br>
That means that if we can figure out a way to have<br>
ExtensionFileLoader.get_code() emit a Python code object that<br>
delegates to Py_mod_exec, then we&#39;d be well on our way to supporting<br=
>
&quot;python -m &lt;extension module&gt;&quot; without making *any changes =
to runpy*<br>
(or the other modules that are gaining &quot;-m&quot; equivalents).<br>
<br>
If we did decide to go down that path, the main way I could see it<br>
working without any new features in the C interface is to structure<br>
things such that the extension module would still run in its own<br>
namespace, with the interface adaptation code returned from get_code()<br>
(after compilation) looking something like:<br>
<br>
=C2=A0 =C2=A0 ns =3D globals()<br>
=C2=A0 =C2=A0 if ns is not locals():<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 raise RuntimeError(&quot;Cannot execute extensi=
on module<br>
{&lt;interpolated_name&gt;} with separate local namespace&quot;)<br>
=C2=A0 =C2=A0 module =3D _imp.create_dynamic(&lt;interpolated_spec_details&=
gt;)<br>
=C2=A0 =C2=A0 module.__dict__.update(ns)<br>
=C2=A0 =C2=A0 _imp.exec_dynamic(module)<br>
=C2=A0 =C2=A0 ns.update(module.__dict__)<br>
<br>
The biggest advantages of this approach are that it would still work<br>
for Cython (and other) modules that defined Py_mod_create, and it<br>
would implicitly interoperate (at least to some degree) with anything<br>
that relied on the &quot;get code and exec it&quot; model of interacting wi=
th<br>
Python modules.<br>
<br>
Alternatively, we could instead push the decision on how to handle<br>
this case down to extension module authors as follows:<br>
<br>
1. Define a new Py_mod_exec_in_namespace slot that accepts a target<br>
namespace as its parameter instead of a pre-existing module<br>
2. Add a new &quot;_imp.exec_dynamic_in_namespace(spec, namespace)&quot; AP=
I<br>
3. When Py_mod_exec_in_namespace is defined, make the adapter code<br>
look something like:<br>
<br>
=C2=A0 =C2=A0 ns =3D globals()<br>
=C2=A0 =C2=A0 if ns is not locals():<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 import collections<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ns =3D collections.ChainMap(locals(), ns)<br>
=C2=A0 =C2=A0 _imp.exec_dynamic_in_namespace(&lt;interpolated_spec_details&=
gt;, ns)<br>
<br>
(There are several ways the functionality could be split up between<br>
the generated code and the _imp module, this is just an example that<br>
suggests the idea is technically feasible)<br>
<br>
The nice thing about including the new slot in the design is that it<br>
gives extension modules a way to avoid the overhead of copying<br>
attributes in and out, as would be needed if relying solely on the PEP<br>
489 APIs.<br>
<br>
Cheers,<br>
Nick.<br>
<br>
P.S. Given these changes we could technically define &quot;get_source()&quo=
t; on<br>
extension modules as well, but that doesn&#39;t seem especially useful.<br>
<br>
--<br>
Nick Coghlan=C2=A0 =C2=A0|=C2=A0 =C2=A0<a href=3D"mailto:[email protected]=
" target=3D"_blank">[email protected]</a>=C2=A0 =C2=A0|=C2=A0 =C2=A0Brisba=
ne, Australia<br>
_______________________________________________<br>
Import-SIG mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">Import-SIG@pytho=
n.org</a><br>
<a href=3D"https://mail.python.org/mailman/listinfo/import-sig" rel=3D"nore=
ferrer" target=3D"_blank">https://mail.python.org/mailman/listinfo/import-s=
ig</a><br>
</blockquote></div>

--001a11407a12cfd095056297ee2e--

--===============8474104652976345864==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Import-SIG mailing list
[email protected]
https://mail.python.org/mailman/listinfo/import-sig

--===============8474104652976345864==--