Re: PEP 547: Could we implement a usable "get_code()" for extension modules?

I took a look at the get_code() and it works basically just as is in
Nick's mail. Spec and name are accessible through passed globals.
Unlike the current proposal for -m switch for extension modules [PR
1761], this approach does not require multiphase initialization. That
means that every module can be run with this switch just as it is
right now, including the standard library:

    $ python -im math
    >>> print(e)
    2.718281828459045

Should I open a bug for this, or reuse [bpo-30403]? Or does this need a PEP?

You can see the required changes here:
https://github.com/Traceur759/cpython/pull/6/files

[PR 1761]: https://github.com/python/cpython/pull/1761
[bpo-30403]: https://bugs.python.org/issue30403

On Sat, Jan 13, 2018 at 8:49 PM, Brett Cannon <[email protected]> wrote:
> Awesome! Thanks for looking into it.
>
>
> On Sat, Jan 13, 2018, 07:49 Petr Viktorin, <[email protected]> wrote:
>>
>> On 01/12/2018 06:52 PM, Brett Cannon wrote:
>> > So obviously implementing get_code() for the extension module loader
>> > would be great. :) So the question becomes how?
>>
>> Marcel took a quick look at it already. It seems it's quite a simple
>> addition, and it makes tests developed for PEP 547 pass. Hopefully we
>> can have a PR early next week :)
>>
>>
>> > On Thu, 11 Jan 2018 at 22:57 Nick Coghlan <[email protected]
>> > <mailto:[email protected]>> wrote:
>> >
>> >     (cc'ed a couple of folks that I expect will be interested in this
>> >     question, but may not be subscribed to import-sig)
>> >
>> >     The current version of PEP 547 (supporting the -m switch for
>> > extension
>> >     modules) works by defining a new optional "exec_in_module" API for
>> >     loaders to implement, and then updating runpy._run_module_as_main to
>> >     call it.
>> >
>> >     However, reviewing Mario Corchero's patches for
>> >     https://bugs.python.org/issue9325 (adding "-m" switch support to
>> >     assorted modules) has highlighted a potential challenge with that
>> >     approach: it turns out the most useful private API in runpy for
>> >     emulating the -m switch is "mod_name, mod_spec, code =
>> >     runpy._get_module_details(module_name)".
>> >
>> >     That means that if we can figure out a way to have
>> >     ExtensionFileLoader.get_code() emit a Python code object that
>> >     delegates to Py_mod_exec, then we'd be well on our way to supporting
>> >     "python -m <extension module>" without making *any changes to runpy*
>> >     (or the other modules that are gaining "-m" equivalents).
>> >
>> >     If we did decide to go down that path, the main way I could see it
>> >     working without any new features in the C interface is to structure
>> >     things such that the extension module would still run in its own
>> >     namespace, with the interface adaptation code returned from
>> > get_code()
>> >     (after compilation) looking something like:
>> >
>> >          ns = globals()
>> >          if ns is not locals():
>> >              raise RuntimeError("Cannot execute extension module
>> >     {<interpolated_name>} with separate local namespace")
>> >          module = _imp.create_dynamic(<interpolated_spec_details>)
>> >          module.__dict__.update(ns)
>> >          _imp.exec_dynamic(module)
>> >          ns.update(module.__dict__)
>> >
>> >     The biggest advantages of this approach are that it would still work
>> >     for Cython (and other) modules that defined Py_mod_create, and it
>> >     would implicitly interoperate (at least to some degree) with
>> > anything
>> >     that relied on the "get code and exec it" model of interacting with
>> >     Python modules.
>> >
>> >     Alternatively, we could instead push the decision on how to handle
>> >     this case down to extension module authors as follows:
>> >
>> >     1. Define a new Py_mod_exec_in_namespace slot that accepts a target
>> >     namespace as its parameter instead of a pre-existing module
>> >     2. Add a new "_imp.exec_dynamic_in_namespace(spec, namespace)" API
>> >     3. When Py_mod_exec_in_namespace is defined, make the adapter code
>> >     look something like:
>> >
>> >          ns = globals()
>> >          if ns is not locals():
>> >              import collections
>> >              ns = collections.ChainMap(locals(), ns)
>> >          _imp.exec_dynamic_in_namespace(<interpolated_spec_details>, ns)
>> >
>> >     (There are several ways the functionality could be split up between
>> >     the generated code and the _imp module, this is just an example that
>> >     suggests the idea is technically feasible)
>> >
>> >     The nice thing about including the new slot in the design is that it
>> >     gives extension modules a way to avoid the overhead of copying
>> >     attributes in and out, as would be needed if relying solely on the
>> > PEP
>> >     489 APIs.
>> >
>> >     Cheers,
>> >     Nick.
>> >
>> >     P.S. Given these changes we could technically define "get_source()"
>> > on
>> >     extension modules as well, but that doesn't seem especially useful.
>> >
>> _______________________________________________
>> Import-SIG mailing list
>> [email protected]
>> https://mail.python.org/mailman/listinfo/import-sig
>
>
> _______________________________________________
> Import-SIG mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/import-sig
>