Re: PEP 451 (ModuleSpec) round 3

On Sat, Aug 31, 2013 at 7:53 PM, Nick Coghlan <[email protected]> wrote:

> Yep, that's exactly what I had in mind, although reload would also have an
> extra check to ensure the module returned was the same as the one passed in
> (that way, in-place reloading support for custom loaders that define
> create_module would always be opt-in rather than opt-out).
>
> Talking to Stefan about making this work on the extension module API side
> has confirmed my belief that this is the way to go, since it also deals
> nicely with placing custom objects in sys.modules.
>
> The one downside is that it means preconditions will be checked twice in
> the reload case (once in prepare, once in exec), but I can live with that
> for the likely reliability gains in the reloading API.
>
> If it works as well as I hope, I may finally be comfortable with proposing
> "imp.reload_fresh" for 3.5 :)
>
> Cheers,
> Nick.
>
I haven't had a chance to respond to a few comments in detail yet, nor have
I been able to read through the extension module API threads, but what I
have seen has gotten me thinking about what exactly matters here regarding
preparing and executing modules.

* There are two kinds of module state: internal (not exposed in Python) and
external (module.__dict__).
* External state is associated with a module object, but internal state is
not--it may be associated with a module name, a location (locatable
resource), or something else.

* Internal state (if it exists) is not necessarily created at execution
(load/reload) time and may be shared between modules.
* The internal module state may or may not be managed by the interpreter
(regardless of Python implementation).

* External module state is established at import execution time (load or
reload).
* Loading puts the external state into a new module and reloading into an
existing one (likely overwriting at least some contents).
* During module execution (during load/reload), external module state is
copied from internal state, dynamically generated (e.g. .py files), or a
mix of both.
* Dynamic external state generation is only allowed once for some modules.
* Dynamic generation is not necessarily (but sometimes is) an idempotent
operation.
* Dynamic generation may be associated with a locatable resource.
* Non-locatable sources are not necessarily unchanging.

* Loaders are in charge of managing module execution and may be involved
with managing internal state.

The life-cycle of module state, both internal and external, is pretty
congruent with objects in Python:

1. create
2. init
3. modify
4. destroy

Modules have some special cases that fit in there:

2a. dynamically generate
2b. populate from another namespace
3a/2b. reset

* In the generation case, population may happen simultaneously (e.g. .py
files).
* Resetting a module's state may not be the same operation as init.

Am I missing anything in all of this?

---

Some questions that come to mind:

* Should loaders cover all the permutations of module state and it
lifecycle?  Our proposed APIs are moving in that direction.  Are they
enough?
* When does internal state get generated and how is it managed?  Should
loaders be the official liaison for the import system?  Python
implementation extension module APIs cover this somewhat (particularly for
CPython).
* Should the language provide a non-implementation-specific API for
associating internal APIs with modules?  (PEP 3121-ish)
* Does reset deserve its own explicit API?
* How do you keep init from happening more than once?  IOW, what happens
when ModuleSpec.create() is called more than once?

I have more questions but they mostly line up with the details above.

Anyway, there are the things I am mulling over.  For PEP 451 I'm not going
to try to accomplish module API perfection, but I do want to make sure
we're on the right track with a more explicit perspective.  The confusion
with create_module() and exec_module() made it clear to me that the picture
should be more clear before we start

-eric


p.s. Tangential shot in the dark idea: Optionally cache deep copy of
external state for generate-only-once case?

_______________________________________________
Import-SIG mailing list
[email protected]
http://mail.python.org/mailman/listinfo/import-sig