Re: PEP 489: Redesigning extension module loading

Petr Viktorin schrieb am 19.03.2015 um 14:37:
> On 03/19/2015 11:31 AM, Stefan Behnel wrote:
>> thanks for working on this. I added my comments inline.
> 
> Thanks for your comments, they're a nice reality check.

Sorry if it was a bit too much. I didn't mean to shoot it down or so. I
think we're on the right track, and the PEP will allow us to get a better
idea of where we should be heading.


> I'm feeling a bit like I and Nick misunderstood Cython requirements
> somewhat, and concentrated on unimportant points (loading into pre-created
> modules)

You mean the split between Create and Exec? I think that's a very good and
simple design. It gives the extension module full control over the module
instance and its implementation (if it wants to), while leaving the core
runtime full control over the basic setup of the module object's common API
(__file__, __name__, etc.).

Simple extension modules should be able to get away without implementing
Create, so separating the two steps sounds better than requiring the module
instantiation on user side and providing a callback into CPython to
initialise it.


> while ignoring important ones (fast access to module state).

Yes, that *is* important. And I believe that a custom module (sub)type is a
good way to achieve that, at least for Cython. For manually written
modules, it might be easier to call PyModule_GetState().


> You also pointed out interesting things we didn't think about too much
> (non-ASCII names, multi-module extensions).

I just mentioned what came to my mind. We should still try to keep the PEP
focussed on the problem at hand, but having some idea of what else might
lie ahead can help with design decisions.


> The "inittab" idea made me think of this:
> 
> An extension could export an array of PyModuleDef, which has all the needed
> data for module creation and initialization:

I remember discussing this on python-dev, it was one of the ideas in the
original thread that lead to the Create-Exec proto-pep:

http://thread.gmane.org/gmane.comp.python.devel/135764/focus=140986

I think the main counter argument at the time was that there should be a
way to control the module object instantiation. :)


> - m_name - for the "requested" name for the module (not necessarily what
> it'll be loaded as), for identifying modules in multi-module extensions
> - m_size - for requesting per-module C state)
> - m_reload (currently unused) would be the exec function (called for
> initialization and reload)
> 
> This would rule out completely custom module objects, but are those needed
> anyway? A module can always replace itself in sys.modules if it needs extra
> magic. Getting rid of Create entirely supports a lot of the other goals
> (running user code in Create, pushing for subinterpreter support). And
> things like module properties or callable modules are not possible in
> source modules as well; perhaps those should be solved at a higher level.
> 
> With this, you couldn't load extensions into arbitrary objects. But it
> would be possible to load into pre-created modules, as long as they were
> pre-created with the correct ModuleDef. It would probably be somewhat more
> difficult to make runpy (or custom loading libraries) ­work with these
> extension modules, but it should be possible.
> 
> Implementation-wise, having m_reload filled in from the start would help:
> the PEP calls for looking up two entrypoints, and the lookup is relatively
> expensive (judging by the amount of caching in current code).
> 
> It would also help with non-ASCII names, since the name is a string rather
> than a C identifier. Entrypoint and file names would need some design to
> make everything work. But before I go thinking about that: Does this seem
> like a better direction than Create/Exec?

It's still an alternative, I think. Nick objected to extending PyModuleDef
because it's (obviously) part of the stable ABI. But we could instead
export a new struct that *contains* a PyModuleDef, with additional callback
functions like "new(spec)", as known from other extension types (tp_new).
That would give us the Create() functionality (if set to non-NULL), or
allow CPython to instantiate a regular module object (if set to NULL).

With a magic version field at the top of the struct, this would also make
it easy to extend in the future if we ever need more metadata or callbacks
that we can't foresee now. Updating the version magic and appending to the
struct is so much easier than writing a new PEP and redesigning the entire
extension module init process again...

So, yes, exporting a struct with module metadata and callbacks sounds like
a very generic and straight forward interface to me.

Stefan


_______________________________________________
Import-SIG mailing list
[email protected]
https://mail.python.org/mailman/listinfo/import-sig