Re: Pre-PEP: Module State Access from C Extension Methods

--===============8964196349805118765==
Content-Type: multipart/alternative; boundary=001a114dd9500eef930534a57702

--001a114dd9500eef930534a57702
Content-Type: text/plain; charset=UTF-8

A quick read-through didn't raise any red flags from me (bit I don't write
extension modules very often so I'm not the target audience).

On Fri, Jun 3, 2016, 14:16 Petr Viktorin <[email protected]> wrote:

> Hello fro the PyCon sprints!
>
> Here is the first approximation for a PEP that introduces efficient
> access to the module state from methods of extension types. This is the
> biggest unresolved issue of PEP 489 (multi-phase init).
> There are still a few XXXs to be fleshed out, but the base is pretty
> solid. Please leave your comments!
>
> I also posted the pre-PEP at:
>
> https://github.com/encukou/peps/blob/3fdf3b1fad0220c7fe39044dfefc6d76759e0d1a/pep-9999.txt
>
> The beginnings of an implementation live at:
> https://github.com/encukou/cpython/tree/module-state-access
>
> Thanks to Nick and Eric for helping out!
>
> ----
>
> PEP: XXX
> Title: Module State Access from C Extension Methods
> Version: $Revision$
> Last-Modified: $Date$
> Author: Petr Viktorin <[email protected]>,
>         Nick Coghlan <[email protected]>,
>         Eric Snow <[email protected]>
> Discussions-To: [email protected]
> Status: Active
> Type: Process
> Content-Type: text/x-rst
> Created: 02-Jun-2016
> Python-Version: 3.6
> Post-History:
>
>
> Abstract
> ========
>
> This PEP proposes to add a way for CPython extension methods to access
> context such as
> the state of the modules they are defined in.
>
> This will allow extension methods to use direct pointer dereferences
> rather than PyState_FindModule for looking up module state, reducing or
> eliminating the
> performance cost of using module-scoped state over process global state.
>
> This fixes one of the remaining roadblocks for adoption of PEP 3121
> (Extension
> module initialization and finalization) and PEP 489
> (Multi-phase extension module initialization).
>
> While this PEP takes an additional step towards fully solving the
> problems that PEP 3121 and PEP 489 started
> tackling, it does not attempt to resolve *all* remaining concerns. In
> particular, accessing the module state from slot methods (``nb_add``,
> etc) remains slower than accessing that state from other extension methods.
>
>
> Rationale
> =========
>
> PEP 489 introduced a new way to initialize extension modules, which brings
> several advantages to extensions that implement it:
>
>     * The extension modules behave more like their Python counterparts.
>     * The extension modules can easily support loading into pre-existing
>       module objects, which paves the way for extension module support for
>       ``runpy`` or for systems that enable extension module reloading.
>     * Loading multiple modules from the same extension is possible, which
>       makes testing module isolation (a key feature for proper
> sub-interpreter
>       support) possible from a single interpreter.
>
> The biggest hurdle for adoption of PEP 489 is allowing access to module
> state
> from methods of extension types.
> Currently, the way to access this state from extension methods is by
> looking up the module via
> ``PyState_FindModule`` (in contrast to module level functions in
> extension modules, which
> receive a module reference as an argument).
> However, ``PyState_FindModule`` queries the thread-local state, making
> it relatively
> costly compared to C level process global access and consequently
> deterring module authors from using it.
>
> Also, ``PyState_FindModule`` relies on the assumption that in each
> subinterpreter, there is at most one module corresponding to
> a given ``PyModuleDef``.  This does not align well with Python's import
> machinery.  Since PEP 489 aimed to fix that,  the assumption does
> not hold for modules that use multi-phase initialization, so
> ``PyState_FindModule`` is unavailable for these modules.
>
> A faster, safer way of accessing module-level state from extension methods
> is needed.
>
>
> Background
> ===========
>
> The implementation of a Python method may need access to one or more of
> the following pieces of information:
>
>    * The instance it is called on (``self``)
>    * The underlying function
>    * The class the method was defined in
>    * The corresponding module
>    * The module state
>
> In Python code, the Python-level equivalents may be retrieved as::
>
>     import sys
>
>     class Foo:
>         def meth(self):
>             instance = self
>             module_globals = globals()
>             module_object = sys.modules[__name__]
>             underlying_function = Foo.meth
>             defining_class = Foo
>
> .. note::
>
>     The defining class is not ``type(self)``, since ``type(self)`` might
>     be a subclass of ``Foo``.
>
> Implicitly, the last three of those rely on name-based lookup via the
> function's ``__globals__`` attribute:
> either the ``Foo`` attribute to access the defining class and Python
> function object, or ``__name__`` to find the module object in
> ``sys.modules``.
> In Python code, this is feasible, as ``__globals__`` is set
> appropriately when the function definition is executed, and
> even if the namespace has been manipulated to return a different object,
> at worst an exception will be raised.
>
> By contrast, extension methods are typically implemented as normal C
> functions. This means that they only have access to their arguments, and
> any C level thread local and process global state. Traditionally, many
> extension modules have stored
> their shared state in C level process globals, causing problems when:
>
>     * running multiple initialize/finalize cycles in the same process
>     * reloading modules (e.g. to test conditional imports)
>     * loading extension modules in subinterpreters
>
> PEP 3121 attempted to resolve this by offering the
> ``PyState_FindModule`` API, but this still had significant problems when
> it comes to extension methods (rather than module level functions):
>
>     * it is markedly slower than directly accessing C level process
> global state
>     * there is still some inherent reliance on process global state that
> means it still doesn't reliably handle module reloading
>
> It's also the case that when looking up a C-level struct such as module
> state, supplying
> an unexpected object layout can crash the interpreter, so it's
> significantly more important to ensure that extension
> methods receive the kind of object they expect.
>
> Proposal
> ========
>
> Currently, a bound extension method (``PyCFunction`` or
> ``PyCFunctionWithKeywords``) receives only
> ``self``, and (if applicable) the supplied positional and keyword
> arguments.
>
> While module-level extension functions already receive access to the
> defining module object via their
> ``self`` argument, methods of extension types don't have that luxury:
> they receive the bound instance
> via ``self``, and hence have no direct access to the defining class or
> the module level state.
>
> The additional module level context described above can be made
> available with two changes.
> Both additions are optional; extension authors need to opt in to start
> using them:
>
>     * Add a pointer to the module to heap type objects.
>
>     * Pass the defining class to the underlying C function.
>
>       The defining class is readily available at the time built-in
>       method objects (``PyCFunctionObject``) are created, so it can be
> stored
>       in a new struct that extends ``PyCFunctionObject``.
>
> The module state can then be retrieved from the module object via
> ``PyModule_GetState``.
>
> Note that this proposal implies that any type whose method needs to access
> module-global state must be a heap type dynamically created during
> extension
> module initialisation, rather than a static type predefined when the
> extension
> module is compiled.
>
> This is necessary to support loading multiple module objects from a single
> extension: a static type, as a C-level global, has no information about
> which module it belongs to.
>
>
> Slot methods
> ------------
>
> The above changes don't cover slot methods, such as ``tp_iter`` or
> ``nb_add``.
>
> The problem with slot methods is that their C API is fixed, so we can't
> simply add a new argument to pass in the defining class.
> Two possible solutions have been proposed to this problem:
>
>     * Look up the class through walking the MRO.
>       This is potentially expensive, but will be useful if performance
> is not
>       a problem (such as when raising a module-level exception).
>     * Storing a pointer to the defining class of each slot in a separate
> table,
>       ``__typeslots__`` [#typeslots-mail]_.  This is technically
> feasible and fast,
>       but quite invasive.
>
> Due to the invasiveness of the latter approach, this PEP proposes adding
> a MRO walking helper for use in slot method implementations, deferring
> the more complex alternative as a potential future optimisation.
>
>
> Specification
> =============
>
> Adding module references to heap types
> --------------------------------------
>
> The ``PyHeapTypeObject`` struct will get a new member, ``PyObject
> *ht_module``,
> that can store a pointer to the module object for which the type was
> defined.
> It will be ``NULL`` by default, and should not be modified after the type
> object is created.
>
> A new flag, ``Py_TPFLAGS_HAVE_MODULE``, will be set on any type object
> where
> the ``ht_module`` member is present and non-NULL.
>
> A new factory method will be added for creating modules::
>
>     PyObject* PyType_FromModuleAndSpec(PyObject *module,
>                                        PyType_Spec *spec,
>                                        PyObject *bases)
>
> This acts the same as ``PyType_FromSpecWithBases``, and additionally sets
> ``ht_module`` to the provided module object.
>
> Additionally, an accessor, ``PyObject * PyType_GetModule(PyTypeObject *)``
> will be provided.
> It will return the ``ht_module`` if a heap type with
> Py_TPFLAGS_HAVE_MODULE is passed in,
> otherwise it will set a SystemError and return NULL.
>
> Usually, creating a class with ``ht_module`` set will create a reference
> cycle involving the class and the module.
> This is not a problem, as tearing down modules is not a
> performance-sensitive
> operation.
> Module-level functions typically also create reference cycles.
>
>
> Passing the defining class to extension methods
> -----------------------------------------------
>
> A new style of C-level functions will be added to the current selection of
> ``PyCFunction`` and ``PyCFunctionWithKeywords``::
>
>     PyObject *PyCMethod(PyObject *self,
>                         PyTypeObject *defining_class,
>                         PyObject *args, PyObject *kwargs)
>
> A new method object flag, ``METH_METHOD``, will be added to signal that
> the underlying C function is ``PyCMethod``.
>
> To hold the extra information, a new structure extending
> ``PyCFunctionObject``
> will be added::
>
>     typedef struct {
>         PyCFunctionObject func;
>         PyTypeObject *mm_class; /* Passed as 'defining_class' arg to the
> C func */
>     } PyCMethodObject;
>
> Method construction and calling code and will be updated to honor
> ``METH_METHOD``.
>
> Slot methods
> ------------
>
> XXX: Exact API TBD
>
>
> Helpers
> -------
>
> XXX: I'd like to port a bunch of modules to see what helpers would be
> convenient
>
>
> Argument Clinic
> ---------------
>
> XXX [How does this affect Argument Clinic?]
>
>
>
> Summary of API Changes and Additions
> ====================================
>
> XXX, see above for now
>
>
> Backwards Compatibility
> =======================
>
> One new pointer is added to all heap types.
> All other changes are adding new functions and structures.
>
>
> Implementation
> ==============
>
> An initial implementation is available in a Github repository [#gh-repo]_;
> a patchset is at [#gh-patch]_.
>
>
> Possible Future Extensions
> ==========================
>
> Easy creation of types with module references
> ---------------------------------------------
>
> It would be possible to add a PEP 489 execution slot type make
> creating heap types significantly easier than calling
> ``PyType_FromModuleAndSpec``.
> This is left to a future PEP.
>
>
> Optimization
> ------------
>
> CPython optimizes calls to methods that have restricted signatures,
> such as not allowing keyword arguments.
>
> As proposed here, methods defined with the ``METH_METHOD`` flag do not
> support
> these optimizations.
>
>
> Discussion
> ==========
>
> XXX Static exceptions
>
>
> References
> ==========
>
> .. [#typeslots-mail] [Import-SIG] On singleton modules, heap types, and
> subinterpreters
>    (https://mail.python.org/pipermail/import-sig/2015-July/001035.html)
>
> .. [#gh-repo]
>    https://github.com/encukou/cpython/commits/module-state-access
>
> .. [#gh-patch]
>
>
> https://github.com/encukou/cpython/compare/master...encukou:module-state-access.patch
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
>
>
> ..
>    Local Variables:
>    mode: indented-text
>    indent-tabs-mode: nil
>    sentence-end-double-space: t
>    fill-column: 70
>    coding: utf-8
>    End:
>
>
> _______________________________________________
> Import-SIG mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/import-sig
>

--001a114dd9500eef930534a57702
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<p dir=3D"ltr">A quick read-through didn&#39;t raise any red flags from me =
(bit I don&#39;t write extension modules very often so I&#39;m not the targ=
et audience).</p>
<br><div class=3D"gmail_quote"><div dir=3D"ltr">On Fri, Jun 3, 2016, 14:16 =
Petr Viktorin &lt;<a href=3D"mailto:[email protected]">[email protected]</a=
>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0=
 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello fro the PyCon sp=
rints!<br>
<br>
Here is the first approximation for a PEP that introduces efficient<br>
access to the module state from methods of extension types. This is the<br>
biggest unresolved issue of PEP 489 (multi-phase init).<br>
There are still a few XXXs to be fleshed out, but the base is pretty<br>
solid. Please leave your comments!<br>
<br>
I also posted the pre-PEP at:<br>
<a href=3D"https://github.com/encukou/peps/blob/3fdf3b1fad0220c7fe39044dfef=
c6d76759e0d1a/pep-9999.txt" rel=3D"noreferrer" target=3D"_blank">https://gi=
thub.com/encukou/peps/blob/3fdf3b1fad0220c7fe39044dfefc6d76759e0d1a/pep-999=
9.txt</a><br>
<br>
The beginnings of an implementation live at:<br>
<a href=3D"https://github.com/encukou/cpython/tree/module-state-access" rel=
=3D"noreferrer" target=3D"_blank">https://github.com/encukou/cpython/tree/m=
odule-state-access</a><br>
<br>
Thanks to Nick and Eric for helping out!<br>
<br>
----<br>
<br>
PEP: XXX<br>
Title: Module State Access from C Extension Methods<br>
Version: $Revision$<br>
Last-Modified: $Date$<br>
Author: Petr Viktorin &lt;<a href=3D"mailto:[email protected]" target=3D"_b=
lank">[email protected]</a>&gt;,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Nick Coghlan &lt;<a href=3D"mailto:ncoghlan@gma=
il.com" target=3D"_blank">[email protected]</a>&gt;,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Eric Snow &lt;<a href=3D"mailto:ericsnowcurrent=
[email protected]" target=3D"_blank">[email protected]</a>&gt;<br>
Discussions-To: <a href=3D"mailto:[email protected]" target=3D"_blank">=
[email protected]</a><br>
Status: Active<br>
Type: Process<br>
Content-Type: text/x-rst<br>
Created: 02-Jun-2016<br>
Python-Version: 3.6<br>
Post-History:<br>
<br>
<br>
Abstract<br>
=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
This PEP proposes to add a way for CPython extension methods to access<br>
context such as<br>
the state of the modules they are defined in.<br>
<br>
This will allow extension methods to use direct pointer dereferences<br>
rather than PyState_FindModule for looking up module state, reducing or<br>
eliminating the<br>
performance cost of using module-scoped state over process global state.<br=
>
<br>
This fixes one of the remaining roadblocks for adoption of PEP 3121<br>
(Extension<br>
module initialization and finalization) and PEP 489<br>
(Multi-phase extension module initialization).<br>
<br>
While this PEP takes an additional step towards fully solving the<br>
problems that PEP 3121 and PEP 489 started<br>
tackling, it does not attempt to resolve *all* remaining concerns. In<br>
particular, accessing the module state from slot methods (``nb_add``,<br>
etc) remains slower than accessing that state from other extension methods.=
<br>
<br>
<br>
Rationale<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
PEP 489 introduced a new way to initialize extension modules, which brings<=
br>
several advantages to extensions that implement it:<br>
<br>
=C2=A0 =C2=A0 * The extension modules behave more like their Python counter=
parts.<br>
=C2=A0 =C2=A0 * The extension modules can easily support loading into pre-e=
xisting<br>
=C2=A0 =C2=A0 =C2=A0 module objects, which paves the way for extension modu=
le support for<br>
=C2=A0 =C2=A0 =C2=A0 ``runpy`` or for systems that enable extension module =
reloading.<br>
=C2=A0 =C2=A0 * Loading multiple modules from the same extension is possibl=
e, which<br>
=C2=A0 =C2=A0 =C2=A0 makes testing module isolation (a key feature for prop=
er<br>
sub-interpreter<br>
=C2=A0 =C2=A0 =C2=A0 support) possible from a single interpreter.<br>
<br>
The biggest hurdle for adoption of PEP 489 is allowing access to module<br>
state<br>
from methods of extension types.<br>
Currently, the way to access this state from extension methods is by<br>
looking up the module via<br>
``PyState_FindModule`` (in contrast to module level functions in<br>
extension modules, which<br>
receive a module reference as an argument).<br>
However, ``PyState_FindModule`` queries the thread-local state, making<br>
it relatively<br>
costly compared to C level process global access and consequently<br>
deterring module authors from using it.<br>
<br>
Also, ``PyState_FindModule`` relies on the assumption that in each<br>
subinterpreter, there is at most one module corresponding to<br>
a given ``PyModuleDef``.=C2=A0 This does not align well with Python&#39;s i=
mport<br>
machinery.=C2=A0 Since PEP 489 aimed to fix that,=C2=A0 the assumption does=
<br>
not hold for modules that use multi-phase initialization, so<br>
``PyState_FindModule`` is unavailable for these modules.<br>
<br>
A faster, safer way of accessing module-level state from extension methods<=
br>
is needed.<br>
<br>
<br>
Background<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
The implementation of a Python method may need access to one or more of<br>
the following pieces of information:<br>
<br>
=C2=A0 =C2=A0* The instance it is called on (``self``)<br>
=C2=A0 =C2=A0* The underlying function<br>
=C2=A0 =C2=A0* The class the method was defined in<br>
=C2=A0 =C2=A0* The corresponding module<br>
=C2=A0 =C2=A0* The module state<br>
<br>
In Python code, the Python-level equivalents may be retrieved as::<br>
<br>
=C2=A0 =C2=A0 import sys<br>
<br>
=C2=A0 =C2=A0 class Foo:<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 def meth(self):<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 instance =3D self<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 module_globals =3D globals()<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 module_object =3D sys.modules[__n=
ame__]<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 underlying_function =3D Foo.meth<=
br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 defining_class =3D Foo<br>
<br>
.. note::<br>
<br>
=C2=A0 =C2=A0 The defining class is not ``type(self)``, since ``type(self)`=
` might<br>
=C2=A0 =C2=A0 be a subclass of ``Foo``.<br>
<br>
Implicitly, the last three of those rely on name-based lookup via the<br>
function&#39;s ``__globals__`` attribute:<br>
either the ``Foo`` attribute to access the defining class and Python<br>
function object, or ``__name__`` to find the module object in<br>
``sys.modules``.<br>
In Python code, this is feasible, as ``__globals__`` is set<br>
appropriately when the function definition is executed, and<br>
even if the namespace has been manipulated to return a different object,<br=
>
at worst an exception will be raised.<br>
<br>
By contrast, extension methods are typically implemented as normal C<br>
functions. This means that they only have access to their arguments, and<br=
>
any C level thread local and process global state. Traditionally, many<br>
extension modules have stored<br>
their shared state in C level process globals, causing problems when:<br>
<br>
=C2=A0 =C2=A0 * running multiple initialize/finalize cycles in the same pro=
cess<br>
=C2=A0 =C2=A0 * reloading modules (e.g. to test conditional imports)<br>
=C2=A0 =C2=A0 * loading extension modules in subinterpreters<br>
<br>
PEP 3121 attempted to resolve this by offering the<br>
``PyState_FindModule`` API, but this still had significant problems when<br=
>
it comes to extension methods (rather than module level functions):<br>
<br>
=C2=A0 =C2=A0 * it is markedly slower than directly accessing C level proce=
ss<br>
global state<br>
=C2=A0 =C2=A0 * there is still some inherent reliance on process global sta=
te that<br>
means it still doesn&#39;t reliably handle module reloading<br>
<br>
It&#39;s also the case that when looking up a C-level struct such as module=
<br>
state, supplying<br>
an unexpected object layout can crash the interpreter, so it&#39;s<br>
significantly more important to ensure that extension<br>
methods receive the kind of object they expect.<br>
<br>
Proposal<br>
=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
Currently, a bound extension method (``PyCFunction`` or<br>
``PyCFunctionWithKeywords``) receives only<br>
``self``, and (if applicable) the supplied positional and keyword<br>
arguments.<br>
<br>
While module-level extension functions already receive access to the<br>
defining module object via their<br>
``self`` argument, methods of extension types don&#39;t have that luxury:<b=
r>
they receive the bound instance<br>
via ``self``, and hence have no direct access to the defining class or<br>
the module level state.<br>
<br>
The additional module level context described above can be made<br>
available with two changes.<br>
Both additions are optional; extension authors need to opt in to start<br>
using them:<br>
<br>
=C2=A0 =C2=A0 * Add a pointer to the module to heap type objects.<br>
<br>
=C2=A0 =C2=A0 * Pass the defining class to the underlying C function.<br>
<br>
=C2=A0 =C2=A0 =C2=A0 The defining class is readily available at the time bu=
ilt-in<br>
=C2=A0 =C2=A0 =C2=A0 method objects (``PyCFunctionObject``) are created, so=
 it can be<br>
stored<br>
=C2=A0 =C2=A0 =C2=A0 in a new struct that extends ``PyCFunctionObject``.<br=
>
<br>
The module state can then be retrieved from the module object via<br>
``PyModule_GetState``.<br>
<br>
Note that this proposal implies that any type whose method needs to access<=
br>
module-global state must be a heap type dynamically created during extensio=
n<br>
module initialisation, rather than a static type predefined when the<br>
extension<br>
module is compiled.<br>
<br>
This is necessary to support loading multiple module objects from a single<=
br>
extension: a static type, as a C-level global, has no information about<br>
which module it belongs to.<br>
<br>
<br>
Slot methods<br>
------------<br>
<br>
The above changes don&#39;t cover slot methods, such as ``tp_iter`` or<br>
``nb_add``.<br>
<br>
The problem with slot methods is that their C API is fixed, so we can&#39;t=
<br>
simply add a new argument to pass in the defining class.<br>
Two possible solutions have been proposed to this problem:<br>
<br>
=C2=A0 =C2=A0 * Look up the class through walking the MRO.<br>
=C2=A0 =C2=A0 =C2=A0 This is potentially expensive, but will be useful if p=
erformance<br>
is not<br>
=C2=A0 =C2=A0 =C2=A0 a problem (such as when raising a module-level excepti=
on).<br>
=C2=A0 =C2=A0 * Storing a pointer to the defining class of each slot in a s=
eparate<br>
table,<br>
=C2=A0 =C2=A0 =C2=A0 ``__typeslots__`` [#typeslots-mail]_.=C2=A0 This is te=
chnically<br>
feasible and fast,<br>
=C2=A0 =C2=A0 =C2=A0 but quite invasive.<br>
<br>
Due to the invasiveness of the latter approach, this PEP proposes adding<br=
>
a MRO walking helper for use in slot method implementations, deferring<br>
the more complex alternative as a potential future optimisation.<br>
<br>
<br>
Specification<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
Adding module references to heap types<br>
--------------------------------------<br>
<br>
The ``PyHeapTypeObject`` struct will get a new member, ``PyObject<br>
*ht_module``,<br>
that can store a pointer to the module object for which the type was<br>
defined.<br>
It will be ``NULL`` by default, and should not be modified after the type<b=
r>
object is created.<br>
<br>
A new flag, ``Py_TPFLAGS_HAVE_MODULE``, will be set on any type object wher=
e<br>
the ``ht_module`` member is present and non-NULL.<br>
<br>
A new factory method will be added for creating modules::<br>
<br>
=C2=A0 =C2=A0 PyObject* PyType_FromModuleAndSpec(PyObject *module,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PyType_Sp=
ec *spec,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PyObject =
*bases)<br>
<br>
This acts the same as ``PyType_FromSpecWithBases``, and additionally sets<b=
r>
``ht_module`` to the provided module object.<br>
<br>
Additionally, an accessor, ``PyObject * PyType_GetModule(PyTypeObject *)``<=
br>
will be provided.<br>
It will return the ``ht_module`` if a heap type with<br>
Py_TPFLAGS_HAVE_MODULE is passed in,<br>
otherwise it will set a SystemError and return NULL.<br>
<br>
Usually, creating a class with ``ht_module`` set will create a reference<br=
>
cycle involving the class and the module.<br>
This is not a problem, as tearing down modules is not a<br>
performance-sensitive<br>
operation.<br>
Module-level functions typically also create reference cycles.<br>
<br>
<br>
Passing the defining class to extension methods<br>
-----------------------------------------------<br>
<br>
A new style of C-level functions will be added to the current selection of<=
br>
``PyCFunction`` and ``PyCFunctionWithKeywords``::<br>
<br>
=C2=A0 =C2=A0 PyObject *PyCMethod(PyObject *self,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 PyTypeObject *defining_class,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 PyObject *args, PyObject *kwargs)<br>
<br>
A new method object flag, ``METH_METHOD``, will be added to signal that<br>
the underlying C function is ``PyCMethod``.<br>
<br>
To hold the extra information, a new structure extending<br>
``PyCFunctionObject``<br>
will be added::<br>
<br>
=C2=A0 =C2=A0 typedef struct {<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 PyCFunctionObject func;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 PyTypeObject *mm_class; /* Passed as &#39;defin=
ing_class&#39; arg to the<br>
C func */<br>
=C2=A0 =C2=A0 } PyCMethodObject;<br>
<br>
Method construction and calling code and will be updated to honor<br>
``METH_METHOD``.<br>
<br>
Slot methods<br>
------------<br>
<br>
XXX: Exact API TBD<br>
<br>
<br>
Helpers<br>
-------<br>
<br>
XXX: I&#39;d like to port a bunch of modules to see what helpers would be<b=
r>
convenient<br>
<br>
<br>
Argument Clinic<br>
---------------<br>
<br>
XXX [How does this affect Argument Clinic?]<br>
<br>
<br>
<br>
Summary of API Changes and Additions<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
XXX, see above for now<br>
<br>
<br>
Backwards Compatibility<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
One new pointer is added to all heap types.<br>
All other changes are adding new functions and structures.<br>
<br>
<br>
Implementation<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
An initial implementation is available in a Github repository [#gh-repo]_;<=
br>
a patchset is at [#gh-patch]_.<br>
<br>
<br>
Possible Future Extensions<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D<br>
<br>
Easy creation of types with module references<br>
---------------------------------------------<br>
<br>
It would be possible to add a PEP 489 execution slot type make<br>
creating heap types significantly easier than calling<br>
``PyType_FromModuleAndSpec``.<br>
This is left to a future PEP.<br>
<br>
<br>
Optimization<br>
------------<br>
<br>
CPython optimizes calls to methods that have restricted signatures,<br>
such as not allowing keyword arguments.<br>
<br>
As proposed here, methods defined with the ``METH_METHOD`` flag do not<br>
support<br>
these optimizations.<br>
<br>
<br>
Discussion<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
XXX Static exceptions<br>
<br>
<br>
References<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
.. [#typeslots-mail] [Import-SIG] On singleton modules, heap types, and<br>
subinterpreters<br>
=C2=A0 =C2=A0(<a href=3D"https://mail.python.org/pipermail/import-sig/2015-=
July/001035.html" rel=3D"noreferrer" target=3D"_blank">https://mail.python.=
org/pipermail/import-sig/2015-July/001035.html</a>)<br>
<br>
.. [#gh-repo]<br>
=C2=A0 =C2=A0<a href=3D"https://github.com/encukou/cpython/commits/module-s=
tate-access" rel=3D"noreferrer" target=3D"_blank">https://github.com/encuko=
u/cpython/commits/module-state-access</a><br>
<br>
.. [#gh-patch]<br>
<br>
<a href=3D"https://github.com/encukou/cpython/compare/master...encukou:modu=
le-state-access.patch" rel=3D"noreferrer" target=3D"_blank">https://github.=
com/encukou/cpython/compare/master...encukou:module-state-access.patch</a><=
br>
<br>
<br>
Copyright<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
This document has been placed in the public domain.<br>
<br>
<br>
<br>
..<br>
=C2=A0 =C2=A0Local Variables:<br>
=C2=A0 =C2=A0mode: indented-text<br>
=C2=A0 =C2=A0indent-tabs-mode: nil<br>
=C2=A0 =C2=A0sentence-end-double-space: t<br>
=C2=A0 =C2=A0fill-column: 70<br>
=C2=A0 =C2=A0coding: utf-8<br>
=C2=A0 =C2=A0End:<br>
<br>
<br>
_______________________________________________<br>
Import-SIG mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">Import-SIG@pytho=
n.org</a><br>
<a href=3D"https://mail.python.org/mailman/listinfo/import-sig" rel=3D"nore=
ferrer" target=3D"_blank">https://mail.python.org/mailman/listinfo/import-s=
ig</a><br>
</blockquote></div>

--001a114dd9500eef930534a57702--

--===============8964196349805118765==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Import-SIG mailing list
[email protected]
https://mail.python.org/mailman/listinfo/import-sig

--===============8964196349805118765==--