Re: PEP 451: Big update.

Looking good! Comments inline.


On Wed, Sep 18, 2013 at 5:51 AM, Eric Snow <[email protected]>wrote:

> Hi all,
>
> I finally got some time to update the PEP.  I've simplified a few things,
> most notably by making the 4 ModuleSpec methods (create, exec, load,
> reload) "private".
>
> Also notable is that the new loader method is still create_module() and
> there is still no flag for is_reload on either of the loader methods.  I'm
> still not clear on what the flag buys us and on why anything we'd do in a
> prepare_module() we couldn't do in exec_module().  I'm trying to keep this
> simple. :)
>
> Anyway, I still need to take some time to clean up the PEP formatting and
> run a spell checker.  I probably also missed some artifact of an older
> version of the API.  Otherwise I think it's in a good spot.  Comments
> welcome.
>
> -eric
>
> p.s. I also plan on getting the implementation up one of these days. :P
>
> ===============================================================
>
> PEP: 451
> Title: A ModuleSpec Type for the Import System
> Version: $Revision$
> Last-Modified: $Date$
> Author: Eric Snow <[email protected]>
> Discussions-To: [email protected]
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 8-Aug-2013
> Python-Version: 3.4
> Post-History: 8-Aug-2013, 28-Aug-2013, 18-Sep-2013
> Resolution:
>
>
 [SNIP]


> Specification
> =============
>
> The goal is to address the gap between finders and loaders while
> changing as little of their semantics as possible.  Though some
> functionality and information is moved to the new ``ModuleSpec`` type,
> their behavior should remain the same.  However, for the sake of clarity
> the finder and loader semantics will be explicitly identified.
>
> This is a high-level summary of the changes described by this PEP.  More
> detail is available in later sections.
>
> importlib.machinery.ModuleSpec (new)
> ------------------------------------
>
> A specification for a module's import-system-related state.
>
> * ModuleSpec(name, loader, \*, origin=None, loading_info=None,
> is_package=None)
>
> Attributes:
>
> * name - a string for the name of the module.
> * loader - the loader to use for loading and for module data.
>

Just drop the "and for module data"; sentence is awkward with it and is a
margin use-case.


> * origin - a string for the location from which the module is loaded,
>   e.g. "builtin" for built-in modules and the filename for modules
>   loaded from source.
> * submodule_search_locations - strings for where to find submodules,
>   if a package.
>

Very subtle hint that it's a sequence of of strings; might want to make it
more explicit that it's a list.


>  * loading_info - a container of extra data for use during loading.
> * cached (property) - a string for where the compiled module will be
>   stored (see PEP 3147).
> * package (RO-property) - the name of the module's parent (or None).
> * has_location (RO-property) - the module's origin refers to a location.
>
> Instance Methods:
>
> * module_repr() - provide a repr string for the spec'ed module.
> * init_module_attrs(module) - set any of a module's import-related
>   attributes that aren't already set.
>
> importlib.util Additions
> ------------------------
>
> * spec_from_file_location(name, location, \*, loader=None,
> submodule_search_locations=None)
>   - factory for file-based module specs.
> * from_loader(name, loader, \*, origin=None, is_package=None) - factory
>   based on information provided by loaders.
> * spec_from_module(module, loader=None) - factory based on existing
>   import-related module attributes.  This function is expected to be
>   used only in some backward-compatibility situations.
>
> Other API Additions
> -------------------
>
> * importlib.abc.Loader.exec_module(module) will execute a module in its
>   own namespace.  It replaces ``importlib.abc.Loader.load_module()``.
> * importlib.abc.Loader.create_module(spec) (optional) will return a new
>   module to use for loading.
> * Module objects will have a new attribute: ``__spec__``.
> * importlib.find_spec(name, path=None) will return the spec for a
>   module.
>
> exec_module() and create_module() should not set any import-related
> module attributes.  The fact that load_module() does is a design flaw
> that this proposal aims to correct.
>

This is a rather jarring place to make this statement since you're just
outlining API additions, not design decisions.


>
> API Changes
> -----------
>
> * ``InspectLoader.is_package()`` will become optional.
>
> Deprecations
> ------------
>
> * importlib.abc.MetaPathFinder.find_module()
> * importlib.abc.PathEntryFinder.find_module()
> * importlib.abc.PathEntryFinder.find_loader()
> * importlib.abc.Loader.load_module()
> * importlib.abc.Loader.module_repr()
> * The parameters and attributes of the various loaders in
>   importlib.machinery
> * importlib.util.set_package()
> * importlib.util.set_loader()
> * importlib.find_loader()
>

Yay to all of this! =)


>
> Removals
> --------
>
> These were introduced prior to Python 3.4's release.
>
> * importlib.abc.Loader.init_module_attrs()
> * importlib.util.module_to_load()
>
> Other Changes
> -------------
>
> * The import system implementation in importlib will be changed to make
>   use of ModuleSpec.
> * Import-related module attributes (other than ``__spec__``) will no
>   longer be used directly by the import system.
> * Import-related attributes should no longer be added to modules
>   directly.
> * The module type's ``__repr__()`` will be thin wrapper around a pure
>   Python implementation which will leverage ModuleSpec.
>

"be a thin"


> * The spec for the ``__main__`` module will reflect the appropriate
>   name and origin.
>
> Backward-Compatibility
> ----------------------
>
> * If a finder does not define find_spec(), a spec is derived from
>   the loader returned by find_module().
> * PathEntryFinder.find_loader() still takes priority over
>   find_module().
> * Loader.load_module() is used if exec_module() is not defined.
>
> What Will not Change?
> ---------------------
>
> * The syntax and semantics of the import statement.
> * Existing finders and loaders will continue to work normally.
> * The import-related module attributes will still be initialized with
>   the same information.
> * Finders will still create loaders (now storing them in specs).
> * Loader.load_module(), if a module defines it, will have all the
>   same requirements and may still be called directly.
> * Loaders will still be responsible for module data APIs.
> * importlib.reload() will still overwrite the import-related attributes.
>
>
> What Will Existing Finders and Loaders Have to Do Differently?
> ==============================================================
>
> Immediately?  Nothing.  The status quo will be deprecated, but will
> continue working.  However, here are the things that the authors of
> finders and loaders should change relative to this PEP:
>
> * Implement ``find_spec()`` on finders.
> * Implement ``exec_module()`` on loaders, if possible.
>
> The ModuleSpec factory functions in importlib.util are intended to be
> helpful for converting existing finders.  ``from_loader()`` and
> ``from_file_location()`` are both straight-forward utilities in this
> regard.  In the case where loaders already expose methods for creating
> and preparing modules, ``ModuleSpec.from_module()`` may be useful to
> the corresponding finder.
>
> For existing loaders, exec_module() should be a relatively direct
> conversion from the non-boilerplate portion of load_module().  In some
> uncommon cases the loader should also implement create_module().
>
>
> ModuleSpec Users
> ================
>
> ``ModuleSpec`` objects has 3 distinct target audiences: Python itself,
> import hooks, and normal Python users.
>

"has" -> "have"


>
> Python will use specs in the import machinery, in interpreter startup,
> and in various standard library modules.  Some modules are
> import-oriented, like pkgutil, and others are not, like pickle and
> pydoc.  In all cases, the full ``ModuleSpec`` API will get used.
>
> Import hooks (finders and loaders) will make use of the spec in specific
> ways.  First of all, finders may use the spec factory functions in
> importlib.util to create spec objects.  They may also directly adjust
> the spec attributes after the spec is created.  Secondly, the finder may
> bind additional information to the spec (in finder_extras) for the
> loader to consume during module creation/execution.  Finally, loaders
> will make use of the attributes on a spec when creating and/or executing
> a module.
>
> Python users will be able to inspect a module's ``__spec__`` to get
> import-related information about the object.  Generally, Python
> applications and interactive users will not be using the ``ModuleSpec``
> factory functions nor any the instance methods.
>
>
> How Loading Will Work
> =====================
>
> This is an outline of what happens in ModuleSpec's loading
> functionality::
>
>    def load(spec):
>        if not hasattr(spec.loader, 'exec_module'):
>            module = spec.loader.load_module(spec.name)
>            spec.init_module_attrs(module)
>            return sys.modules[spec.name]
>
>        module = None
>        if hasattr(spec.loader, 'create_module'):
>            module = spec.loader.create_module(spec)
>        if module is None:
>            module = ModuleType(spec.name)
>        spec.init_module_attrs(module)
>
>        spec._initializing = True
>        sys.modues[spec.name] = module
>        try:
>            spec.loader.exec_module(module)
>        except Exception:
>            del sys.modules[spec.name]
>        finally:
>            spec._initializing = False
>        return sys.modules[spec.name]
>
> These steps are exactly what ``Loader.load_module()`` is already
> expected to do.  Loaders will thus be simplified since they will only
> need to implement exec_module().
>

Two things. One, it's not exactly what loaders do as that _initializing is
done by import itself. Any specific reason you added it here?

Two, you forgot to re-raise the exception in the except clause.


>
> Note that we must return the module from sys.modules.  During loading
> the module may have replaced itself in sys.modules.  Since we don't have
> a post-import hook API to accommodate the use case, we have to deal with
> it.  However, in the replacement case we do not worry about setting the
> import-related module attributes on the object.  The module writer is on
> their own if they are doing this.
>
>
> ModuleSpec
> ==========
>
> Attributes
> ----------
>
> Each of the following names is an attribute on ModuleSpec objects.  A
> value of ``None`` indicates "not set".  This contrasts with module
> objects where the attribute simply doesn't exist.  Most of the
> attributes correspond to the import-related attributes of modules.  Here
> is the mapping.  The reverse of this mapping is used by
> ModuleSpec.init_module_attrs().
>
> ========================== ==============
> On ModuleSpec              On Modules
> ========================== ==============
> name                       __name__
> loader                     __loader__
> package                    __package__
> origin                     __file__*
> cached                     __cached__*,**
> submodule_search_locations __path__**
> loading_info                \-
> has_location                \-
> ========================== ==============
>
> \* Set only if has_location is true.
> \*\* Set only if the spec attribute is not None.
>

"Set on the module if the spec"


>
> While package and has_location are read-only properties, the remaining
> attributes can be replaced after the module spec is created and even
> after import is complete.  This allows for unusual cases where directly
> modifying the spec is the best option.  However, typical use should not
> involve changing the state of a module's spec.
>
> **origin**
>
> origin is a string for the place from which the module originates.
> Aside from the informational value, it is also used in module_repr().
>
> The module attribute ``__file__`` has a similar but more restricted
> meaning.  Not all modules have it set (e.g. built-in modules).  However,
> ``origin`` is applicable to all modules.  For built-in modules it would
> be set to "built-in".
>
> **has_location**
>
> Some modules can be loaded by reference to a location, e.g. a filesystem
> path or a URL or something of the sort.  Having the location lets you
> load the module, but in theory you could load that module under various
> names.
>
> In contrast, non-located modules can't be loaded in this fashion, e.g.
> builtin modules and modules dynamically created in code.  For these, the
> name is the only way to access them, so they have an "origin" but not a
> "location".
>
> This attribute reflects whether or not the module is locatable.  If it
> is, origin must be set to the module's location and ``__file__`` will be
> set on the module.  Not all locatable modules will be cachable, but most
> will.
>
> The corresponding module attribute name, ``__file__``, is somewhat
> inaccurate and potentially confusion,
>

"confusion" -> "confusing"


> so we will use a more explicit
> combination of origin and has_location to represent the same
> information.  Having a separate filename is unncessary since we have
> origin.
>

Quote 'origin' so you don't read it like it should have been written "we
have an origin".


>
> **submodule_search_locations**
>
> The list of location strings, typically directory paths, in which to
> search for submodules.  If the module is a package this will be set to
> a list (even an empty one).  Otherwise it is ``None``.
>
> The corresponding module attribute's name, ``__path__``, is relatively
> ambiguous.  Instead of mirroring it, we use a more explicit name that
> makes the purpose clear.
>
> **loading_info**
>
> A finder may set loading_info to any value to provide additional
> data for the loader to use during loading.  A value of None is the
> default and indicates that there is no additional data.  Otherwise it
> can be set to any object, such as a dict, list, or
> types.SimpleNamespace, containing the relevant extra information.
>
> For example, zipimporter could use it to pass the zip archive name
> to the loader directly, rather than needing to derive it from origin
> or create a custom loader for each find operation.
>
> loading_info is meant for use by the finder and corresponding loader.
> It is not guaranteed to be a stable resource for any other use.
>
> Omitted Attributes and Methods
> ------------------------------
>
> The following ModuleSpec methods are not part of the public API since
> it is easy to use them incorrectly and only the import system really
> needs them (i.e. they would be an attractive nuisance).
>
> * create() - provide a new module to use for loading.
> * exec(module) - execute the spec into a module namespace.
> * load() - prepare a module and execute it in a protected way.
> * reload(module) - re-execute a module in a protected way.
>

If they are not part of the public API they should have a leading
underscore.


>
> Here are other omissions:
>
> There is no PathModuleSpec subclass of ModuleSpec that separates out
> has_location, cached, and submodule_search_locations.  While that might
> make the separation cleaner, module objects don't have that distinction.
> ModuleSpec will support both cases equally well.
>
> While is_package would be a simple additional attribute (aliasing
> ``self.submodule_search_locations is not None``), it perpetuates the
> artificial (and mostly erroneous) distinction between modules and
> packages.
>
> Conceivably, a ModuleSpec.load() method could optionally take a list of
> modules with which to interact instead of sys.modules.  That
> capability is left out of this PEP, but may be pursued separately at
> some other time, including relative to PEP 406 (import engine).
>
> Likewise load() could be leveraged to implement multi-version
> imports.  While interesting, doing so is outside the scope of this
> proposal.
>
> Others:
>
> * Add ModuleSpec.submodules (RO-property) - returns possible submodules
>   relative to the spec.
> * Add ModuleSpec.loaded (RO-property) - the module in sys.module, if
>   any.
> * Add ModuleSpec.data - a descriptor that wraps the data API of the
>   spec's loader.
> * Also see [3].
>
>
> Backward Compatibility
> ----------------------
>
> ModuleSpec doesn't have any.  This would be a different story if
> Finder.find_module() were to return a module spec instead of loader.
> In that case, specs would have to act like the loader that would have
> been returned instead.  Doing so would be relatively simple, but is an
> unnecessary complication.  It was part of earlier versions of this PEP.
>
> Subclassing
> -----------
>
> Subclasses of ModuleSpec are allowed, but should not be necessary.
> Simply setting loading_info or adding functionality to a custom
> finder or loader will likely be a better fit and should be tried first.
> However, as long as a subclass still fulfills the requirements of the
> import system, objects of that type are completely fine as the return
> value of Finder.find_spec().
>
>
>
>
[SNIP]


>
>
> Open Issues
> ==============
>
> \* The impact of this change on pkgutil (and setuptools) needs looking
> into.  It has some generic function-based extensions to PEP 302.  These
> may break if importlib starts wrapping loaders without the tools'
> knowledge.
>
> \* Other modules to look at: runpy (and pythonrun.c), pickle, pydoc,
> inspect.
>
> For instance, pickle should be updated in the __main__ case to look at
> ``module.__spec__.name``.
>
> \* Impact on some kinds of lazy loading modules.  See [3].
>
> \* Find a better name than loading_info?  Perhaps loading_data,
> loader_state, or loader_info.
>

loader_state or loader_data get my vote.


>
> \* Change loader.create_module() to prepare_module()?
>

-0 from me.

-Brett

_______________________________________________
Import-SIG mailing list
[email protected]
https://mail.python.org/mailman/listinfo/import-sig