A ModuleData API (was: Re: making it feasible to rely on loaders for reading intra-package data files)

On Sat, Feb 1, 2014 at 11:44 AM, Brett Cannon <[email protected]> wrote:
> Over on distutils-sig it came up that getting people to not simply assume
> that __file__ points to an actual file and thus avoid using open() directly
> to read intra-package files is an issue. In order to make using a loader's
> get_data reasonable (let alone set_data), there needs to be a clear
> specification of how things are expected to work and make sure that
> everything that people need is available.
>
> The docs for importlib.ResourceLoader.get_data
> (http://docs.python.org/3.4/library/importlib.html#importlib.abc.ResourceLoader.get_data)
> say that things are expected to be based off of __file__, and with Python
> 3.4 using only absolute paths (except for __main__) that means all paths
> would be absolute by default. As long as people stick to pathlib/os.path and
> don't use non-standard path separators then this should just work.
>
> But what if people don't do that? I honestly say that it should either be
> explicitly undefined or that it's an IOError. IOW either we say "use
> absolute paths or else you're on your own" or "use absolute paths, period".
> That prevents having to make a decision as to whether a relative path is
> relative to the module the loader is attached to or relative to the package
> (e.g. easier for pre-module loaders or per-package loaders, respectively).
> The former is more backwards-compatible so I say the docs get updated to say
> that relative paths are undefined behaviour.
>
> The second issue is whether get_data/set_data are enough or if something
> else is needed, e.g. a listdir-like method. Since this is meant for handling
> intra-package data my assumption is that it isn't really necessary as
> chances are you know what files you included in your distribution (or at
> least what the possible names are). I know some have asked for a
> listdir-like API to help discover what modules are available so as to
> provide a plugin API, but I view that as a separate thing and potentially
> more appropriate on finders. Remember, the smaller the API service for the
> common case the better for the stdlib.

Here's a rough idea that helps consolidate behavior and move the focus
of the data APIs away from loaders and toward modules.

In my mind loader methods are low-level and meant particularly for
consumption by the import system.  It would be nice to have
higher-level APIs for everyone else to use.  It would make sense to
wrap that up in a class.

From what I can tell, use cases for the data-related load API are
module-centric, so it would make sense to have the high-level API
focus on modules, rather than loaders:

class ModuleData:
    def __init__(self, module):
        self.module = module
        self.loader = module.__loader__
    def get_data(self, location):
        return self.loader.get_data(location)
    def set_data(self, location, data):
        return self.loader.set_data(location, data)
    ...

This gives us the ability to generalize standard data-related behavior
across all loaders (kind of like PEP 451 did for loading).  It would
also make customization simpler.

File-based modules/loaders are the common case.  It would be nice to
provide default implementations thereby.  I see two approaches:

* Subclass ModuleData (e.g. FileModuleData).
* Add a boolean "filebased" attr to loaders that ModuleData could use
to trigger customized behavior.

In either case, it would make sense to add a method (e.g.
get_data_api(module)) that returns a ModuleData instance, thus
allowing each loader to pick the type returned.

ModuleData.__init__ implies having the module (already imported).  The
current loader API does not require any module, so that low-level API
would still be useful if someone wanted to avoid loading the module
first.  Alternately there could be a mechanism for building a
ModuleData object from a loader without needing to load the module
first.

(I had brought up something similar with PEP 451, but it was too out
of scope to pursue there.)

-eric