Re: Loading Resources From a Python Module/Package

 On Fri, Jan 30, 2015, 19:52 Donald Stufft <[email protected]> wrote:


> On Jan 30, 2015, at 7:18 PM, Paul Moore <[email protected]> wrote:
>
> On 30 January 2015 at 23:37, Donald Stufft <[email protected]> wrote:
>> A. What do people think about pkgutil.get_data_filename and
>>   Loader.get_data_filename?
>
> Sounds reasonable. It's a relatively rare, but useful use case. One
> possible issue, though, would people assume that if they get a
> filename it'd be writeable? For the filesystem loader it would be, but
> that would break subtly (writes work but would get discarded) for
> loaders that don't have a native get_data_filename.

I don’t think you can assume it’s writeable since that’ll break in a lot
of common cases even with the filesystem loader since often times things
in the filesystem will be installed in the system and users won’t have
permissions to write to them anyways.

>
> Related question - how would the temp files be cleaned up? At exit?

My patch registers an atexit handler that cleans up the temporary files yea.

>
>> B. What do people think about modifying Loader.get_data so it can support
>>   relative filenames instead of the calling code needing to handle that?
>
> I'd have to think about that one, but in principle it seems reasonable.
>
> While we're extending the loaders, a far more commonly requested
> feature would be to list available data files. At the moment, code can
> only load data from known paths, which is not ideal. While it's
> unrelated to the original proposal, it makes sense if we're changing
> the spec of loaders to do it in one go, rather than having multiple
> iterations.

Well both pkgutil.get_data and pkgutil.get_data_filename have parallels in
the
pkg_resources library for similar reasons. If we want to extend this to more
things it might make sense to take a look at what all exists there
currently:

resource_exists(package_or_requirement, resource_name)
    Does the named resource exist? Return True or False accordingly.

resource_stream(package_or_requirement, resource_name)
    Return a readable file-like object for the specified resource; it may
be an
    actual file, a StringIO, or some similar object. The stream is in
    “binary mode”, in the sense that whatever bytes are in the resource
will be
    read as-is.

resource_string(package_or_requirement, resource_name)
    Return the specified resource as a string. The resource is read in
binary
    fashion, such that the returned string contains exactly the bytes that
are
    stored in the resource.

resource_isdir(package_or_requirement, resource_name)
    Is the named resource a directory? Return True or False accordingly.

resource_listdir(package_or_requirement, resource_name)
    List the contents of the named resource directory, just like os.listdir
    except that it works even if the resource is in a zipfile.

resource_filename(package_or_requirement, resource_name)
    Sometimes, it is not sufficient to access a resource in string or stream
    form, and a true filesystem filename is needed. In such cases, you can
use
    this method (or module-level function) to obtain a filename for a
resource.
    If the resource is in an archive distribution (such as a zipped egg), it
    will be extracted to a cache directory, and the filename within the
cache
    will be returned. If the named resource is a directory, then all
resources
    within that directory (including subdirectories) are also extracted. If
the
    named resource is a C extension or “eager resource” (see the setuptools
    documentation for details), then all C extensions and eager resources
are
    extracted at the same time.

See
https://pythonhosted.org/setuptools/pkg_resources.html#basic-resource-access
and
https://pythonhosted.org/setuptools/pkg_resources.html#resource-extraction

Obviously the similar functions here are:

* pkgutil.get_data is pkg_resources.resource_string
* pkgutil.get_data_filename is pkg_resources.resource_filename

The major difference being that pkg_resource.resource_filename will extract
to
a cache directory (controllable with an environment variable or
programatically) and won't clean up the extracted files. This means that
they
are (by default) extracted once per user and reused between extractions. I
felt
like it made more sense to just extract to a temporary location (even though
this is less performant) in the stdlib.

That leaves:

* resource_exists
* resource_stream
* resource_isdir
* resource_listdir

Which can be done via pkg_resources but not via the standard library, I
don't
have a major opinion on whether or not the standard library should do all of
them but I don't think it would hurt if it did.

Another interesting question if we're going to add more methods is where
they
should all live. As far as I know pkgutil.get_data predates the importlib
module.

 It does, so you really have to think in terms of finders and loaders.

Perhaps deprecating pkgutil.get_data and adding a importlib.resources
module which supports functions like:

* get_bytes(package, resource)
* get_stream(package, resource)
* get_filename(package, resource)
* exists(package, resource)
* isdir(package, resource)
* listdir(package, resource)

Changing the names (particular get_data -> get_bytes) could also provide the
mechanism for allowing relative files and deprecating the "you must pass in
a full file path to the Loader()" behavior since the get_data method could
be
left alone and a new get_bytes method could be added.


The reason Loader.get_data() takes absolute paths is to do away with
ambiguity. If you have a relative path and ask a loader to read that path,
where should that relative path be anchored? Should it be the top-level
package? What about the module that loader ewas returned to handle? But
then what about if a finder caches loaders and reuses them across modules
(nothing in PEP 302 says you can't do this and in actuality the frozen and
built-in loaders are just static and class methods). The choice of dealing
exclusively in absolute paths was a conscious choice on my part.

Now having said that, there is nothing to say absolute paths require file
system I based paths. What you should really do is think of these paths as
opaque, non-ambiguous paths for the loader which claimed it knew what file
path was needed to pass to get_data(). If you think that way then you
realize you can use markers in the path as necessary, e.g.
some/path/file.zip/pkg/sub/data.txt. As long as loader.get_data() can
unambiguously read that path as returned by get_data_filename() or whatever
the method is called then you have fully abstracted paths out while still
being able to read data from a loader.

Basically any API dealing with paths for loaders needs to abstract away the
concept of files, file-like paths, etc. and rely on using the loader API on
pretty much everything as a simple os.path of its own. This is why I have
not tried to tackle the issue of the list_contents() or some such API to
list modules and potentially data files as it needs to not really have a
concrete concept of file paths (and it really should be on finders and not
loaders which complicates discovery, selecting the right finder, etc.).
This is also why APIs wanting a file path instead of taking a file-like
object simply cannot play well with importlib and loaders which have
alternative back end storage without simply being lucky that the loader
they are working with uses filesystem paths (or writing out to a temp file).

-brett

This would mean people can do things like:

    import importlib.resources
    import socket
    import ssl

    context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
    context.verify_mode = ssl.CERT_REQUIRED
    context.check_hostname = True
    context.load_verify_locations(
        cafile=importlib.resources.get_filename("certifi", "cacert.pem"),
    )

    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    ssl_sock = context.wrap_socket(s, server_hostname='www.verisign.com')
    ssl_sock.connect(('www.verisign.com', 443))

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________
Import-SIG mailing list
[email protected]
https://mail.python.org/mailman/listinfo/import-sig

_______________________________________________
Import-SIG mailing list
[email protected]
https://mail.python.org/mailman/listinfo/import-sig