Re: Loading Resources From a Python Module/Package

> On Jan 30, 2015, at 7:18 PM, Paul Moore <[email protected]> wrote:
> 
> On 30 January 2015 at 23:37, Donald Stufft <[email protected]> wrote:
>> A. What do people think about pkgutil.get_data_filename and
>>   Loader.get_data_filename?
> 
> Sounds reasonable. It's a relatively rare, but useful use case. One
> possible issue, though, would people assume that if they get a
> filename it'd be writeable? For the filesystem loader it would be, but
> that would break subtly (writes work but would get discarded) for
> loaders that don't have a native get_data_filename.

I don’t think you can assume it’s writeable since that’ll break in a lot
of common cases even with the filesystem loader since often times things
in the filesystem will be installed in the system and users won’t have
permissions to write to them anyways.

> 
> Related question - how would the temp files be cleaned up? At exit?

My patch registers an atexit handler that cleans up the temporary files yea.

> 
>> B. What do people think about modifying Loader.get_data so it can support
>>   relative filenames instead of the calling code needing to handle that?
> 
> I'd have to think about that one, but in principle it seems reasonable.
> 
> While we're extending the loaders, a far more commonly requested
> feature would be to list available data files. At the moment, code can
> only load data from known paths, which is not ideal. While it's
> unrelated to the original proposal, it makes sense if we're changing
> the spec of loaders to do it in one go, rather than having multiple
> iterations.

Well both pkgutil.get_data and pkgutil.get_data_filename have parallels in the
pkg_resources library for similar reasons. If we want to extend this to more
things it might make sense to take a look at what all exists there currently:

resource_exists(package_or_requirement, resource_name)
    Does the named resource exist? Return True or False accordingly.

resource_stream(package_or_requirement, resource_name)
    Return a readable file-like object for the specified resource; it may be an
    actual file, a StringIO, or some similar object. The stream is in
    “binary mode”, in the sense that whatever bytes are in the resource will be
    read as-is.

resource_string(package_or_requirement, resource_name)
    Return the specified resource as a string. The resource is read in binary
    fashion, such that the returned string contains exactly the bytes that are
    stored in the resource.

resource_isdir(package_or_requirement, resource_name)
    Is the named resource a directory? Return True or False accordingly.

resource_listdir(package_or_requirement, resource_name)
    List the contents of the named resource directory, just like os.listdir
    except that it works even if the resource is in a zipfile.

resource_filename(package_or_requirement, resource_name)
    Sometimes, it is not sufficient to access a resource in string or stream
    form, and a true filesystem filename is needed. In such cases, you can use
    this method (or module-level function) to obtain a filename for a resource.
    If the resource is in an archive distribution (such as a zipped egg), it
    will be extracted to a cache directory, and the filename within the cache
    will be returned. If the named resource is a directory, then all resources
    within that directory (including subdirectories) are also extracted. If the
    named resource is a C extension or “eager resource” (see the setuptools
    documentation for details), then all C extensions and eager resources are
    extracted at the same time.

See https://pythonhosted.org/setuptools/pkg_resources.html#basic-resource-access
and https://pythonhosted.org/setuptools/pkg_resources.html#resource-extraction

Obviously the similar functions here are:

* pkgutil.get_data is pkg_resources.resource_string
* pkgutil.get_data_filename is pkg_resources.resource_filename

The major difference being that pkg_resource.resource_filename will extract to
a cache directory (controllable with an environment variable or
programatically) and won't clean up the extracted files. This means that they
are (by default) extracted once per user and reused between extractions. I felt
like it made more sense to just extract to a temporary location (even though
this is less performant) in the stdlib.

That leaves:

* resource_exists
* resource_stream
* resource_isdir
* resource_listdir

Which can be done via pkg_resources but not via the standard library, I don't
have a major opinion on whether or not the standard library should do all of
them but I don't think it would hurt if it did.

Another interesting question if we're going to add more methods is where they
should all live. As far as I know pkgutil.get_data predates the importlib
module. Perhaps deprecating pkgutil.get_data and adding a importlib.resources
module which supports functions like:

* get_bytes(package, resource)
* get_stream(package, resource)
* get_filename(package, resource)
* exists(package, resource)
* isdir(package, resource)
* listdir(package, resource)

Changing the names (particular get_data -> get_bytes) could also provide the
mechanism for allowing relative files and deprecating the "you must pass in
a full file path to the Loader()" behavior since the get_data method could be
left alone and a new get_bytes method could be added.

This would mean people can do things like:

    import importlib.resources
    import socket
    import ssl

    context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
    context.verify_mode = ssl.CERT_REQUIRED
    context.check_hostname = True
    context.load_verify_locations(
        cafile=importlib.resources.get_filename("certifi", "cacert.pem"),
    )

    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    ssl_sock = context.wrap_socket(s, server_hostname='www.verisign.com')
    ssl_sock.connect(('www.verisign.com', 443))

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________
Import-SIG mailing list
[email protected]
https://mail.python.org/mailman/listinfo/import-sig