Re: Loading Resources From a Python Module/Package

On 31 January 2015 at 00:52, Donald Stufft <[email protected]> wrote:
>> Sounds reasonable. It's a relatively rare, but useful use case. One
>> possible issue, though, would people assume that if they get a
>> filename it'd be writeable? For the filesystem loader it would be, but
>> that would break subtly (writes work but would get discarded) for
>> loaders that don't have a native get_data_filename.
>
> I don’t think you can assume it’s writeable since that’ll break in a lot
> of common cases even with the filesystem loader since often times things
> in the filesystem will be installed in the system and users won’t have
> permissions to write to them anyways.

Agreed, It's just that it could happen (either deliberately or by
accident). One example I found was pytz, which downloads and builds
the timezone data by doing dirname(__file__) in an "update the DB" API
call - it'd be an "obvious" case for using resource data. (That was
from a long time ago - checking the code now they seem to have tidied
this up so it's no longer that way).

But yes, documenting it as "don't do that" is probably fine.

>> Related question - how would the temp files be cleaned up? At exit?
>
> My patch registers an atexit handler that cleans up the temporary files yea.

Great.

>>> B. What do people think about modifying Loader.get_data so it can support
>>>   relative filenames instead of the calling code needing to handle that?
>>
>> I'd have to think about that one, but in principle it seems reasonable.
>>
>> While we're extending the loaders, a far more commonly requested
>> feature would be to list available data files. At the moment, code can
>> only load data from known paths, which is not ideal. While it's
>> unrelated to the original proposal, it makes sense if we're changing
>> the spec of loaders to do it in one go, rather than having multiple
>> iterations.
>
> Well both pkgutil.get_data and pkgutil.get_data_filename have parallels in the
> pkg_resources library for similar reasons. If we want to extend this to more
> things it might make sense to take a look at what all exists there currently:

+1 on following pkg_resources.

> resource_exists(package_or_requirement, resource_name)
>     Does the named resource exist? Return True or False accordingly.
>
> resource_stream(package_or_requirement, resource_name)
>     Return a readable file-like object for the specified resource; it may be an
>     actual file, a StringIO, or some similar object. The stream is in
>     “binary mode”, in the sense that whatever bytes are in the resource will be
>     read as-is.
>
> resource_string(package_or_requirement, resource_name)
>     Return the specified resource as a string. The resource is read in binary
>     fashion, such that the returned string contains exactly the bytes that are
>     stored in the resource.
>
> resource_isdir(package_or_requirement, resource_name)
>     Is the named resource a directory? Return True or False accordingly.
>
> resource_listdir(package_or_requirement, resource_name)
>     List the contents of the named resource directory, just like os.listdir
>     except that it works even if the resource is in a zipfile.
>
> resource_filename(package_or_requirement, resource_name)
>     Sometimes, it is not sufficient to access a resource in string or stream
>     form, and a true filesystem filename is needed. In such cases, you can use
>     this method (or module-level function) to obtain a filename for a resource.
>     If the resource is in an archive distribution (such as a zipped egg), it
>     will be extracted to a cache directory, and the filename within the cache
>     will be returned. If the named resource is a directory, then all resources
>     within that directory (including subdirectories) are also extracted. If the
>     named resource is a C extension or “eager resource” (see the setuptools
>     documentation for details), then all C extensions and eager resources are
>     extracted at the same time.
>
> See https://pythonhosted.org/setuptools/pkg_resources.html#basic-resource-access
> and https://pythonhosted.org/setuptools/pkg_resources.html#resource-extraction
>
> Obviously the similar functions here are:
>
> * pkgutil.get_data is pkg_resources.resource_string
> * pkgutil.get_data_filename is pkg_resources.resource_filename
>
> The major difference being that pkg_resource.resource_filename will extract to
> a cache directory (controllable with an environment variable or
> programatically) and won't clean up the extracted files. This means that they
> are (by default) extracted once per user and reused between extractions. I felt
> like it made more sense to just extract to a temporary location (even though
> this is less performant) in the stdlib.
>
> That leaves:
>
> * resource_exists
> * resource_stream
> * resource_isdir
> * resource_listdir
>
> Which can be done via pkg_resources but not via the standard library, I don't
> have a major opinion on whether or not the standard library should do all of
> them but I don't think it would hurt if it did.
>
> Another interesting question if we're going to add more methods is where they
> should all live. As far as I know pkgutil.get_data predates the importlib
> module. Perhaps deprecating pkgutil.get_data and adding a importlib.resources
> module which supports functions like:
>
> * get_bytes(package, resource)
> * get_stream(package, resource)
> * get_filename(package, resource)
> * exists(package, resource)
> * isdir(package, resource)
> * listdir(package, resource)
>
> Changing the names (particular get_data -> get_bytes) could also provide the
> mechanism for allowing relative files and deprecating the "you must pass in
> a full file path to the Loader()" behavior since the get_data method could be
> left alone and a new get_bytes method could be added.
>
> This would mean people can do things like:
>
>     import importlib.resources
>     import socket
>     import ssl
>
>     context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
>     context.verify_mode = ssl.CERT_REQUIRED
>     context.check_hostname = True
>     context.load_verify_locations(
>         cafile=importlib.resources.get_filename("certifi", "cacert.pem"),
>     )
>
>     s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>     ssl_sock = context.wrap_socket(s, server_hostname='www.verisign.com')
>     ssl_sock.connect(('www.verisign.com', 443))

+1 on all of the above. Obviously, a lot of the support methods in
loaders would need to be optional, but that's fine - and the vast
majority of use cases are the filesystem and zipfiles, both of which
support these methods, and can be handled in the stdlib.

Paul
_______________________________________________
Import-SIG mailing list
[email protected]
https://mail.python.org/mailman/listinfo/import-sig