Re: Proposed design for importlib.resources()
Brett Cannon <[email protected]> Tue, 24 Nov 2015 20:21:40 +0000
| Newsgroups | gmane.comp.python.import |
|---|---|
| Message-ID | <CAP1=2W533fgMTn-Bu2o5nAN2yny4-bOaH-J_E4J60jcCiaP6WQ@mail.gmail.com> |
--===============6700514378526190725== Content-Type: multipart/alternative; boundary=94eb2c0550ea83f9a505254f17bd --94eb2c0550ea83f9a505254f17bd Content-Type: text/plain; charset=UTF-8 On Mon, 23 Nov 2015 at 13:46 Barry Warsaw <[email protected]> wrote: > On Nov 20, 2015, at 09:23 PM, Brett Cannon wrote: > > >I have created a Jupyter Notebook to explain my thinking on what > >importlib.resources() should be (at least initially). > > Just a few thoughts based on a review of two projects' use of > pkg_resources. > +1 on getting *something* into Python 3.6. > > Module API vs package API. Doesn't pkg_resources actually support > something > similar to both, with the module function providing a convenience API? Yes, but that doesn't sway me. This isn't a "pkg_resources++" but a "make reading data from a package make sense in a modern import world". IOW I'm purposefully not using pkg_resources as a template but simply as a motivating factor. > I like > that a lot because the convenience API is so darn... convenient! You just > give it the Python dotted-path and the resource and it does the rest. > I don't see how that's any different than the other approach since you're still providing the exact same data; no more, no less. > Generally I don't care about caching the results of the search; these calls > are almost never in performance critical code. > Unfortunately for you the poll liked the other approach and TOOWTDI. So either convince me that resources.read_bytes(pkg, path) is better than resources(pkg).read_bytes(path) or consider the bike shed painted. :) > > resource_filename(). Doesn't pkg_resources already have a strategy for the > temporary file that sometimes has to be created? Yes and I don't like it. :) Basically you either create an instance or implicitly use a global instance of a class that stores the references and registers with atexit a cleanup function to be executed. > Maybe it doesn't work so > well on some platforms (I've never noticed a problem on *nix). A context > manager as proposed seems like the most reasonable approach. We definitely > need this API though. I see plenty of examples where e.g. test data files > have to be shutil.copy()'d, passed to subprocess command line arguments, > etc. > OK, between you and Donald saying you have real needs for the API you can rest assured that it will be in the initial version, especially since I already coded up the tempfile implementation. > > read_bytes(). Thank you for the truth in advertising! IIRC in Python 3, > pkg_resource.resource_string() actually returns bytes. from-import-as to > the > rescue. An actual resource_string() would have to accept an encoding > argument > (as would any resource-based open() method). > Yes, which is why I don't think it's worth it to provide a resource_strinng() since calling decode isn't difficult (and is something you must know in Python 3). > > resource_stream(). IIRC, the pkg_resource's version is not a context > manager > so it has to be closed explicitly (or wrapped in contextlib.closing()). We > can do better. > I'm not convinced it's necessary to provide an equivalent open() yet; if you have an API that requires a file-like object then io.BytesIO to the rescue for read_bytes(). There is nothing tricky to get right like with a file path that may or may not be backed by a temporary file. This is a somewhat low-level API and if people want to provide convenience wrappers that's fine but I don't want to start guessing at needs beyond core APIs or ones that are hard to get right and allow for composability to higher APIs like file-like objects which others can handle. > > I do have one use of resource_listdir() which is used to find importable > plugin modules at runtime. It's handy. > I'm going to punt on this for as long as possible because it's asking for trouble to get right. For example, if I do resources(pkg).listdir(), then I will end up returning relative paths, but if you disassociate those paths from pkg then you have lost proper context. You could return tuples of (pkg, relative_path), but that just doesn't seem satisfactory either. I'm just not convinced yet it is needed enough to support (at least initially). -Brett > > That's all for now. > -Barry > _______________________________________________ > Import-SIG mailing list > [email protected] > https://mail.python.org/mailman/listinfo/import-sig > --94eb2c0550ea83f9a505254f17bd Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><br><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Mon= , 23 Nov 2015 at 13:46 Barry Warsaw <<a href=3D"mailto:[email protected]"= >[email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_quote"= style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On= Nov 20, 2015, at 09:23 PM, Brett Cannon wrote:<br> <br> >I have created a Jupyter Notebook to explain my thinking on what<br> >importlib.resources() should be (at least initially).<br> <br> Just a few thoughts based on a review of two projects' use of pkg_resou= rces.<br> +1 on getting *something* into Python 3.6.<br> <br> Module API vs package API.=C2=A0 Doesn't pkg_resources actually support= something<br> similar to both, with the module function providing a convenience API?</blo= ckquote><div><br></div><div>Yes, but that doesn't sway me. This isn'= ;t a "pkg_resources++" but a "make reading data from a packa= ge make sense in a modern import world". IOW I'm purposefully not = using pkg_resources as a template but simply as a motivating factor.</div><= div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8e= x;border-left:1px #ccc solid;padding-left:1ex">=C2=A0 I like<br> that a lot because the convenience API is so darn... convenient!=C2=A0 You = just<br> give it the Python dotted-path and the resource and it does the rest.<br></= blockquote><div><br></div><div>I don't see how that's any different= than the other approach since you're still providing the exact same da= ta; no more, no less.</div><div>=C2=A0</div><blockquote class=3D"gmail_quot= e" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Generally I don't care about caching the results of the search; these c= alls<br> are almost never in performance critical code.<br></blockquote><div><br></d= iv><div>Unfortunately for you the poll liked the other approach and TOOWTDI= . So either convince me that resources.read_bytes(pkg, path) is better than= resources(pkg).read_bytes(path) or consider the bike shed painted. :)</div= ><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .= 8ex;border-left:1px #ccc solid;padding-left:1ex"> <br> resource_filename().=C2=A0 Doesn't pkg_resources already have a strateg= y for the<br> temporary file that sometimes has to be created?</blockquote><div><br></div= ><div>Yes and I don't like it. :) Basically you either create an instan= ce or implicitly use a global instance of a class that stores the reference= s and registers with atexit a cleanup function to be executed.</div><div>= =C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo= rder-left:1px #ccc solid;padding-left:1ex">=C2=A0 Maybe it doesn't work= so<br> well on some platforms (I've never noticed a problem on *nix).=C2=A0 A = context<br> manager as proposed seems like the most reasonable approach.=C2=A0 We defin= itely<br> need this API though.=C2=A0 I see plenty of examples where e.g. test data f= iles<br> have to be shutil.copy()'d, passed to subprocess command line arguments= , etc.<br></blockquote><div><br></div><div>OK, between you and Donald sayin= g you have real needs for the API you can rest assured that it will be in t= he initial version, especially since I already coded up the tempfile implem= entation.</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"= margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <br> read_bytes().=C2=A0 Thank you for the truth in advertising!=C2=A0 IIRC in P= ython 3,<br> pkg_resource.resource_string() actually returns bytes. from-import-as to th= e<br> rescue.=C2=A0 An actual resource_string() would have to accept an encoding = argument<br> (as would any resource-based open() method).<br></blockquote><div><br></div= ><div>Yes, which is why I don't think it's worth it to provide a re= source_strinng() since calling decode isn't difficult (and is something= you must know in Python 3).</div><div>=C2=A0</div><blockquote class=3D"gma= il_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-lef= t:1ex"> <br> resource_stream().=C2=A0 IIRC, the pkg_resource's version is not a cont= ext manager<br> so it has to be closed explicitly (or wrapped in contextlib.closing()).=C2= =A0 We<br> can do better.<br></blockquote><div><br></div><div>I'm not convinced it= 's necessary to provide an equivalent open() yet; if you have an API th= at requires a file-like object then io.BytesIO to the rescue for read_bytes= (). There is nothing tricky to get right like with a file path that may or = may not be backed by a temporary file. This is a somewhat low-level API and= if people want to provide convenience wrappers that's fine but I don&#= 39;t want to start guessing at needs beyond core APIs or ones that are hard= to get right and allow for composability to higher APIs like file-like obj= ects which others can handle.</div><div>=C2=A0</div><blockquote class=3D"gm= ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-le= ft:1ex"> <br> I do have one use of resource_listdir() which is used to find importable<br= > plugin modules at runtime.=C2=A0 It's handy.<br></blockquote><div><br><= /div><div>I'm going to punt on this for as long as possible because it&= #39;s asking for trouble to get right. For example, if I do resources(pkg).= listdir(), then I will end up returning relative paths, but if you disassoc= iate those paths from pkg then you have lost proper context. You could retu= rn tuples of (pkg, relative_path), but that just doesn't seem satisfact= ory either. I'm just not convinced yet it is needed enough to support (= at least initially).</div><div><br></div><div>-Brett</div><div>=C2=A0</div>= <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <br> That's all for now.<br> -Barry<br> _______________________________________________<br> Import-SIG mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">Import-SIG@pytho= n.org</a><br> <a href=3D"https://mail.python.org/mailman/listinfo/import-sig" rel=3D"nore= ferrer" target=3D"_blank">https://mail.python.org/mailman/listinfo/import-s= ig</a><br> </blockquote></div></div> --94eb2c0550ea83f9a505254f17bd-- --===============6700514378526190725== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Import-SIG mailing list [email protected] https://mail.python.org/mailman/listinfo/import-sig --===============6700514378526190725==--