Re: Proposed design for importlib.resources()
Brett Cannon <[email protected]> Wed, 25 Nov 2015 00:28:18 +0000
| Newsgroups | gmane.comp.python.import |
|---|---|
| Message-ID | <CAP1=2W64fsxTY2Ek4=C66rbw-Pd5VKLqoSfwn-wJVaKAKHDs-g@mail.gmail.com> |
--===============0435531213968532824== Content-Type: multipart/alternative; boundary=001a114d6f027e5f45052552891f --001a114d6f027e5f45052552891f Content-Type: text/plain; charset=UTF-8 If we make it e.g., __loader__.resources().read_bytes(path) then I may be more amenable to creating a importlib.resources module with the bastardized pkg_resources API. Going to have to think about it, though. On Tue, 24 Nov 2015, 16:25 Brett Cannon <[email protected]> wrote: > On Tue, 24 Nov 2015 at 14:13 Barry Warsaw <[email protected]> wrote: > >> On Nov 24, 2015, at 08:21 PM, Brett Cannon wrote: >> >> >> Module API vs package API. Doesn't pkg_resources actually support >> >> something similar to both, with the module function providing a >> convenience >> >> API? >> > >> >Yes, but that doesn't sway me. This isn't a "pkg_resources++" but a "make >> >reading data from a package make sense in a modern import world". IOW I'm >> >purposefully not using pkg_resources as a template but simply as a >> >motivating factor. >> >> You're not taking into account a migration path for existing users of >> pkg_resources. If you make it difficult to convert, then people are much >> less >> likely to do it for existing code, despite the ability to remove a >> dependency. >> > > It's one of those situations where it's balancing future code with > migrating old code. I'm doing this to solve the problem of importlib > lacking any standardized way to get at resources in a package, not to > migrate pkg_resources users who want to eliminate that dependency (although > that would be a perk). > > >> >> If my existing code already has >> >> from pkg_resources import resource_string as resource_bytes >> >> all I'd need to do is change this one line to >> >> from importlib.resource import read_bytes as resource_bytes >> >> and I'm done. If I need to support multiple versions of Python, I can >> even >> do: >> >> try: >> from importlib.resource import read_bytes as resource_bytes >> except ImportError: >> from pkg_resources import resource_string as resource_bytes >> >> Without this API, it's much more difficult for me to convert my existing >> code, >> either incrementally or whole-hog, because now I have to either add that >> convenience function myself (and import it everywhere) or rewrite all my >> call >> sites. Why bother? >> > > Nothing is stopping people from writing their own pkg_resources > compatibility layer. Hell, I will promise to create shim_resources or > something and put it on PyPI for those that want a really simple migration > path. But the base API that goes into the stdlib and will need to be > supported forever doesn't need to go the way of compatibility if its going > to feel out of place in importlib (which the pkg_resources API will, > especially if we put a method on loaders to get a resource loader). > > >> >> >Unfortunately for you the poll liked the other approach and TOOWTDI. So >> >either convince me that resources.read_bytes(pkg, path) is better than >> >resources(pkg).read_bytes(path) or consider the bike shed painted. :) >> >> It's not better or worse, it's just different. As pkg_resources has >> shown, it >> doesn't have to be either-or. >> >> I never saw the poll since I don't pay attention to Google+. > > > Which is why I also linked to it on Twitter. :) I actually tried to do it > on twitter initially but guess whose poll support restricts option lengths > so much you can't type a method call out. :p > > > > How > representative are those 59 votes of the current pkg_resource users and > potential future users of this API? If I had seen the poll I would have > complained that it didn't give me a chance to choose both APIs <wink>. > > >I'm not convinced it's necessary to provide an equivalent open() yet; > > Right, I'm not necessarily advocating for it, just describing what it would > have to do if it were there. It's something I occasionally wish I had, but > all the building blocks are there to invent it when needed. > > >> I do have one use of resource_listdir() which is used to find importable > >> plugin modules at runtime. It's handy. > > > >I'm going to punt on this for as long as possible because it's asking for > >trouble to get right. For example, if I do resources(pkg).listdir(), then > I > >will end up returning relative paths, but if you disassociate those paths > >from pkg then you have lost proper context. You could return tuples of > >(pkg, relative_path), but that just doesn't seem satisfactory either. I'm > >just not convinced yet it is needed enough to support (at least > initially). > > It's a tougher API to recreate from the building blocks, so it would be > nice > not to have to reinvent the wheel everywhere, but it's also a much less > common > API. I'm not at all worried about the disassociation problem, since > os.listdir() gives you relative paths anyway so it's a familiar behavior. > > Yeah, I realize it's something you can't make from scratch, but I'm still > going to avoid it while I can because as soon as this goes in then people > are going to want a similar API for discovering modules in a package and > would abuse this API if they don't get the other API. > > -brett > > Cheers, > -Barry > _______________________________________________ > Import-SIG mailing list > [email protected] > https <https://mail.python.org/mailman/listinfo/import-sig>:// > <https://mail.python.org/mailman/listinfo/import-sig>mail.python.org > <https://mail.python.org/mailman/listinfo/import-sig>/mailman/ > <https://mail.python.org/mailman/listinfo/import-sig>listinfo > <https://mail.python.org/mailman/listinfo/import-sig>/ > <https://mail.python.org/mailman/listinfo/import-sig>import-sig > <https://mail.python.org/mailman/listinfo/import-sig> > > > --001a114d6f027e5f45052552891f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <p dir=3D"ltr">If we make it e.g.,=C2=A0 __loader__.resources().read_bytes(= path) then I may be more amenable to creating a importlib.resources module = with the bastardized pkg_resources API. Going to have to think about it, th= ough.</p> <br><div class=3D"gmail_quote"><div dir=3D"ltr">On Tue, 24 Nov 2015, 16:25= =C2=A0Brett Cannon <<a href=3D"mailto:[email protected]">[email protected]= </a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:= 0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><d= iv class=3D"gmail_quote"><div dir=3D"ltr">On Tue, 24 Nov 2015 at 14:13 Barr= y Warsaw <<a href=3D"mailto:[email protected]" target=3D"_blank">barry@py= thon.org</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D= "margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Nov 24, = 2015, at 08:21 PM, Brett Cannon wrote:<br> <br> >> Module API vs package API.=C2=A0 Doesn't pkg_resources actuall= y support<br> >> something similar to both, with the module function providing a co= nvenience<br> >> API?<br> ><br> >Yes, but that doesn't sway me. This isn't a "pkg_resources= ++" but a "make<br> >reading data from a package make sense in a modern import world". = IOW I'm<br> >purposefully not using pkg_resources as a template but simply as a<br> >motivating factor.<br> <br> You're not taking into account a migration path for existing users of<b= r> pkg_resources.=C2=A0 If you make it difficult to convert, then people are m= uch less<br> likely to do it for existing code, despite the ability to remove a dependen= cy.<br></blockquote><div><br></div></div></div><p dir=3D"ltr">It's one = of those situations where it's balancing future code with migrating old= code. I'm doing this to solve the problem of importlib lacking any sta= ndardized way to get at resources in a package, not to migrate pkg_resource= s users who want to eliminate that dependency (although that would be a per= k).</p> <div dir=3D"ltr"><div class=3D"gmail_quote"><div>=C2=A0</div><blockquote cl= ass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;p= adding-left:1ex"> <br> If my existing code already has<br> <br> =C2=A0 =C2=A0 from pkg_resources import resource_string as resource_bytes<b= r> <br> all I'd need to do is change this one line to<br> <br> =C2=A0 =C2=A0 from importlib.resource import read_bytes as resource_bytes<b= r> <br> and I'm done.=C2=A0 If I need to support multiple versions of Python, I= can even<br> do:<br> <br> =C2=A0 =C2=A0 try:<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 from importlib.resource import read_bytes as re= source_bytes<br> =C2=A0 =C2=A0 except ImportError:<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 from pkg_resources import resource_string as re= source_bytes<br> <br> Without this API, it's much more difficult for me to convert my existin= g code,<br> either incrementally or whole-hog, because now I have to either add that<br= > convenience function myself (and import it everywhere) or rewrite all my ca= ll<br> sites.=C2=A0 Why bother?<br></blockquote><div><br></div></div></div><p dir= =3D"ltr">Nothing is stopping people from writing their own pkg_resources co= mpatibility layer. Hell, I will promise to create shim_resources or somethi= ng and put it on PyPI for those that want a really simple migration path. B= ut the base API that goes into the stdlib and will need to be supported for= ever doesn't need to go the way of compatibility if its going to feel o= ut of place in importlib (which the pkg_resources API will, especially if w= e put a method on loaders to get a resource loader).</p> <div dir=3D"ltr"><div class=3D"gmail_quote"><div>=C2=A0</div><blockquote cl= ass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;p= adding-left:1ex"> <br> >Unfortunately for you the poll liked the other approach and TOOWTDI. So= <br> >either convince me that resources.read_bytes(pkg, path) is better than<= br> >resources(pkg).read_bytes(path) or consider the bike shed painted. :)<b= r> <br> It's not better or worse, it's just different.=C2=A0 As pkg_resourc= es has shown, it<br> doesn't have to be either-or.<br> <br> I never saw the poll since I don't pay attention to Google+.</blockquot= e><div><br></div></div></div><p dir=3D"ltr">Which is why I also linked to i= t on Twitter. :) I actually tried to do it on twitter initially but guess w= hose poll support restricts option lengths so much you can't type a met= hod call out. :p</p> <p dir=3D"ltr">=C2=A0</p> <blockquote><p dir=3D"ltr">=C2=A0 How<br> representative are those 59 votes of the current pkg_resource users and<br> potential future users of this API?=C2=A0 If I had seen the poll I would ha= ve<br> complained that it didn't give me a chance to choose both APIs <wink= >.</p> <p dir=3D"ltr">>I'm not convinced it's necessary to provide an e= quivalent open() yet;</p> <p dir=3D"ltr">Right, I'm not necessarily advocating for it, just descr= ibing what it would<br> have to do if it were there.=C2=A0 It's something I occasionally wish I= had, but<br> all the building blocks are there to invent it when needed.</p> <p dir=3D"ltr">>> I do have one use of resource_listdir() which is us= ed to find importable<br> >> plugin modules at runtime.=C2=A0 It's handy.<br> ><br> >I'm going to punt on this for as long as possible because it's = asking for<br> >trouble to get right. For example, if I do resources(pkg).listdir(), th= en I<br> >will end up returning relative paths, but if you disassociate those pat= hs<br> >from pkg then you have lost proper context. You could return tuples of<= br> >(pkg, relative_path), but that just doesn't seem satisfactory eithe= r. I'm<br> >just not convinced yet it is needed enough to support (at least initial= ly).</p> <p dir=3D"ltr">It's a tougher API to recreate from the building blocks,= so it would be nice<br> not to have to reinvent the wheel everywhere, but it's also a much less= common<br> API.=C2=A0 I'm not at all worried about the disassociation problem, sin= ce<br> os.listdir() gives you relative paths anyway so it's a familiar behavio= r.</p> </blockquote> <p dir=3D"ltr"></p> <p dir=3D"ltr">Yeah, I realize it's something you can't make from s= cratch, but I'm still going to avoid it while I can because as soon as = this goes in then people are going to want a similar API for discovering mo= dules in a package and would abuse this API if they don't get the other= API.</p> <p dir=3D"ltr">-brett<br> </p> <blockquote><p dir=3D"ltr"></p> <p dir=3D"ltr">Cheers,<br> -Barry<br> _______________________________________________<br> Import-SIG mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">Import-SIG@pytho= n.org</a><br> <a href=3D"https://mail.python.org/mailman/listinfo/import-sig" target=3D"_= blank">https</a><a href=3D"https://mail.python.org/mailman/listinfo/import-= sig" target=3D"_blank">://</a><a href=3D"https://mail.python.org/mailman/li= stinfo/import-sig" target=3D"_blank">mail.python.org</a><a href=3D"https://= mail.python.org/mailman/listinfo/import-sig" target=3D"_blank">/mailman/</a= ><a href=3D"https://mail.python.org/mailman/listinfo/import-sig" target=3D"= _blank">listinfo</a><a href=3D"https://mail.python.org/mailman/listinfo/imp= ort-sig" target=3D"_blank">/</a><a href=3D"https://mail.python.org/mailman/= listinfo/import-sig" target=3D"_blank">import-sig</a><br> </p> </blockquote> <p dir=3D"ltr"><br> </p> </blockquote></div> --001a114d6f027e5f45052552891f-- --===============0435531213968532824== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Import-SIG mailing list [email protected] https://mail.python.org/mailman/listinfo/import-sig --===============0435531213968532824==--