Re: Proposed design for importlib.resources()
Brett Cannon <[email protected]> Tue, 24 Nov 2015 23:25:50 +0000
| Newsgroups | gmane.comp.python.import |
|---|---|
| Message-ID | <CAP1=2W4QWAF1HvypoZhi3Yk2CSjJ0iifheU_VTCpB=Q8=FMJCw@mail.gmail.com> |
--===============4758308980202471935== Content-Type: multipart/alternative; boundary=001a114d6f02312811052551aa9f --001a114d6f02312811052551aa9f Content-Type: text/plain; charset=UTF-8 On Tue, 24 Nov 2015 at 14:13 Barry Warsaw <[email protected]> wrote: > On Nov 24, 2015, at 08:21 PM, Brett Cannon wrote: > > >> Module API vs package API. Doesn't pkg_resources actually support > >> something similar to both, with the module function providing a > convenience > >> API? > > > >Yes, but that doesn't sway me. This isn't a "pkg_resources++" but a "make > >reading data from a package make sense in a modern import world". IOW I'm > >purposefully not using pkg_resources as a template but simply as a > >motivating factor. > > You're not taking into account a migration path for existing users of > pkg_resources. If you make it difficult to convert, then people are much > less > likely to do it for existing code, despite the ability to remove a > dependency. > It's one of those situations where it's balancing future code with migrating old code. I'm doing this to solve the problem of importlib lacking any standardized way to get at resources in a package, not to migrate pkg_resources users who want to eliminate that dependency (although that would be a perk). > > If my existing code already has > > from pkg_resources import resource_string as resource_bytes > > all I'd need to do is change this one line to > > from importlib.resource import read_bytes as resource_bytes > > and I'm done. If I need to support multiple versions of Python, I can even > do: > > try: > from importlib.resource import read_bytes as resource_bytes > except ImportError: > from pkg_resources import resource_string as resource_bytes > > Without this API, it's much more difficult for me to convert my existing > code, > either incrementally or whole-hog, because now I have to either add that > convenience function myself (and import it everywhere) or rewrite all my > call > sites. Why bother? > Nothing is stopping people from writing their own pkg_resources compatibility layer. Hell, I will promise to create shim_resources or something and put it on PyPI for those that want a really simple migration path. But the base API that goes into the stdlib and will need to be supported forever doesn't need to go the way of compatibility if its going to feel out of place in importlib (which the pkg_resources API will, especially if we put a method on loaders to get a resource loader). > > >Unfortunately for you the poll liked the other approach and TOOWTDI. So > >either convince me that resources.read_bytes(pkg, path) is better than > >resources(pkg).read_bytes(path) or consider the bike shed painted. :) > > It's not better or worse, it's just different. As pkg_resources has > shown, it > doesn't have to be either-or. > > I never saw the poll since I don't pay attention to Google+. Which is why I also linked to it on Twitter. :) I actually tried to do it on twitter initially but guess whose poll support restricts option lengths so much you can't type a method call out. :p How representative are those 59 votes of the current pkg_resource users and potential future users of this API? If I had seen the poll I would have complained that it didn't give me a chance to choose both APIs <wink>. >I'm not convinced it's necessary to provide an equivalent open() yet; Right, I'm not necessarily advocating for it, just describing what it would have to do if it were there. It's something I occasionally wish I had, but all the building blocks are there to invent it when needed. >> I do have one use of resource_listdir() which is used to find importable >> plugin modules at runtime. It's handy. > >I'm going to punt on this for as long as possible because it's asking for >trouble to get right. For example, if I do resources(pkg).listdir(), then I >will end up returning relative paths, but if you disassociate those paths >from pkg then you have lost proper context. You could return tuples of >(pkg, relative_path), but that just doesn't seem satisfactory either. I'm >just not convinced yet it is needed enough to support (at least initially). It's a tougher API to recreate from the building blocks, so it would be nice not to have to reinvent the wheel everywhere, but it's also a much less common API. I'm not at all worried about the disassociation problem, since os.listdir() gives you relative paths anyway so it's a familiar behavior. Yeah, I realize it's something you can't make from scratch, but I'm still going to avoid it while I can because as soon as this goes in then people are going to want a similar API for discovering modules in a package and would abuse this API if they don't get the other API. -brett Cheers, -Barry _______________________________________________ Import-SIG mailing list [email protected] https <https://mail.python.org/mailman/listinfo/import-sig>:// <https://mail.python.org/mailman/listinfo/import-sig>mail.python.org <https://mail.python.org/mailman/listinfo/import-sig>/mailman/ <https://mail.python.org/mailman/listinfo/import-sig>listinfo <https://mail.python.org/mailman/listinfo/import-sig>/ <https://mail.python.org/mailman/listinfo/import-sig>import-sig <https://mail.python.org/mailman/listinfo/import-sig> --001a114d6f02312811052551aa9f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_quote"><div dir=3D"ltr">On Tue, 24 Nov= 2015 at 14:13 Barry Warsaw <<a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a>> wrote:<br></div><blockquote class=3D"g= mail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-l= eft:1ex">On Nov 24, 2015, at 08:21 PM, Brett Cannon wrote:<br> <br> >> Module API vs package API.=C2=A0 Doesn't pkg_resources actuall= y support<br> >> something similar to both, with the module function providing a co= nvenience<br> >> API?<br> ><br> >Yes, but that doesn't sway me. This isn't a "pkg_resources= ++" but a "make<br> >reading data from a package make sense in a modern import world". = IOW I'm<br> >purposefully not using pkg_resources as a template but simply as a<br> >motivating factor.<br> <br> You're not taking into account a migration path for existing users of<b= r> pkg_resources.=C2=A0 If you make it difficult to convert, then people are m= uch less<br> likely to do it for existing code, despite the ability to remove a dependen= cy.<br></blockquote><div><br></div></div></div><p dir=3D"ltr">It's one = of those situations where it's balancing future code with migrating old= code. I'm doing this to solve the problem of importlib lacking any sta= ndardized way to get at resources in a package, not to migrate pkg_resource= s users who want to eliminate that dependency (although that would be a per= k).</p> <div dir=3D"ltr"><div class=3D"gmail_quote"><div>=C2=A0</div><blockquote cl= ass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;p= adding-left:1ex"> <br> If my existing code already has<br> <br> =C2=A0 =C2=A0 from pkg_resources import resource_string as resource_bytes<b= r> <br> all I'd need to do is change this one line to<br> <br> =C2=A0 =C2=A0 from importlib.resource import read_bytes as resource_bytes<b= r> <br> and I'm done.=C2=A0 If I need to support multiple versions of Python, I= can even<br> do:<br> <br> =C2=A0 =C2=A0 try:<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 from importlib.resource import read_bytes as re= source_bytes<br> =C2=A0 =C2=A0 except ImportError:<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 from pkg_resources import resource_string as re= source_bytes<br> <br> Without this API, it's much more difficult for me to convert my existin= g code,<br> either incrementally or whole-hog, because now I have to either add that<br= > convenience function myself (and import it everywhere) or rewrite all my ca= ll<br> sites.=C2=A0 Why bother?<br></blockquote><div><br></div></div></div><p dir= =3D"ltr">Nothing is stopping people from writing their own pkg_resources co= mpatibility layer. Hell, I will promise to create shim_resources or somethi= ng and put it on PyPI for those that want a really simple migration path. B= ut the base API that goes into the stdlib and will need to be supported for= ever doesn't need to go the way of compatibility if its going to feel o= ut of place in importlib (which the pkg_resources API will, especially if w= e put a method on loaders to get a resource loader).</p> <div dir=3D"ltr"><div class=3D"gmail_quote"><div>=C2=A0</div><blockquote cl= ass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;p= adding-left:1ex"> <br> >Unfortunately for you the poll liked the other approach and TOOWTDI. So= <br> >either convince me that resources.read_bytes(pkg, path) is better than<= br> >resources(pkg).read_bytes(path) or consider the bike shed painted. :)<b= r> <br> It's not better or worse, it's just different.=C2=A0 As pkg_resourc= es has shown, it<br> doesn't have to be either-or.<br> <br> I never saw the poll since I don't pay attention to Google+.</blockquot= e><div><br></div></div></div><p dir=3D"ltr">Which is why I also linked to i= t on Twitter. :) I actually tried to do it on twitter initially but guess w= hose poll support restricts option lengths so much you can't type a met= hod call out. :p</p> <p dir=3D"ltr">=C2=A0</p> <blockquote><p dir=3D"ltr">=C2=A0 How<br> representative are those 59 votes of the current pkg_resource users and<br> potential future users of this API?=C2=A0 If I had seen the poll I would ha= ve<br> complained that it didn't give me a chance to choose both APIs <wink= >.</p> <p dir=3D"ltr">>I'm not convinced it's necessary to provide an e= quivalent open() yet;</p> <p dir=3D"ltr">Right, I'm not necessarily advocating for it, just descr= ibing what it would<br> have to do if it were there.=C2=A0 It's something I occasionally wish I= had, but<br> all the building blocks are there to invent it when needed.</p> <p dir=3D"ltr">>> I do have one use of resource_listdir() which is us= ed to find importable<br> >> plugin modules at runtime.=C2=A0 It's handy.<br> ><br> >I'm going to punt on this for as long as possible because it's = asking for<br> >trouble to get right. For example, if I do resources(pkg).listdir(), th= en I<br> >will end up returning relative paths, but if you disassociate those pat= hs<br> >from pkg then you have lost proper context. You could return tuples of<= br> >(pkg, relative_path), but that just doesn't seem satisfactory eithe= r. I'm<br> >just not convinced yet it is needed enough to support (at least initial= ly).</p> <p dir=3D"ltr">It's a tougher API to recreate from the building blocks,= so it would be nice<br> not to have to reinvent the wheel everywhere, but it's also a much less= common<br> API.=C2=A0 I'm not at all worried about the disassociation problem, sin= ce<br> os.listdir() gives you relative paths anyway so it's a familiar behavio= r.</p> </blockquote> <p dir=3D"ltr"></p> <p dir=3D"ltr">Yeah, I realize it's something you can't make from s= cratch, but I'm still going to avoid it while I can because as soon as = this goes in then people are going to want a similar API for discovering mo= dules in a package and would abuse this API if they don't get the other= API.</p> <p dir=3D"ltr">-brett<br> </p> <blockquote><p dir=3D"ltr"></p> <p dir=3D"ltr">Cheers,<br> -Barry<br> _______________________________________________<br> Import-SIG mailing list<br> <a href=3D"mailto:[email protected]">[email protected]</a><br> <a href=3D"https://mail.python.org/mailman/listinfo/import-sig">https</a><a= href=3D"https://mail.python.org/mailman/listinfo/import-sig">://</a><a hre= f=3D"https://mail.python.org/mailman/listinfo/import-sig">mail.python.org</= a><a href=3D"https://mail.python.org/mailman/listinfo/import-sig">/mailman/= </a><a href=3D"https://mail.python.org/mailman/listinfo/import-sig">listinf= o</a><a href=3D"https://mail.python.org/mailman/listinfo/import-sig">/</a><= a href=3D"https://mail.python.org/mailman/listinfo/import-sig">import-sig</= a><br> </p> </blockquote> <p dir=3D"ltr"><br> </p> --001a114d6f02312811052551aa9f-- --===============4758308980202471935== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Import-SIG mailing list [email protected] https://mail.python.org/mailman/listinfo/import-sig --===============4758308980202471935==--