Re: Any practical reason type(environ) must be dict (not subclass)?
Alan Kennedy <[email protected]> Thu, 24 Mar 2016 16:09:20 +0000
| Newsgroups | gmane.comp.python.web |
|---|---|
| Message-ID | <CAMte6h=j8yA7eDBpF-HMp3sUU__8NMDKHUpHeigGKXtAfw-aTg@mail.gmail.com> |
--===============3792746054553973495== Content-Type: multipart/alternative; boundary=001a113c319e461997052ecdab1d --001a113c319e461997052ecdab1d Content-Type: text/plain; charset=UTF-8 I don't see this relevant message in your references. https://mail.python.org/pipermail/web-sig/2004-September/000749.html Perhaps that, and following messages, might shed more light? On Thu, Mar 24, 2016 at 3:18 PM, Jason Madden <[email protected]> wrote: > Hi all, > > > Is there any practical reason that the type of the `environ` object must > be exactly `dict`, as specified in PEP3333? > > I'm asking because it was recently pointed out that gevent's WSGI server > can sometimes print `environ` (on certain error cases), but that can lead > to sensitive information being kept in the server's logs (e.g., > HTTP_AUTHORIZATION, HTTP_COOKIE, maybe other things). The simplest and most > flexible way to prevent this from happening, not just inadvertently within > gevent itself but also for client applications, I thought, was to have > `environ` be a subclass of `dict` with a customized `__repr__` (much like > WebOb does for MultiDict, and repoze.who does for Identity, both for > similar reasons). > > Unfortunately, when I implemented that in [0], I discovered that > `wsgiref.validator` asserts that type(environ) is dict. I looked up the > PEP, and sure enough, PEP 3333 states that environ "must be a builtin > Python dictionary (not a subclass, UserDict or other dictionary > emulation)." [1] > > Background/History > ================== > > That seemed overly restrictive to me, so I tried to backtrack the history > of that language in hopes of discovering the rationale. > > - It was present in the predecessor of PEP 3333, PEP 0333, in the first > version committed to the repository in August 2004. [2] > - Prior to that, it was in both drafts of what would become PEP 0333 > posted to this mailing list, again from August 2004: [3], [4]. > - The ancestor of those drafts, the "Python Web Container Interface v1.0" > was posted in December of 2003 with somewhat less restrictive language: > "the environ object *must* be a Python dictionary....The rationale for > requiring a dictionary is to maximize portability > between containers" [5]. > > Now, the discussion on that earliest draft in [5] specifically brought up > using other types that implement all the methods of a dictionary, like > UserDict.DictMixin [6]. The last post on the subject in that thread seemed > to be leaning towards accepting non-dict objects, at least if they were > good enough [7]. > > By the time the draft became recognizable as the precursor to PEP 0333 in > [3], the very strict language we have now was in place. That draft, > however, specifically stated that it was intended to be compatible with > Python 1.5.2. In Python 1.5.2, it wasn't possible to subclass the builtin > dict, so imitations, like UserDict.DictMixin, were necessarily imprecise. > This was later changed to the much-maligned Python 2.2.2 release [8]; > Python 2.2 added the ability to subclass dict, but the language wasn't > changed. > > Today > ===== > > Given that today, we can subclass dict with full fidelity, is there still > any practical reason not to be able to do so? I'm probably OK with gevent > violating the letter of the spec in this regard, so long as there are no > practical consequences. I was able to think of two possible objections, but > both can be solved: > > - Pickling the custom `environ` type and then loading it in another > process might not work if the class is not available. I can imagine this > coming up with Celery, for example. This is easily fixed by adding an > appropriate `__reduce_ex__` implementation. > > - Code somewhere relies on `if type(some_object) is dict:` (where > `environ` became `some_object`, presumably through several levels of > calls), instead of `isinstance(some_object, dict)` or > `isinstance(some_object, collections.MutableMapping)`. The solution here is > simply to not do that :) Pylint, among other linters, produces warnings if > you do. > > Can anyone think of any other practical reasons I've overlooked? Is this > just a horrible idea for other reasons? > > I appreciate any discussion! > > Thanks, > Jason > > [0] https://github.com/gevent/gevent/compare/secure-environ > [1] https://www.python.org/dev/peps/pep-3333/#specification-details > [2] > https://github.com/python/peps/commit/d5864f018f58a35fa787492e6763e382f98b923c#diff-ff370d50af3db062b015d1ef85935779 > [3] https://mail.python.org/pipermail/web-sig/2004-August/000518.html > [4] https://mail.python.org/pipermail/web-sig/2004-August/000562.html > [5] https://mail.python.org/pipermail/web-sig/2003-December/000394.html > [7] https://mail.python.org/pipermail/web-sig/2003-December/000401.html > [8] https://mail.python.org/pipermail/web-sig/2004-August/000565.html > > _______________________________________________ > Web-SIG mailing list > [email protected] > Web SIG: http://www.python.org/sigs/web-sig > Unsubscribe: > https://mail.python.org/mailman/options/web-sig/alan%40xhaus.com > --001a113c319e461997052ecdab1d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>I don't see this relevant message in your referen= ces.<br><br><a href=3D"https://mail.python.org/pipermail/web-sig/2004-Septe= mber/000749.html" target=3D"_blank">https://mail.python.org/pipermail/web-s= ig/2004-September/000749.html</a><br><br></div>Perhaps that, and following = messages, might shed more light?</div><div class=3D"gmail_extra"><br><div c= lass=3D"gmail_quote">On Thu, Mar 24, 2016 at 3:18 PM, Jason Madden <span di= r=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D"_bl= ank">[email protected]</a>></span> wrote:<br><blockquote clas= s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad= ding-left:1ex">Hi all,<br> <br> <br> Is there any practical reason that the type of the `environ` object must be= exactly `dict`, as specified in PEP3333?<br> <br> I'm asking because it was recently pointed out that gevent's WSGI s= erver can sometimes print `environ` (on certain error cases), but that can = lead to sensitive information being kept in the server's logs (e.g., HT= TP_AUTHORIZATION, HTTP_COOKIE, maybe other things). The simplest and most f= lexible way to prevent this from happening, not just inadvertently within g= event itself but also for client applications, I thought, was to have `envi= ron` be a subclass of `dict` with a customized `__repr__` (much like WebOb = does for MultiDict, and repoze.who does for Identity, both for similar reas= ons).<br> <br> Unfortunately, when I implemented that in [0], I discovered that `wsgiref.v= alidator` asserts that type(environ) is dict. I looked up the PEP, and sure= enough, PEP 3333 states that environ "must be a builtin Python dictio= nary (not a subclass, UserDict or other dictionary emulation)." [1]<br= > <br> Background/History<br> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br> <br> That seemed overly restrictive to me, so I tried to backtrack the history o= f that language in hopes of discovering the rationale.<br> <br> - It was present in the predecessor of PEP 3333, PEP 0333, in the first ver= sion committed to the repository in August 2004. [2]<br> - Prior to that, it was in both drafts of what would become PEP 0333 posted= to this mailing list, again from August 2004: [3], [4].<br> - The ancestor of those drafts, the "Python Web Container Interface v1= .0" was posted in December of 2003 with somewhat less restrictive lang= uage: "the environ object *must* be a Python dictionary....The rationa= le for requiring a dictionary is to maximize portability<br> between containers" [5].<br> <br> Now, the discussion on that earliest draft in [5] specifically brought up u= sing other types that implement all the methods of a dictionary, like UserD= ict.DictMixin [6]. The last post on the subject in that thread seemed to be= leaning towards accepting non-dict objects, at least if they were good eno= ugh [7].<br> <br> By the time the draft became recognizable as the precursor to PEP 0333 in [= 3], the very strict language we have now was in place. That draft, however,= specifically stated that it was intended to be compatible with Python 1.5.= 2. In Python 1.5.2, it wasn't possible to subclass the builtin dict, so= imitations, like UserDict.DictMixin, were necessarily imprecise. This was = later changed to the much-maligned Python 2.2.2 release [8]; Python 2.2 add= ed the ability to subclass dict, but the language wasn't changed.<br> <br> Today<br> =3D=3D=3D=3D=3D<br> <br> Given that today, we can subclass dict with full fidelity, is there still a= ny practical reason not to be able to do so? I'm probably OK with geven= t violating the letter of the spec in this regard, so long as there are no = practical consequences. I was able to think of two possible objections, but= both can be solved:<br> <br> - Pickling the custom `environ` type and then loading it in another process= might not work if the class is not available. I can imagine this coming up= with Celery, for example. This is easily fixed by adding an appropriate `_= _reduce_ex__` implementation.<br> <br> - Code somewhere relies on `if type(some_object) is dict:` (where `environ`= became `some_object`, presumably through several levels of calls), instead= of `isinstance(some_object, dict)` or `isinstance(some_object, collections= .MutableMapping)`. The solution here is simply to not do that :) Pylint, am= ong other linters, produces warnings if you do.<br> <br> Can anyone think of any other practical reasons I've overlooked? Is thi= s just a horrible idea for other reasons?<br> <br> I appreciate any discussion!<br> <br> Thanks,<br> Jason<br> <br> [0] <a href=3D"https://github.com/gevent/gevent/compare/secure-environ" rel= =3D"noreferrer" target=3D"_blank">https://github.com/gevent/gevent/compare/= secure-environ</a><br> [1] <a href=3D"https://www.python.org/dev/peps/pep-3333/#specification-deta= ils" rel=3D"noreferrer" target=3D"_blank">https://www.python.org/dev/peps/p= ep-3333/#specification-details</a><br> [2] <a href=3D"https://github.com/python/peps/commit/d5864f018f58a35fa78749= 2e6763e382f98b923c#diff-ff370d50af3db062b015d1ef85935779" rel=3D"noreferrer= " target=3D"_blank">https://github.com/python/peps/commit/d5864f018f58a35fa= 787492e6763e382f98b923c#diff-ff370d50af3db062b015d1ef85935779</a><br> [3] <a href=3D"https://mail.python.org/pipermail/web-sig/2004-August/000518= .html" rel=3D"noreferrer" target=3D"_blank">https://mail.python.org/piperma= il/web-sig/2004-August/000518.html</a><br> [4] <a href=3D"https://mail.python.org/pipermail/web-sig/2004-August/000562= .html" rel=3D"noreferrer" target=3D"_blank">https://mail.python.org/piperma= il/web-sig/2004-August/000562.html</a><br> [5] <a href=3D"https://mail.python.org/pipermail/web-sig/2003-December/0003= 94.html" rel=3D"noreferrer" target=3D"_blank">https://mail.python.org/piper= mail/web-sig/2003-December/000394.html</a><br> [7] <a href=3D"https://mail.python.org/pipermail/web-sig/2003-December/0004= 01.html" rel=3D"noreferrer" target=3D"_blank">https://mail.python.org/piper= mail/web-sig/2003-December/000401.html</a><br> [8] <a href=3D"https://mail.python.org/pipermail/web-sig/2004-August/000565= .html" rel=3D"noreferrer" target=3D"_blank">https://mail.python.org/piperma= il/web-sig/2004-August/000565.html</a><br> <br> _______________________________________________<br> Web-SIG mailing list<br> <a href=3D"mailto:[email protected]">[email protected]</a><br> Web SIG: <a href=3D"http://www.python.org/sigs/web-sig" rel=3D"noreferrer" = target=3D"_blank">http://www.python.org/sigs/web-sig</a><br> Unsubscribe: <a href=3D"https://mail.python.org/mailman/options/web-sig/ala= n%40xhaus.com" rel=3D"noreferrer" target=3D"_blank">https://mail.python.org= /mailman/options/web-sig/alan%40xhaus.com</a><br> </blockquote></div><br></div> --001a113c319e461997052ecdab1d-- --===============3792746054553973495== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Web-SIG mailing list [email protected] Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/gcpw-web-sig%40m.gmane.org --===============3792746054553973495==--