Re: internal vs. external hostname in Request.getHost9)
Ilya Skriblovsky <[email protected]> Mon, 20 Mar 2017 19:23:58 +0000
| Newsgroups | gmane.comp.python.twisted.web |
|---|---|
| Message-ID | <CAOG7vkwzKyDtxC6HUeYC2k9HjRQG1JOdRBy-SHqnPNGVjc2JqA@mail.gmail.com> |
--===============0007370320639162671== Content-Type: multipart/alternative; boundary=001a113cec02b739bf054b2e787c --001a113cec02b739bf054b2e787c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable This thread is mostly about X-Forwarded-Host & X-Forwarded-Proto because the original issue was inability of Twisted Web server to obtain it's public hostname. X-Forwarded-For is another (and probably more complex) story. Django indeed dropped support for X-Forwarded-For, but it does support X-Forwarded-Host [1] and X-Forwarded-Proto [2] on opt-in basis. I'm agree that none of the headers should be trusted by default and that opting-in should be done at Site level. -- ilya [1]: https://docs.djangoproject.com/en/1.10/ref/settings/#std:setting-USE_X_FORW= ARDED_HOST [2]: https://docs.djangoproject.com/en/1.10/ref/settings/#std:setting-SECURE_PRO= XY_SSL_HEADER =D0=BF=D0=BD, 20 =D0=BC=D0=B0=D1=80. 2017 =D0=B3. =D0=B2 21:32, Tom Most <t= [email protected]>: > On Fri, Mar 17, 2017 at 2:52 AM, Ilya Skriblovsky < > [email protected]> wrote: > > > The code calling request.URLPath(), in a given Resource, or > application, is highly unlikely to know whether it wants to honor > (x-)forwarded-for. > You are right, I haven't thought about it. > But I'm in doubt whether trusting X-Forwarded-* by default can damage > security if Twisted app is running with naked HTTP(S) port exposed withou= t > reverse proxy that handles these headers. > There are three headers: > 1. X-Forwarded-For specifying original client IP and IPs of proxies > 2. X-Forwarded-Host specifying original Host header from the client > 3. X-Forwarded-Proto specifying original client's scheme > (there is also new-style "Forwarded:" header but it is not widely used > yet, AFAIK) > > X-Forwarded-For definetly can't be trusted if comes from untrusted client > client. Fortunately we don't need it at all for generating URLs :) It wil= l > be in question when refactoring getClientIP() somewhen later. > > But can we trust X-Forwarded-Host & X-Forwarded-Proto? From the first > glance it isn't a problem since we are using them to display URLs for the > same client, so nasty client will get his nasty URLs, that's it. But if a= pp > is doing something like storing URL in DB or (more likely) sending an ema= il > with a link to another client, this would be an issue. > > > It's not safe to enable support for X-Forwarded-For and friends by > default, since you can't know how application code will use that > information and in many configurations it may be spoofed by clients. > > A proper deployment which sets these headers looks like this: > > 1. Configure your frontend reverse-proxy (nginx or whatever) to discard > incoming X-Forwarded-* headers and *set* X-Forwarded-* as appropriate. > Note that X-Forwarded-For is actually a *list* of hops[1], so if you do > this naively your server may append to the list! [2] > 2. Configure your backend services to respect exactly the headers passed > by your frontend. > > Note that this requires administrative action in two places, and that if > you don't do the frontend config it will probably pass the headers throug= h, > allowing them to be spoofed. > > A non-exhaustive list of possible bad stuff I client could use spoofing > for: > > 1. Evade IP-based access control (a reasonable defense-in-depth measure). > 2. Evade pinning of user sessions to IP addresses or subnets. > 2. Evade IP-based rate limiting, e.g. as discussed at [3]. > > It's also probably worth noting that Django used to offer support for > X-Forwarded-For and removed it[4]. X-Forwarded-* and friends just too > varied in the wild to reasonably support. > > If Twisted is to support this in any way, I think that it should be opt-i= n > support for the Forwarded header as specified in RFC 7239. This should be= a > parameter applicable to all of twisted.web.server rather than per-method > call, since it's something the administrator needs to set. > > =E2=80=94Tom > > [1]: Standardized as Forwarded in https://tools.ietf.org/html/rfc7239 > [2]: Your frontend proxy should also validate Host is a domain you contro= l > to prevent cookie theft. Plus lots of other stuff. Web security is hard, > and every default everywhere sets users up for failure. > [3]: http://django-ratelimit.readthedocs.io/en/v1.0.0/security.html > [4]: > https://www.djangoproject.com/weblog/2009/jul/28/security/#secondary-issu= e > _______________________________________________ > Twisted-web mailing list > [email protected] > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web > --001a113cec02b739bf054b2e787c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>This thread is mostly about X-Forwarded-Host & X-= Forwarded-Proto because the original issue was inability of Twisted Web ser= ver to obtain it's public hostname. X-Forwarded-For is another (and pro= bably more complex) story.</div><div><br></div><div>Django indeed dropped s= upport for X-Forwarded-For, but it does support X-Forwarded-Host [1] and X-= Forwarded-Proto [2] on opt-in basis.</div><div><br></div><div>I'm agree= that none of the headers should be trusted by default and that opting-in s= hould be done at Site level.</div><div><br></div><div>-- ilya</div><div><br= ></div><div>[1]:=C2=A0<a href=3D"https://docs.djangoproject.com/en/1.10/ref= /settings/#std:setting-USE_X_FORWARDED_HOST">https://docs.djangoproject.com= /en/1.10/ref/settings/#std:setting-USE_X_FORWARDED_HOST</a></div><div>[2]:= =C2=A0<a href=3D"https://docs.djangoproject.com/en/1.10/ref/settings/#std:s= etting-SECURE_PROXY_SSL_HEADER">https://docs.djangoproject.com/en/1.10/ref/= settings/#std:setting-SECURE_PROXY_SSL_HEADER</a></div><div><br><div class= =3D"gmail_quote"><div dir=3D"ltr">=D0=BF=D0=BD, 20 =D0=BC=D0=B0=D1=80. 2017= =D0=B3. =D0=B2 21:32, Tom Most <<a href=3D"mailto:[email protected]">to= [email protected]</a>>:<br></div><blockquote class=3D"gmail_quote" style= =3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir= =3D"ltr" class=3D"gmail_msg"><div class=3D"gmail_extra gmail_msg"><div clas= s=3D"gmail_quote gmail_msg">On Fri, Mar 17, 2017 at 2:52 AM, Ilya Skriblovs= ky <span dir=3D"ltr" class=3D"gmail_msg"><<a href=3D"mailto:ilyaskriblov= [email protected]" class=3D"gmail_msg" target=3D"_blank">ilyaskriblovsky@gmail.= com</a>></span> wrote:<br class=3D"gmail_msg"><blockquote class=3D"gmail= _quote gmail_msg" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid r= gb(204,204,204);padding-left:1ex"><div dir=3D"ltr" class=3D"gmail_msg"><spa= n class=3D"m_-6904254408352713843gmail-m_388361609629121005gmail- gmail_msg= "><div dir=3D"ltr" class=3D"m_-6904254408352713843gmail-m_38836160962912100= 5gmail-m_-525099954024514115gmail_msg gmail_msg">>=C2=A0<span style=3D"c= olor:rgb(33,33,33)" class=3D"m_-6904254408352713843gmail-m_3883616096291210= 05gmail-m_-525099954024514115gmail_msg gmail_msg">The code calling request.= URLPath(), in a given Resource, or application, is highly unlikely to know = whether it wants to honor (x-)forwarded-for.</span></div></span><div dir=3D= "ltr" class=3D"m_-6904254408352713843gmail-m_388361609629121005gmail-m_-525= 099954024514115gmail_msg gmail_msg"><div class=3D"m_-6904254408352713843gma= il-m_388361609629121005gmail-m_-525099954024514115gmail_msg gmail_msg"><spa= n style=3D"color:rgb(33,33,33)" class=3D"m_-6904254408352713843gmail-m_3883= 61609629121005gmail-m_-525099954024514115gmail_msg gmail_msg">You are right= , I haven't thought about it.</span></div><div class=3D"m_-690425440835= 2713843gmail-m_388361609629121005gmail-m_-525099954024514115gmail_msg gmail= _msg"><span style=3D"color:rgb(33,33,33)" class=3D"m_-6904254408352713843gm= ail-m_388361609629121005gmail-m_-525099954024514115gmail_msg gmail_msg">But= I'm in doubt whether trusting X-Forwarded-* by default can damage secu= rity if Twisted app is running with naked HTTP(S) port exposed without reve= rse proxy that handles these headers.</span></div><div class=3D"m_-69042544= 08352713843gmail-m_388361609629121005gmail-m_-525099954024514115gmail_msg g= mail_msg"><span style=3D"color:rgb(33,33,33)" class=3D"m_-69042544083527138= 43gmail-m_388361609629121005gmail-m_-525099954024514115gmail_msg gmail_msg"= >There are three headers:</span></div><div class=3D"m_-6904254408352713843g= mail-m_388361609629121005gmail-m_-525099954024514115gmail_msg gmail_msg"><s= pan style=3D"color:rgb(33,33,33)" class=3D"m_-6904254408352713843gmail-m_38= 8361609629121005gmail-m_-525099954024514115gmail_msg gmail_msg">1. X-Forwar= ded-For specifying original client IP and IPs of proxies</span></div><div c= lass=3D"m_-6904254408352713843gmail-m_388361609629121005gmail-m_-5250999540= 24514115gmail_msg gmail_msg"><span style=3D"color:rgb(33,33,33)" class=3D"m= _-6904254408352713843gmail-m_388361609629121005gmail-m_-525099954024514115g= mail_msg gmail_msg">2. X-Forwarded-Host specifying original Host header fro= m the client</span></div><div class=3D"m_-6904254408352713843gmail-m_388361= 609629121005gmail-m_-525099954024514115gmail_msg gmail_msg"><span style=3D"= color:rgb(33,33,33)" class=3D"m_-6904254408352713843gmail-m_388361609629121= 005gmail-m_-525099954024514115gmail_msg gmail_msg">3. X-Forwarded-Proto spe= cifying original client's scheme</span></div><div class=3D"m_-690425440= 8352713843gmail-m_388361609629121005gmail-m_-525099954024514115gmail_msg gm= ail_msg"><span style=3D"color:rgb(33,33,33)" class=3D"m_-690425440835271384= 3gmail-m_388361609629121005gmail-m_-525099954024514115gmail_msg gmail_msg">= (there is also new-style "Forwarded:" header but it is not widely= used yet, AFAIK)</span></div><div class=3D"m_-6904254408352713843gmail-m_3= 88361609629121005gmail-m_-525099954024514115gmail_msg gmail_msg"><span styl= e=3D"color:rgb(33,33,33)" class=3D"m_-6904254408352713843gmail-m_3883616096= 29121005gmail-m_-525099954024514115gmail_msg gmail_msg"><br class=3D"m_-690= 4254408352713843gmail-m_388361609629121005gmail-m_-525099954024514115gmail_= msg gmail_msg"></span></div><div class=3D"m_-6904254408352713843gmail-m_388= 361609629121005gmail-m_-525099954024514115gmail_msg gmail_msg"><span style= =3D"color:rgb(33,33,33)" class=3D"m_-6904254408352713843gmail-m_38836160962= 9121005gmail-m_-525099954024514115gmail_msg gmail_msg">X-Forwarded-For defi= netly can't be trusted if comes from untrusted client client. Fortunate= ly we don't need it at all for generating URLs :) It will be in questio= n when refactoring getClientIP() somewhen later.</span><br class=3D"m_-6904= 254408352713843gmail-m_388361609629121005gmail-m_-525099954024514115gmail_m= sg gmail_msg"></div><div class=3D"m_-6904254408352713843gmail-m_38836160962= 9121005gmail-m_-525099954024514115gmail_msg gmail_msg"><span style=3D"color= :rgb(33,33,33)" class=3D"m_-6904254408352713843gmail-m_388361609629121005gm= ail-m_-525099954024514115gmail_msg gmail_msg"><br class=3D"gmail_msg"></spa= n></div><div class=3D"m_-6904254408352713843gmail-m_388361609629121005gmail= -m_-525099954024514115gmail_msg gmail_msg"><span style=3D"color:rgb(33,33,3= 3)" class=3D"m_-6904254408352713843gmail-m_388361609629121005gmail-m_-52509= 9954024514115gmail_msg gmail_msg">But can we trust=C2=A0</span><span style= =3D"color:rgb(33,33,33)" class=3D"m_-6904254408352713843gmail-m_38836160962= 9121005gmail-m_-525099954024514115gmail_msg gmail_msg">X-Forwarded-Host &am= p; X-Forwarded-Proto? From the first glance it isn't a problem since we= are using them to display URLs for the same client, so nasty client will g= et his nasty URLs, that's it. But if app is doing something like storin= g URL in DB or (more likely) sending an email with a link to another client= , this would be an issue.</span></div></div></div></blockquote><div class= =3D"gmail_msg"><br class=3D"gmail_msg"></div></div></div></div><div dir=3D"= ltr" class=3D"gmail_msg"><div class=3D"gmail_extra gmail_msg"><div class=3D= "gmail_quote gmail_msg"><div class=3D"gmail_msg">It's not safe to enabl= e support for X-Forwarded-For and friends by default, since you can't k= now how application code will use that information and in many configuratio= ns it may be spoofed by clients.<br class=3D"gmail_msg"><br class=3D"gmail_= msg">A proper deployment which sets these headers looks like this:<br class= =3D"gmail_msg"><br class=3D"gmail_msg"></div><div class=3D"gmail_msg">1. Co= nfigure your frontend reverse-proxy (nginx or whatever) to discard incoming= X-Forwarded-* headers and <i class=3D"gmail_msg">set</i> X-Forwarded-* as = appropriate. Note that X-Forwarded-For is actually a <i class=3D"gmail_msg"= >list</i> of hops[1], so if you do this naively your server may append to t= he list! [2]<br class=3D"gmail_msg"></div><div class=3D"gmail_msg">2. Confi= gure your backend services to respect exactly the headers passed by your fr= ontend.<br class=3D"gmail_msg"></div><div class=3D"gmail_msg"><br class=3D"= gmail_msg"></div><div class=3D"gmail_msg">Note that this requires administr= ative action in two places, and that if you don't do the frontend confi= g it will probably pass the headers through, allowing them to be spoofed.<b= r class=3D"gmail_msg"></div><div class=3D"gmail_msg"><br class=3D"gmail_msg= "></div><div class=3D"gmail_msg">A non-exhaustive list of possible bad stuf= f I client could use spoofing for:<br class=3D"gmail_msg"><br class=3D"gmai= l_msg"></div><div class=3D"gmail_msg">1. Evade IP-based access control (a r= easonable defense-in-depth measure).<br class=3D"gmail_msg"></div><div clas= s=3D"gmail_msg">2. Evade pinning of user sessions to IP addresses or subnet= s.<br class=3D"gmail_msg"></div><div class=3D"gmail_msg">2. Evade IP-based = rate limiting, e.g. as discussed at [3].<br class=3D"gmail_msg"><br class= =3D"gmail_msg"></div><div class=3D"gmail_msg">It's also probably worth = noting that Django used to offer support for X-Forwarded-For and removed it= [4]. X-Forwarded-* and friends just too varied in the wild to reasonably s= upport.<br class=3D"gmail_msg"><br class=3D"gmail_msg"></div><div class=3D"= gmail_msg">If Twisted is to support this in any way, I think that it should= be opt-in support for the Forwarded header as specified in RFC 7239. This = should be a parameter applicable to all of twisted.web.server rather than p= er-method call, since it's something the administrator needs to set.<br= class=3D"gmail_msg"></div><div class=3D"gmail_msg"><br class=3D"gmail_msg"= ></div><div class=3D"gmail_msg">=E2=80=94Tom<br class=3D"gmail_msg"></div><= div class=3D"gmail_msg"><br class=3D"gmail_msg">[1]: Standardized as Forwar= ded in <a href=3D"https://tools.ietf.org/html/rfc7239" class=3D"gmail_msg" = target=3D"_blank">https://tools.ietf.org/html/rfc7239</a><br class=3D"gmail= _msg"></div><div class=3D"gmail_msg">[2]: Your frontend proxy should also v= alidate Host is a domain you control to prevent cookie theft. Plus lots of = other stuff. Web security is hard, and every default everywhere sets users = up for failure.<br class=3D"gmail_msg">[3]: <a href=3D"http://django-rateli= mit.readthedocs.io/en/v1.0.0/security.html" class=3D"gmail_msg" target=3D"_= blank">http://django-ratelimit.readthedocs.io/en/v1.0.0/security.html</a><b= r class=3D"gmail_msg">[4]: <a href=3D"https://www.djangoproject.com/weblog/= 2009/jul/28/security/#secondary-issue" class=3D"gmail_msg" target=3D"_blank= ">https://www.djangoproject.com/weblog/2009/jul/28/security/#secondary-issu= e</a></div></div></div></div> _______________________________________________<br class=3D"gmail_msg"> Twisted-web mailing list<br class=3D"gmail_msg"> <a href=3D"mailto:[email protected]" class=3D"gmail_msg" target= =3D"_blank">[email protected]</a><br class=3D"gmail_msg"> <a href=3D"http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web" r= el=3D"noreferrer" class=3D"gmail_msg" target=3D"_blank">http://twistedmatri= x.com/cgi-bin/mailman/listinfo/twisted-web</a><br class=3D"gmail_msg"> </blockquote></div></div></div> --001a113cec02b739bf054b2e787c-- --===============0007370320639162671== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Twisted-web mailing list [email protected] http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web --===============0007370320639162671==--