Re: Offline mode + HTTPS does not seem possible
Donovan Baarda <[email protected]> Fri, 10 Jul 2015 11:00:25 +1000
| Newsgroups | gmane.comp.web.polipo.user |
|---|---|
| Message-ID | <CAKfm1duz2xCymtTKaoEH7hYpZe-48ReK_+pK1SMUitZDWDJTfA@mail.gmail.com> |
--===============3852519449718602637== Content-Type: multipart/alternative; boundary=001a11c2558eb02d47051a7ae5ae --001a11c2558eb02d47051a7ae5ae Content-Type: text/plain; charset=UTF-8 Unfortunately, the enthusiastic push for better internet security with https is breaking cacheability. None of the people serving this content care, because traffic is one of their metrics for success, and they can cache right up to their boundary by terminating the ssl at their edge, so they don't care if it's non-cachable beyond there. Companies like Google "solve" the caching problem by just pushing their "edge" further and further out, owning more and more of the internet infrustructure between customers and their servers. I've been considering trying to create an unholy chimera by duct-taping polipo and https://mitmproxy.org/ together with Python to fix this. I'm getting sick of the 5 different android devices in the house insisting on downloading all the firmware and app updates 5x over https. This was particularly bad when I was briefly connected using stupidly expensive 4G and found the android devices had significantly contributed to exhausting my 5GB pre-paid quota in less than 2 days. Things like mitmproxy are pretty bad from a security perspective, but unfortunately that's the only way to work around this. Unfortunately the push for more security is working on breaking them too. Cert-pinning is the "magic solution" to breaking on-the-fly cert generation used by mitmproxy, and it's already starting to appear in chrome etc. Not sure how easy it's going to be to work around. On 10 July 2015 at 04:38, Pottinger, Hardy J. <[email protected]> wrote: > Hi, I'm trying to use Polipo as a way to facilitate offline access to code > repositories (Maven central, github, etc) for my Vagrant-managed Virtualbox > development environment. Our build process uses Grunt and Bower to pull in > various pieces of front-end "stuff" and many of these URLs are https URLs > to github, for example: > > https://github.com/Modernizr/Modernizr.git > > Now, I can have Polipo running, and the network on, on my guest machine I > can successfully run > > wget https://github.com/Modernizr/Modernizr.git > > Which, I would assume, is caching the content. I then disable my network > connection on my host machine, put Polipo in offline mode, and then > re-attempt a wget of Modernizer... but I get the following: > > Connecting to 10.0.2.2:8123... connected. > Proxy tunneling failed: Couldn't connect: Network is unreachableUnable to > establish SSL connection > > Running a tail -f on /var/log/polipo/polipo.log does not show me anything > during this error. > > Here is the only additions I've made to my /etc/polipo/config file: > > disableIndexing = false > disableServersList = false > dnsUseGethostbyname = reluctantly > dnsQueryIPv6 = no > dnsQueryIPv6 = reluctantly > proxyOffline = yes > > Any advice would be much appreciated, thanks! > > On a positive note, despite not getting offline caching to work, I *have* > noticed a considerable bump in compile performance. Down from 5 minutes to > 2 minutes, with tests turned on. So, I'm moderately pleased about that. > But, i would like to figure out this offline proxy stuff. :-) > > --Hardy > > > ------------------------------------------------------------------------------ > Don't Limit Your Business. Reach for the Cloud. > GigeNET's Cloud Solutions provide you with the tools and support that > you need to offload your IT needs and focus on growing your business. > Configured For All Businesses. Start Your Cloud Today. > https://www.gigenetcloud.com/ > _______________________________________________ > Polipo-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/polipo-users > > -- Donovan Baarda <[email protected]> --001a11c2558eb02d47051a7ae5ae Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Unfortunately, the enthusiastic push for better internet s= ecurity with https is breaking cacheability. None of the people serving thi= s content care, because traffic is one of their metrics for success, and th= ey can cache right up to their boundary by terminating the ssl at their edg= e, so they don't care if it's non-cachable beyond there. Companies = like Google "solve" the caching problem by just pushing their &qu= ot;edge" further and further out, owning more and more of the internet= infrustructure between customers and their servers.<div><br></div><div>I&#= 39;ve been considering trying to create an unholy chimera by duct-taping po= lipo and <a href=3D"https://mitmproxy.org/">https://mitmproxy.org/</a> toge= ther with Python to fix this.=C2=A0</div><div><br></div><div>I'm gettin= g sick of the 5 different android devices in the house insisting on downloa= ding all the firmware and app updates 5x over https. This was particularly = bad when I was briefly connected using stupidly expensive 4G and found the = android devices had significantly contributed to exhausting my 5GB pre-paid= quota in less than 2 days.</div><div><br></div><div>Things like mitmproxy = are pretty bad from a security perspective, but unfortunately that's th= e only way to work around this. Unfortunately the push for more security is= working on breaking them too. Cert-pinning is the "magic solution&quo= t; to breaking on-the-fly cert generation used by mitmproxy, and it's a= lready starting to appear in chrome etc. Not sure how easy it's going t= o be to work around.</div><div><br></div></div><div class=3D"gmail_extra"><= br><div class=3D"gmail_quote">On 10 July 2015 at 04:38, Pottinger, Hardy J.= <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a>></span> wrote:<br><blockquote c= lass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;= padding-left:1ex">Hi, I'm trying to use Polipo as a way to facilitate o= ffline access to code repositories (Maven central, github, etc) for my Vagr= ant-managed Virtualbox development environment. Our build process uses Grun= t and Bower to pull in various pieces of front-end "stuff" and ma= ny of these URLs are https URLs to github, for example:<br> <br> <a href=3D"https://github.com/Modernizr/Modernizr.git" rel=3D"noreferrer" t= arget=3D"_blank">https://github.com/Modernizr/Modernizr.git</a><br> <br> Now, I can have Polipo running, and the network on, on my guest machine I c= an successfully run<br> <br> wget <a href=3D"https://github.com/Modernizr/Modernizr.git" rel=3D"noreferr= er" target=3D"_blank">https://github.com/Modernizr/Modernizr.git</a><br> <br> Which, I would assume, is caching the content. I then disable my network co= nnection on my host machine, put Polipo in offline mode, and then re-attemp= t a wget of Modernizer... but I get the following:<br> <br> Connecting to 10.0.2.2:8123... connected.<br> Proxy tunneling failed: Couldn't connect: Network is unreachableUnable = to establish SSL connection<br> <br> Running a tail -f on /var/log/polipo/polipo.log does not show me anything d= uring this error.<br> <br> Here is the only additions I've made to my /etc/polipo/config file:<br> <br> disableIndexing =3D false<br> disableServersList =3D false<br> dnsUseGethostbyname =3D reluctantly<br> dnsQueryIPv6 =3D no<br> dnsQueryIPv6 =3D reluctantly<br> proxyOffline =3D yes<br> <br> Any advice would be much appreciated, thanks!<br> <br> On a positive note, despite not getting offline caching to work, I *have* n= oticed a considerable bump in compile performance. Down from 5 minutes to 2= minutes, with tests turned on. So, I'm moderately pleased about that. = But, i would like to figure out this offline proxy stuff. :-)<br> <br> --Hardy<br> <br> ---------------------------------------------------------------------------= ---<br> Don't Limit Your Business. Reach for the Cloud.<br> GigeNET's Cloud Solutions provide you with the tools and support that<b= r> you need to offload your IT needs and focus on growing your business.<br> Configured For All Businesses. Start Your Cloud Today.<br> <a href=3D"https://www.gigenetcloud.com/" rel=3D"noreferrer" target=3D"_bla= nk">https://www.gigenetcloud.com/</a><br> _______________________________________________<br> Polipo-users mailing list<br> <a href=3D"mailto:[email protected]">[email protected]= urceforge.net</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/polipo-users" rel= =3D"noreferrer" target=3D"_blank">https://lists.sourceforge.net/lists/listi= nfo/polipo-users</a><br> <br> </blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class= =3D"gmail_signature">Donovan Baarda <<a href=3D"mailto:[email protected]= a.org.au" target=3D"_blank">[email protected]</a>></div> </div> --001a11c2558eb02d47051a7ae5ae-- --===============3852519449718602637== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/ --===============3852519449718602637== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Polipo-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/polipo-users --===============3852519449718602637==--