Re: Offline mode + HTTPS does not seem possible
"Pottinger, Hardy J." <[email protected]> Fri, 10 Jul 2015 16:34:54 +0000
| Newsgroups | gmane.comp.web.polipo.user |
|---|---|
| Message-ID | <2CD26578BFA8024EBE6C0B84324FAD4670CD8EF2@UM-MBX-T03.um.umsystem.edu> |
--===============2652549337645121768== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_2CD26578BFA8024EBE6C0B84324FAD4670CD8EF2UMMBXT03umumsys_" --_000_2CD26578BFA8024EBE6C0B84324FAD4670CD8EF2UMMBXT03umumsys_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable It occurred to me that there might already be a Python-based caching proxy,= and there is (kinda/sorta): https://pypicache.readthedocs.org/en/latest/ It's specific to PyPi, but nothing we couldn't hacksaw out of the way. --Hardy ________________________________ From: Pottinger, Hardy J. Sent: Friday, July 10, 2015 11:05 AM To: Donovan Baarda Cc: [email protected] Subject: Re: [Polipo-users] Offline mode + HTTPS does not seem possible Oh, great, I followed that link to mitmproxy.org... I'm sure I'm on some go= vernment list now. ;-) OK, seriously, that sounds like a fun hack. I'm not much of a Pythonista, b= ut I'd love an excuse to play with it a bit. And this sounds like fun. If y= ou need a co-conspirator, let me know. --Hardy ________________________________ From: [email protected] [[email protected]] on behalf of Donovan Baarda [ab= [email protected]] Sent: Thursday, July 09, 2015 8:00 PM To: Pottinger, Hardy J. Cc: [email protected] Subject: Re: [Polipo-users] Offline mode + HTTPS does not seem possible Unfortunately, the enthusiastic push for better internet security with http= s is breaking cacheability. None of the people serving this content care, b= ecause traffic is one of their metrics for success, and they can cache righ= t up to their boundary by terminating the ssl at their edge, so they don't = care if it's non-cachable beyond there. Companies like Google "solve" the c= aching problem by just pushing their "edge" further and further out, owning= more and more of the internet infrustructure between customers and their s= ervers. I've been considering trying to create an unholy chimera by duct-taping pol= ipo and https://mitmproxy.org/ together with Python to fix this. I'm getting sick of the 5 different android devices in the house insisting = on downloading all the firmware and app updates 5x over https. This was par= ticularly bad when I was briefly connected using stupidly expensive 4G and = found the android devices had significantly contributed to exhausting my 5G= B pre-paid quota in less than 2 days. Things like mitmproxy are pretty bad from a security perspective, but unfor= tunately that's the only way to work around this. Unfortunately the push fo= r more security is working on breaking them too. Cert-pinning is the "magic= solution" to breaking on-the-fly cert generation used by mitmproxy, and it= 's already starting to appear in chrome etc. Not sure how easy it's going t= o be to work around. On 10 July 2015 at 04:38, Pottinger, Hardy J. <[email protected]<mai= lto:[email protected]>> wrote: Hi, I'm trying to use Polipo as a way to facilitate offline access to code = repositories (Maven central, github, etc) for my Vagrant-managed Virtualbox= development environment. Our build process uses Grunt and Bower to pull in= various pieces of front-end "stuff" and many of these URLs are https URLs = to github, for example: https://github.com/Modernizr/Modernizr.git Now, I can have Polipo running, and the network on, on my guest machine I c= an successfully run wget https://github.com/Modernizr/Modernizr.git Which, I would assume, is caching the content. I then disable my network co= nnection on my host machine, put Polipo in offline mode, and then re-attemp= t a wget of Modernizer... but I get the following: Connecting to 10.0.2.2:8123... connected. Proxy tunneling failed: Couldn't connect: Network is unreachableUnable to e= stablish SSL connection Running a tail -f on /var/log/polipo/polipo.log does not show me anything d= uring this error. Here is the only additions I've made to my /etc/polipo/config file: disableIndexing =3D false disableServersList =3D false dnsUseGethostbyname =3D reluctantly dnsQueryIPv6 =3D no dnsQueryIPv6 =3D reluctantly proxyOffline =3D yes Any advice would be much appreciated, thanks! On a positive note, despite not getting offline caching to work, I *have* n= oticed a considerable bump in compile performance. Down from 5 minutes to 2= minutes, with tests turned on. So, I'm moderately pleased about that. But,= i would like to figure out this offline proxy stuff. :-) --Hardy ---------------------------------------------------------------------------= --- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/ _______________________________________________ Polipo-users mailing list [email protected]<mailto:[email protected]= t> https://lists.sourceforge.net/lists/listinfo/polipo-users -- Donovan Baarda <[email protected]<mailto:[email protected]>= > --_000_2CD26578BFA8024EBE6C0B84324FAD4670CD8EF2UMMBXT03umumsys_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html dir=3D"ltr"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style id=3D"owaParaStyle" type=3D"text/css">=0A= <!--=0A= p=0A= {margin-top:0;=0A= margin-bottom:0}=0A= -->=0A= P {margin-top:0;margin-bottom:0;}</style> </head> <body ocsi=3D"0" fpstyle=3D"1"> <div style=3D"direction: ltr;font-family: Courier New;color: #000000;font-s= ize: 12pt;"> It occurred to me that there might already be a Python-based caching proxy,= and there is (kinda/sorta):<br> <br> <a href=3D"https://pypicache.readthedocs.org/en/latest/" target=3D"_blank">= https://pypicache.readthedocs.org/en/latest/</a><br> <br> It's specific to PyPi, but nothing we couldn't hacksaw out of the way.<br> <br> --Hardy<br> <br> <div style=3D"font-family: Times New Roman; color: #000000; font-size: 16px= "> <hr tabindex=3D"-1"> <div style=3D"direction: ltr;" id=3D"divRpF140536"><font face=3D"Tahoma" co= lor=3D"#000000" size=3D"2"><b>From:</b> Pottinger, Hardy J.<br> <b>Sent:</b> Friday, July 10, 2015 11:05 AM<br> <b>To:</b> Donovan Baarda<br> <b>Cc:</b> [email protected]<br> <b>Subject:</b> Re: [Polipo-users] Offline mode + HTTPS does not seem p= ossible<br> </font><br> </div> <div></div> <div> <div style=3D"direction:ltr; font-family:Courier New; color:#000000; font-s= ize:12pt"> Oh, great, I followed that link to mitmproxy.org... I'm sure I'm on some go= vernment list now. ;-)<br> <br> OK, seriously, that sounds like a fun hack. I'm not much of a Pythonista, b= ut I'd love an excuse to play with it a bit. And this sounds like fun. If y= ou need a co-conspirator, let me know.<br> <br> --Hardy<br> <br> <div style=3D"font-family:Times New Roman; color:#000000; font-size:16px"> <hr tabindex=3D"-1"> <div id=3D"divRpF650366" style=3D"direction:ltr"><font face=3D"Tahoma" colo= r=3D"#000000" size=3D"2"><b>From:</b> [email protected] [[email protected]]= on behalf of Donovan Baarda [[email protected]]<br> <b>Sent:</b> Thursday, July 09, 2015 8:00 PM<br> <b>To:</b> Pottinger, Hardy J.<br> <b>Cc:</b> [email protected]<br> <b>Subject:</b> Re: [Polipo-users] Offline mode + HTTPS does not seem p= ossible<br> </font><br> </div> <div></div> <div> <div dir=3D"ltr">Unfortunately, the enthusiastic push for better internet s= ecurity with https is breaking cacheability. None of the people serving thi= s content care, because traffic is one of their metrics for success, and th= ey can cache right up to their boundary by terminating the ssl at their edge, so they don't care if it's non-cacha= ble beyond there. Companies like Google "solve" the caching probl= em by just pushing their "edge" further and further out, owning m= ore and more of the internet infrustructure between customers and their servers. <div><br> </div> <div>I've been considering trying to create an unholy chimera by duct-tapin= g polipo and <a href=3D"https://mitmproxy.org/" target=3D"_blank">https://mitmproxy.org/= </a> together with Python to fix this. </div> <div><br> </div> <div>I'm getting sick of the 5 different android devices in the house insis= ting on downloading all the firmware and app updates 5x over https. This wa= s particularly bad when I was briefly connected using stupidly expensive 4G= and found the android devices had significantly contributed to exhausting my 5GB pre-paid quota in less than= 2 days.</div> <div><br> </div> <div>Things like mitmproxy are pretty bad from a security perspective, but = unfortunately that's the only way to work around this. Unfortunately the pu= sh for more security is working on breaking them too. Cert-pinning is the &= quot;magic solution" to breaking on-the-fly cert generation used by mitmproxy, and it's already starting to appear in = chrome etc. Not sure how easy it's going to be to work around.</div> <div><br> </div> </div> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">On 10 July 2015 at 04:38, Pottinger, Hardy J. <s= pan dir=3D"ltr"> <<a href=3D"mailto:[email protected]" target=3D"_blank">Pottinger= [email protected]</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex; border-left:1= px #ccc solid; padding-left:1ex"> Hi, I'm trying to use Polipo as a way to facilitate offline access to code = repositories (Maven central, github, etc) for my Vagrant-managed Virtualbox= development environment. Our build process uses Grunt and Bower to pull in= various pieces of front-end "stuff" and many of these URLs are https URLs to github, for example:<br> <br> <a href=3D"https://github.com/Modernizr/Modernizr.git" rel=3D"noreferrer" t= arget=3D"_blank">https://github.com/Modernizr/Modernizr.git</a><br> <br> Now, I can have Polipo running, and the network on, on my guest machine I c= an successfully run<br> <br> wget <a href=3D"https://github.com/Modernizr/Modernizr.git" rel=3D"noreferr= er" target=3D"_blank"> https://github.com/Modernizr/Modernizr.git</a><br> <br> Which, I would assume, is caching the content. I then disable my network co= nnection on my host machine, put Polipo in offline mode, and then re-attemp= t a wget of Modernizer... but I get the following:<br> <br> Connecting to 10.0.2.2:8123... connected.<br> Proxy tunneling failed: Couldn't connect: Network is unreachableUnable to e= stablish SSL connection<br> <br> Running a tail -f on /var/log/polipo/polipo.log does not show me anything d= uring this error.<br> <br> Here is the only additions I've made to my /etc/polipo/config file:<br> <br> disableIndexing =3D false<br> disableServersList =3D false<br> dnsUseGethostbyname =3D reluctantly<br> dnsQueryIPv6 =3D no<br> dnsQueryIPv6 =3D reluctantly<br> proxyOffline =3D yes<br> <br> Any advice would be much appreciated, thanks!<br> <br> On a positive note, despite not getting offline caching to work, I *have* n= oticed a considerable bump in compile performance. Down from 5 minutes to 2= minutes, with tests turned on. So, I'm moderately pleased about that. But,= i would like to figure out this offline proxy stuff. :-)<br> <br> --Hardy<br> <br> ---------------------------------------------------------------------------= ---<br> Don't Limit Your Business. Reach for the Cloud.<br> GigeNET's Cloud Solutions provide you with the tools and support that<br> you need to offload your IT needs and focus on growing your business.<br> Configured For All Businesses. Start Your Cloud Today.<br> <a href=3D"https://www.gigenetcloud.com/" rel=3D"noreferrer" target=3D"_bla= nk">https://www.gigenetcloud.com/</a><br> _______________________________________________<br> Polipo-users mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">Pol= [email protected]</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/polipo-users" rel= =3D"noreferrer" target=3D"_blank">https://lists.sourceforge.net/lists/listi= nfo/polipo-users</a><br> <br> </blockquote> </div> <br> <br clear=3D"all"> <div><br> </div> -- <br> <div class=3D"gmail_signature">Donovan Baarda <<a href=3D"mailto:abo@min= kirri.apana.org.au" target=3D"_blank">[email protected]</a>></di= v> </div> </div> </div> </div> </div> </div> </div> </body> </html> --_000_2CD26578BFA8024EBE6C0B84324FAD4670CD8EF2UMMBXT03umumsys_-- --===============2652549337645121768== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/ --===============2652549337645121768== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Polipo-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/polipo-users --===============2652549337645121768==--