Re: Crypt::SSLeay HTTPS_PROXY support
jason <[email protected]>
| Newsgroups | gmane.comp.lang.perl.modules.lwp |
|---|---|
| Message-ID | <[email protected]> |
Hi, Atul The content for the failed HTTPS request is this ------------------ 500 proxy connect failed: PROXY ERROR HEADER, could be non-SSL URL: HTTP/1.1 403 Forbidden Date: Wed, 27 Sep 2006 15:01:50 GMT Content-Length: 298 Connection: close Content-Type: text/html; charset=iso-8859-1 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access 207.154.17.141:80 on this server.</p> <hr> ---------------------------- I searched the error message "PROXY ERROR HEADER, could be non-SSL URL" through google. There is not a solution for this. Thanks -jason Atul Saroop wrote: > what does $res->content print out to in the end? > > On 9/20/06, *jason* <[email protected] <mailto:[email protected]>> wrote: > > Hi, Atul > Still it does not work. > --------------My code > > #!/usr/bin/perl > # PROXY SUPPORT > $ENV{HTTPS_PROXY} = '207.154.17.141:80 <http://207.154.17.141:80>'; > #$ENV{HTTPS_PROXY} = 'localhost:8080'; > > # DEBUGGING SWITCH / LOW LEVEL SSL DIAGNOSTICS > $ENV{HTTPS_DEBUG} = 2; > # PROXY_BASIC_AUTH > > $ENV{HTTPS_PROXY_USERNAME} = "test"; > $ENV{HTTPS_PROXY_PASSWORD} = "test"; > > # DEFAULT SSL VERSION > $ENV{HTTPS_VERSION} = '3'; > > # CLIENT CERT SUPPORT > $ENV{HTTPS_CERT_FILE} = 'certs/notacacert.pem'; > $ENV{HTTPS_KEY_FILE} = 'certs/notacakeynopass.pem'; > > # CA CERT PEER VERIFICATION > $ENV{HTTPS_CA_FILE} = 'certs/ca-bundle.crt'; > $ENV{HTTPS_CA_DIR} = 'certs/'; > > # DEFAULT SSL VERSION > $ENV{HTTPS_VERSION} = '3'; > > use LWP::Debug qw(+); > use LWP::UserAgent; > my $ua = LWP::UserAgent->new(env_proxy => 0) ; > $ua->proxy(['https'], ''); > my $req = new HTTP::Request('GET', 'http://www.google.com'); > > my $res = $ua->request($req); > print $res->code."\n"; > my $req = new HTTP::Request('GET', 'https://www.nodeworks.com'); > $res = $ua->request($req); > print $res->code."\n"; > ----------------------------- > This is debug output > ----------------------- > LWP::UserAgent::new: () > LWP::UserAgent::proxy: ARRAY(0x818e314) > LWP::UserAgent::proxy: https > LWP::UserAgent::request: () > LWP::UserAgent::send_request: GET http://www.google.com > > LWP::UserAgent::_need_proxy: Not proxied > LWP::Protocol::http::request: () > LWP::Protocol::collect: read 230 bytes > LWP::UserAgent::request: Simple response: Found > LWP::UserAgent::request: () > LWP::UserAgent::send_request: GET http://www.google.com/intl/zh-CN/ > > LWP::UserAgent::_need_proxy: Not proxied > LWP::Protocol::http::request: () > LWP::Protocol::collect: read 693 bytes > LWP::Protocol::collect: read 406 bytes > LWP::Protocol::collect: read 1430 bytes > LWP::Protocol::collect: read 50 bytes > LWP::UserAgent::request: Simple response: OK > 200 > > LWP::UserAgent::request: () > LWP::UserAgent::send_request: GET https://www.nodeworks.com > LWP::UserAgent::_need_proxy: Not proxied > LWP::Protocol::http::request: () > SSL_connect:before/connect initialization > SSL_connect:SSLv3 write client hello A > SSL3 alert write:fatal:handshake failure > SSL_connect:error in SSLv3 read server hello A > > LWP::UserAgent::request: Simple response: Internal Server Error > 500 > ----------------------------------- > > Thanks for the help > -jason > > > > > Atul Saroop wrote: >> Also, while newing the UserAgent, try the >> >> $ua = LWP::UserAgent->new(env_proxy => 0) ; >> >> This must be it.. damn.. I should have gotten this one earlier... >> >> Regards, >> Atul. >> >> On 9/20/06, *Atul Saroop* <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Jason: >> >> Can you try another thing then... >> >> try setting >> >> $ua->proxy(['https'], ''); >> >> as the null string.. that ensures that libwww necessarily >> goes through Crypt::SSLeay only. I know for sure that this >> has to be done for WWW::Mechanize.. but not sure about LWP. >> >> Also, can you send the whole response content also? >> >> Another pointer .. does your access with simple http take >> place at all to websites like www.google.com >> <http://www.google.com>? >> >> Regards, >> Atul. >> >> >> On 9/20/06, *jason* < [email protected] >> <mailto:[email protected]>> wrote: >> >> Hi, Atul >> My proxy is an apache server. I did not set up >> authentication for proxy. >> >> To test whether it is the username/password that causes >> the issues, I set up proxy authentication. I tested the >> proxy authentication with firefox browser. It is ok for >> firefox. I got the https://www.nodework.com web page and >> here is the server access log. >> - test [27/Sep/2006:09:48:14 -0400] "CONNECT >> www.nodeworks.com:443 <http://www.nodeworks.com:443> >> HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows; U; Windows NT >> 5.1; en-US; rv:1.8.0.7 <http://1.8.0.7>) Gecko/20060909 >> Firefox/1.5.0.7" >> - test [27/Sep/2006:09:48:17 -0400] "CONNECT >> www.nodeworks.com:443 <http://www.nodeworks.com:443> >> HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows; U; Windows NT >> 5.1; en-US; rv:1.8.0.7 <http://1.8.0.7>) Gecko/20060909 >> Firefox/1.5.0.7" >> >> Then I run the code with proxy username/password, >> $ENV{HTTPS_PROXY_USERNAME} = "test"; >> $ENV{HTTPS_PROXY_PASSWORD} = "test"; >> I got the same result. >> - test [27/Sep/2006:09:48:57 -0400] "CONNECT >> www.nodeworks.com:443 <http://www.nodeworks.com:443> >> HTTP/1.0" 200 - "-" "libwww-perl/5.805" >> - test [27/Sep/2006:09:48:58 -0400] "CONNECT >> 207.154.17.141:80 <http://207.154.17.141:80> HTTP/1.0" >> 403 298 "-" "libwww-perl/5.805" >> >> I think the root cause is that the libwww or >> Crypt::SSLeay sends an extra CONNECT to the proxy server >> (here 207.154.17.141 <http://207.154.17.141>) itself. >> This request gets an 403 response from server. >> >> Thanks >> >> -jason >> >> >> >> Atul Saroop wrote: >>> Jason: >>> >>> The >>> $ENV{HTTPS_PROXY_USERNAME} = 'jason_username' ; >>> $ENV{HTTPS_PROXY_PASSWORD} = 'jason_password' ; >>> lines are supposed to contain your proxy username and >>> password. Note that you may not explicitly know that >>> your windows network login (in case you work on an NT >>> network) authentication is used by proxy servers under >>> NT authentication. >>> >>> So, try filling in those authentication details in the >>> above mentioned lines. >>> >>> Also, in my version of your code that was working all >>> this while, I emptied out the username and it gave me a >>> 500 also, but a print of $res->content shows that it is >>> a proxy authentication failure. >>> >>> I hope this should help you solve your problem. >>> >>> Regards, >>> Atul. >>> >>> >>> On 9/19/06, *jason* <[email protected] >>> <mailto:[email protected]> > wrote: >>> >>> Hi, atul >>> >>> I added "use LWP::Debug qw(+);" >>> Here is output from client >>> ---------------------- >>> LWP::UserAgent::new: () >>> LWP::UserAgent::request: () >>> LWP::UserAgent::send_request: GET >>> https://www.nodeworks.com <https://www.nodeworks.com/> >>> LWP::UserAgent::_need_proxy: Not proxied >>> LWP::Protocol::http::request: () >>> SSL_connect:before/connect initialization >>> SSL_connect:SSLv3 write client hello A >>> SSL3 alert read:fatal:handshake failure >>> SSL_connect:failed in SSLv3 read server hello A >>> LWP::UserAgent::request: Simple response: Internal >>> Server Error >>> 500 >>> -------------------------------------- >>> This is the apache access log >>> -------------------------------- >>> - - [26/Sep/2006:08:56:59 -0400] "CONNECT >>> www.nodeworks.com:443 >>> <http://www.nodeworks.com:443/> HTTP/1.0" 200 - "-" >>> "libwww-perl/5.805" >>> - - [26/Sep/2006:08:57:01 -0400] "CONNECT >>> 207.154.17.141:80 <http://207.154.17.141/> HTTP/1.0" >>> 403 298 "-" "libwww-perl/5.805" >>> -------------------------------------------------------- >>> >>> I added other parameters and used the certificate >>> from Crypt-SSLeay distribution. >>> ---------------------- >>> $ENV{HTTPS_DEBUG} = 2; >>> # PROXY_BASIC_AUTH >>> $ENV{HTTPS_PROXY_USERNAME} = ''; >>> $ENV{HTTPS_PROXY_PASSWORD} = ''; >>> >>> # DEFAULT SSL VERSION >>> $ENV{HTTPS_VERSION} = '3'; >>> >>> # CLIENT CERT SUPPORT >>> $ENV{HTTPS_CERT_FILE} = 'certs/notacacert.pem'; >>> $ENV{HTTPS_KEY_FILE} = 'certs/notacakeynopass.pem'; >>> >>> # CA CERT PEER VERIFICATION >>> $ENV{HTTPS_CA_FILE} = 'certs/ca-bundle.crt '; >>> $ENV{HTTPS_CA_DIR} = 'certs/'; >>> >>> # DEFAULT SSL VERSION >>> $ENV{HTTPS_VERSION} = '3'; >>> ---------------------------------------- >>> I get the same result. >>> >>> You can run the script against my proxy server >>> 207.154.17.141 <http://207.154.17.141/>. Maybe my >>> proxy server is not configured properly. I checked >>> my proxy configuration using firefox browser. >>> >>> Thanks >>> >>> >>> -jason >>> >>> >>> [email protected] <mailto:[email protected]> >>> wrote: >>>> Jason: >>>> >>>> Try adding a >>>> use LWP::Debug qw(+); >>>> at the top of the code and see what the perl program reports. >>>> >>>> Btw, your code runs fine on my proxy configuration. I just have to add >>>> a $ENV{HTTPS_PROXY_USERNAME} and a $ENV{HTTPS_PROXY_PASSWORD} with the >>>> >>>> >>>> >>>> other specifications also. >>>> >>>> Regards, >>>> Atul. >>>> >>>> Jason wrote: >>>> >>>>> Hi, >>>>> >>>>> I could not get the example script from Crypt::SSLeay run successfully. >>>>> Here is the script >>>>> --------------------- >>>>> #!/usr/bin/perl >>>>> # PROXY SUPPORT >>>>> $ENV{HTTPS_PROXY} = ' >>>>> >>>>> >>>>> 207.154.17.141:80 <http://207.154.17.141/>'; >>>>> # DEBUGGING SWITCH / LOW LEVEL SSL DIAGNOSTICS >>>>> $ENV{HTTPS_DEBUG} = 2; >>>>> >>>>> >>>>> use LWP::UserAgent; >>>>> my $ua = new LWP::UserAgent; >>>>> my $req = new HTTP::Request('GET', ' >>>>> >>>>> >>>>> https://www.nodeworks.com <https://www.nodeworks.com/>'); >>>>> my $res = $ua->request($req); >>>>> print $res->code."\n"; >>>>> ------------------------------------ >>>>> >>>>> I tested the proxy server with Firefox browser. It worked fine. I >>>>> >>>>> >>>>> >>>>> looked into the access log in the proxy server. When I run the example >>>>> script. >>>>> >>>>> I got two access log entries >>>>> -------------------- >>>>> - - [25/Sep/2006:20:42:23 -0400] "CONNECT >>>>> >>>>> >>>>> www.nodeworks.com:443 <http://www.nodeworks.com:443/> >>>>> HTTP/1.0" 200 - "-" "libwww-perl/5.805" >>>>> - - [25/Sep/2006:20:42:24 -0400] "CONNECT >>>>> >>>>> 207.154.17.141:80 <http://207.154.17.141/> HTTP/1.0" >>>>> 403 298 "-" "libwww-perl/5.805" >>>>> ------------------- >>>>> >>>>> Notice that LWP tried to issue an CONNECT method to the proxy server >>>>> itself. It got a 403 response for this request. On the other hand, >>>>> >>>>> >>>>> >>>>> firefox did not send CONNECT method to proxy server itself. >>>>> - - [25/Sep/2006:19:33:34 -0400] "CONNECT >>>>> >>>>> >>>>> www.nodeworks.com:443 <http://www.nodeworks.com:443/> >>>>> HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; >>>>> rv: >>>>> >>>>> 1.8.0.7 <http://1.8.0.7/>) Gecko/20060909 Firefox/1.5.0.7" >>>>> - - [25/Sep/2006:19:33:38 -0400] "CONNECT www.nodeworks.com:443 >>>>> >>>>> >>>>> <http://www.nodeworks.com:443/> >>>>> HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; >>>>> rv:1.8.0.7 <http://1.8.0.7/>) Gecko/20060909 Firefox/1.5.0.7" >>>>> >>>>> >>>>> - - [25/Sep/2006:19:34:32 -0400] "CONNECT >>>>> www.nodeworks.com:443 <http://www.nodeworks.com:443/> >>>>> HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows; U; Windows NT >>>>> >>>>> 5.1; en-US; >>>>> rv:1.8.0.7 <http://1.8.0.7/>) Gecko/20060909 Firefox/1.5.0.7" >>>>> - - [25/Sep/2006:19:34:29 -0400] "CONNECT >>>>> >>>>> >>>>> www.nodeworks.com:443 <http://www.nodeworks.com:443/> >>>>> HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; >>>>> rv: >>>>> >>>>> 1.8.0.7 <http://1.8.0.7/>) Gecko/20060909 Firefox/1.5.0.7" >>>>> >>>>> Not sure what is trick here. It seemed that other can get the example >>>>> script run properly. Can anyone give me some help on this? >>>>> >>>>> Thanks >>>>> >>>>> -jason >>>>> >>>>> >>>>> >>>>> >>>> >>> >>> >> >> >