Trying to diagnose a LWP on FreeBSD (>8.x) issue

Philip Kizer <[email protected]> Fri, 1 Mar 2013 15:40:38 -0600
Newsgroups gmane.comp.lang.perl.modules.lwp
Message-ID <[email protected]>
I tried mailing the maintainer but no response there has led me to a bit =
more public forum.


Using any perl > 5.8 (since I really don't have older except for one =
system about to be upgraded, they're mostly 5.12 for me right now) and =
LWP 6.04, I see the following behaviour that can be tested as simply a =
this that mimics what a personal proxy is doing for me:

GET -H 'User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)' =
http://www.gocomics.com/doonesbury/2013/01/27/

When doing that, the output I get is truncated at what is essentially =
the end of the first data packet.


To verify the server, requesting machine and network, if I merely repeat =
that action with another tool it succeeds:

curl -H 'User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)' =
http://www.gocomics.com/doonesbury/2013/01/27/


To further narrow it down, the following copied almost verbatim from the =
Net::HTTP perldoc also works just fine:

% cat try-net-http.pl=20
#!/usr/local/bin/perl -w
use Net::HTTP;

my $s =3D Net::HTTP->new(Host =3D> "www.gocomics.com") || die $@;
$s->write_request(GET =3D> "/doonesbury/2013/01/27/", 'User-Agent' =3D> =
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)");
my($code, $mess, %h) =3D $s->read_response_headers;

while (1) {
  my $buf;
  my $n =3D $s->read_entity_body($buf, 1024);
  die "read failed: $!" unless defined $n;
  last unless $n;
  print $buf;
}



The common elements I have found are that I have to be using LWP on =
FreeBSD-8 or FreeBSD-9 to observe the failure, older 6.x and 7.x systems =
all produce full output even with the same version of LWP.  For me at =
least this seems to hold on all 8.x and 9.x systems I have.


Can anyone at least verify that they see the same behaviour?  If so, can =
anyone suggest where to look for fixes?


Thanks,
Philip