Re: Utf-8 and Content-Length header

Gisle Aas píše v čt 08. 06. 2006 v 07:22 -0700:
> Pav Lucistnik <[email protected]> writes:
> 
> > A general problem exists in libwww when POSTing data to HTTP servers,
> > Content-Length headers and syswrite() calls use length() function over
> > payload to determine it's size.  This fails on Utf-8 encoded data,5D
> > because length() function returns the count of _characters_, not count
> > of bytes. It leads to a message truncated.
> 
> If the data you put into the request content is UTF-8 encoded data
> then there is no problem.  Encoded text is already bytes and that's
> what the content and headers of HTTP::Message objects are supposed to
> be.  Use the Encode module to convert Unicode strings that you add to
> the request.

That's what I was doing at first.  But I hit some problems, so I tried
my luck with passing utf8-strings into libwww.  I tested my patches,
they do not affect operation on normal-strings in any way.


The real problem.  For some odd reason, $request_buf is coming with utf8
flag on, spoiling $content_buf thanks to this line

  my $buf = $req_buf . $$content_ref;

and the whole thing goes down in flames afterwards.

code:
    222 warn "WWWBUFFF content_ref ", utf8::is_utf8($$content_ref), " - ", length($$content_ref), "/", bytes::length($$content_ref);
    239 warn "WWWBUFFF req_buf ", utf8::is_utf8($req_buf), " - ", length($req_buf), "/", bytes::length($req_buf);
    240         my $buf = $req_buf . $$content_ref;
    241 warn "WWWBUFFF buf ", utf8::is_utf8($buf), " - ", length($buf), "/", bytes::length($buf);

output:
WWWBUFFF content_ref  - 1369/1369 at /usr/local/lib/perl5/site_perl/5.8.8/LWP/Protocol/http.pm line 222.
WWWBUFFF req_buf 1 - 328/328 at /usr/local/lib/perl5/site_perl/5.8.8/LWP/Protocol/http.pm line 239.
WWWBUFFF buf 1 - 1697/1699 at /usr/local/lib/perl5/site_perl/5.8.8/LWP/Protocol/http.pm line 241.

I'm a bit puzzled as on what's going on here.  Any tips?  I'm still
looking into this.

-- 
Pav Lucistnik <[email protected]>
              <[email protected]>

East or west, ~ is best.