Re: Another utf8 problem

On 10/10/2006 01:13 PM, Dean Brockhausen wrote:
> I have been experiencing problems when I try to put binary data
> via LWP.  The problem only manifests it self when I am making a
> HTTPS request . I have hex coded the binary data for easy viewing.
> the buffer that I pass to LWP does not have the utf8 flag on.
> 
> 
> The data that I put is
> 
> a4cb83b4ce7681417e1c425d1a30d02690c0d31e9490c3c708e229eee13917a3
> 
> The data the server gets is :
> 
> c2a4c38bc283c2b4c38e76c281417e1c425d1a30c39026c290c380c3931ec294
> 
> The c2..c3..c2......c3 etc makes me think :
> 
> Is this my data utf-8 encoded and truncated?
> 
> So I take the put data and encode it to utf8
> e.g. $data=$encode("utf8",$data);
> 
> c2a4c38bc283c2b4c38e76c281417e1c425d1a30c39026c290c380c3931ec294c290c383c38708c3a229c3aec3a13917c2a3 
> 
> 
> Sure enough it is.
> 
> I have followed the data all the way to the syswrite on line 292 of http.pm
> (version # $Id: http.pm,v 1.70 2005/12/08 10:28:01 gisle Exp $)
> and the data is not been encoded at this point. but the utf8 flag has 
> changed.
> 
> 
> at http.pm line 205 $req_buf has the utf8 flag on.
> at http.pm line 233 $$content_ref does not have the utf8 flag.
> the cancatenation at lin 234 gives $buf the utf8 flag.
> 
> this is the same behavior between http and https requests.
> Finaly ethereal shows that the http data has not been encoded.
> While I cannot look at the wire in the https case the data at the
> other end has been utf8 encoded and truncated.
> the truncation seems to be from the issue described in this thread
> http://www.mail-archive.com/[email protected]/msg06097.html
> 
> 
> 
> if I nix the utf8 flag the binary data does not get encoded and
> every thing works.
> e.g. (line 233 ...)
> 
> if ($req_buf) {
>              Encode::_utf8_off($req_buf);# DMB
>         my $buf = $req_buf . $$content_ref;
>         $wbuf = \$buf;
>         }
> 
> I may be doing something wrong, but I think it is a bug somewhere
> under syswrite
> to encode the data that was not utf8 when it came from the user.
> Any help would be appreciated, though for now I will continue to nix
> the utf8 flag.
> 
> Dean
> 

You didn't post a small working example of your code and data, so it's 
impossible to test this problem. All I can do is offer general advice 
such as to uuencode or base64-encode the data before sending (if you 
have control over both scripts).

Perhaps you should create a small script (not your entire program) that 
demonstrates the problem.