problem about posting non-utf8 data

Hi everyone,

I've met a problem relating to posting utf8 data (gbk, actually , which is
required by the target site).

the code (in gbk encoding):

...
...
$mech->form_name('form1');

my $response = $mech->submit_form(
    form_name => 'form1',
    fields => {
        searchContent => '公司',
    },
);
...
...

the content of the resulting post was
''xxxxxxx&searchContent=%C2%B9%C2%AB%C3%8B%C2%BE" which was not correct.
should be "%B9%AB%CB%BE"

I've tried to use utf8; and change the value to encode('gbk', '公司'); (of
course also changed the encoding of the source to utf8) and the problem
still there.

I debuged and found the problem was around URI::_query::query:

sub query
{
    my $self = shift;
    $$self =~ m,^([^?\#]*)(?:\?([^\#]*))?(.*)$,s or die;

    if (@_) {
    my $q = shift;
    $$self = $1;
    if (defined $q) {
        $q =~ s/([^$URI::uric])/ URI::Escape::escape_char($1)/ego;
        $$self .= "?$q";
    }
    $$self .= $3;
    }
    $2;
}

before the subtitution, $q was correct and after that it turns out wrong.
than i looked at URI::Escape::escape_char:

sub escape_char {
    return join '', @URI::Escape::escapes{$_[0] =~ /(\C)/g};
}

this method was called for four times, each time a byte (half a Chinese
character) was passed in. on the entry point, length($_[0]) eq 1 but the
/(\C)/g matching produces two bytes, which couses the problem.

I further discovered that the problem was caused by a join of utf8-flaged
string and a non-utf8-flaged string -- perl automatically marks the
resulting string utf8, which influnces the behavior of \C.

here's a test program:

  1 use strict;
  2 use warnings;
  3 use Encode;
  4 use utf8;
  5
  6 my $s1 = decode('utf8', 's1');
  7 my $s2 = encode('gbk','公司');
  8 my $s3 = "$s1+$s2";
  9
 10 print_is_utf8($s1);
 11 print_is_utf8($s2);
 12 print_is_utf8($s3);
 13
 14
 15 print_str($s1);
 16 print_str($s2);
 17 print_str($s3);
 18
 19 sub print_str {
 20     my $str = shift;
 21     print "$str: ";
 22     print unpack('H*', $str) . '=' . join('+', map {unpack('H*', $_)}
($str=~/(\C)/g)) . "\n";
 23 }
 24
 25 sub print_is_utf8 {
 26     my $str = shift;
 27     print +(Encode::is_utf8($str)?"y":"n"), "\n";
 28 }
~
~
the result:
y
n
y
s1: 7331=73+31
公司: b9abcbbe=b9+ab+cb+be
s1+公司: 73312bb9abcbbe=73+31+2b+c2+b9+c2+ab+c3+8b+c2+be


So, now it's clear: Mechanize automatically decoded the page content, so the
existing keys and values of the form have utf8 flags. in URI::_query this
strings are joined with user provided keys and values, which in my case are
not utf8-flagged. And then URI::Escape use \C to mach the byte and produces
the wrong answer.

I think this can be fix by either Mechnize (don't auto-decode) or
URI::_query (turn off utf8-flag before joining the key and value) .


So now the question is, what should I do?


Thank you. (Sorry for poor English)



msmouse
----------------------------------
[email protected]
[email protected]