Re: Charset Encoding Problem

On Wed, Aug 09, 2006 at 01:00:41PM +0100, Barclay Curtis wrote:
> Hi,
> 
> I've searched the archives but can't seem to find anything on this.
> I've just converted my site from CGI to FCGI.
> It is a multilingual site, written in Perl(5.8.7) using 
> HTML::Mason(1.33), MySQL(4.0.21) and Apache(1.3.33)  on Debian.
> 
> It seemed to be working fine on FCGI until I noticed that the accented 
> characters on some of the Spanish pages aren't displaying correctly.
> The same Mason component pages are being used for the CGI and FCGI versions.
> The page is declaring its charset properly.
> 
> The problem pages are fine under CGI.
> 
> Does anybody know why this would happen?

at a guess, this will be because the `real' STDOUT which
you use in CGI mode has a different encoding layer to the
faked up one which FCGI provides.

In particular, consider something like this script:

    #!/usr/bin/perl -w
    use charnames ':full';
    use FCGI;
    my $R = FCGI::Request();
    while ($R->Accept() >= 0) {
        print "Content-Type: text/plain\r\n\r\n";
        print "\N{POUND SIGN}\n";
    }

run in CGI mode or from the command-line, it will produce
the bytes 0xa3, 0x0a (after the header), corresponding to
a `£' in ISO-8859-1 followed by a carriage-return. That's
because the default translation for STDOUT is ISO-8859-1
in perl.

If you run it as FastCGI, however, you get 0xc2, 0xa3,
0x0a, corresponding to the pound sign in UTF-8. That's
because the fake filehandle made up by FCGI just passes
the entered bytes back to the webserver, without any
transcoding.

My guess is that your application was implicitly relying
on the translation to ISO-8859-1 previously. In principle
this can be restored by doing
    binmode(STDOUT, ":encoding(ISO-8859-1)");
but actually that doesn't work, I think because FCGI.xs
doesn't use the underlying PerlIO stuff for its IO.

The correct thing to do here is probably to transcode
everything into the character set you actually want to use
for output before outputting it, and in CGI mode set
binmode ':bytes' rather than relying on the encoding
layer, though there may be a neater solution that escapes
me. If you're lucky you might be able to get away with
just saying that the output pages are charset="utf-8".

-- 
``That's the thing about the Internet: unless you're actually
  looking at them, you can't hear anybody screaming.'' (Miles Martin)
___________________________________
fastcgi-developers mailing list
http://fastcgi.com/fastcgi-developers/