Re: unicode data in ascii template

On Thu, Jan 9, 2014 at 2:55 AM, Dave Howorth <dhoworth-fDajt2Yx3S8pY9vWkoisglpr/1R2p/[email protected]>wrote:

> >   ENCODING => 'UTF-8'.
> >
> > Then Template::Provider will decode the templates when loading.
>
> Hi Bill, Thanks for the detailed reply. I'd spotted this option, but the
> templates aren't actually utf-8 so I figured that I really shouldn't
> need it, though it's an obvious possible thing to play with to see what
> effect it has. But I don't like blind guessing, so I didn't try it yet.
>

Here's a rule I use: all textual data outside your program (files, over a
socket, database) is encoded.   Therefore it must *always* be decoded when
reading into your program to be used as characters inside your program.
All character data inside Perl must also be encoded when leaving your
program.   (Obviously, character data in Perl is in some encoding, but I
think it helps to imagine it's just some abstract concept of characters.)



So, your template files contain textual data that is encoded.   So, you
should always decode your templates.

ENCODING => 'UTF-8'

if your templates are encoded in UTF-8.  (ASCII is a subset.)

You said you have a copyright symbol in your template.   You need to know
what encoding that is in currently and specify that encoding or (better)
replace it with the UTF-8 encoded copyright character and specify UTF-8.


> Then, as you mentioned in your DBIx::Class post, you need to tell the DBD
> > driver that your database is in UTF-8.  Then the DBD driver will decode
> > when reading.
> >
> > Then you have "characters" inside of Perl.
>
> Indeed so, and I think I had done that.
>

Check it.  Ask perl if a text field from the database is character data.
That is, if it was decoded.   Even just ASCII characters will be flagged:

$ perl -le 'use Encode;  my $x = Encode::decode_utf8( "hello" );  print
Encode::is_utf8( $x ) ? "flag is on\n" : "flag is off\n";'
flag is on





>
> >> What is the best way to persuade TT to generate a Unicode file?
> >
> > I would do something like this:
> >
> > $tt->process( $template, \%vars, \$output );
> > $bytes = Encode::encode_utf8( $output );
> >
> > Or have $output be a filehandle with a utf-8 output layer.
>
> This is a bit I have trouble with. My $output is just a filename, so TT
> is responsible for the output and should therefore be responsible for
> the encoding IMHO.


Encoding into what?

As Ben pointed out:

$tt->process ("input_file_name", $variables, $output_scalar,
          {binmode => ":utf8"})

where $output_scalar is a filename.

Take a look at Template.pm.   It's just calling binmode ":utf8" on the
filehandle for that file.

        elsif (open(FP, ">$where")) {
            # binmode option can be 1 or a specific layer, e.g. :utf8
            my $bm = $options->{ binmode  };
            if ($bm && $bm eq 1) {
                binmode FP;
            }
            elsif ($bm){
                *binmode FP, $bm;*
            }
            print FP $$textref;
            close FP;
        }




> The content of the file is made up by joining some
> ASCII strings (the template etc) to some utf-8 strings


What's a string? :)

Remember the rule that all encoded textual data (bytes) must be decoded
when bringing into your program.  Then you just have character inside your
program.   Doesn't joining characters sound better?

And it doesn't sound like your copyright symbol is encoded either as either
ASCII or UTF-8.






> ( the template
> variables obtained from the database. AAUI, perl should automagically
> make the resulting content string be a utf-8 internal string value. So
> when asked to print it to a file, TT ought to encode it appropriately in
> my view. Instead it outputs a file containing both utf-8 and
> windows-1252 byte sequences. But obviously I'm missing some principle.
>

There's no magic needed.

"TT ought to encode it appropriately"?  You are writing a file which is
just bytes.  You must tell TT what encoding you want the character data
written in.



> (1) I'm not seeing any wide character errors, which I think I ought to
> if binmode is a problem.
>

See rule above.


>
> (2) I've got a lot of calls to process, so hacking them all is
> unattractive. I'd rather like to find a single central change to get the
> behaviour I want if possible.
>

Well, that's a different problem.    Subclass Template?   Use a framework?
  Fix your code to not duplicate calls?




> I have now tried it and it makes things worse! The copyright symbol
> seems to be double encoded and the left quote becomes <U+0091> i.e. PU1.
>
> So I'm still completely confused. I'm obviously missing something but
> I've no idea where to look.
>

Dave, you might be over thinking this.   Again:

   1. Your template files (and text data in your database) are encoded.
    You must decode.   And you must know what encoding they are in, obviously.
     There's no copyright symbol in ASCII.
   2.  Character data inside perl must be encoded when outputting.

Doing both of those things at the "edge" of your program will make life
much easier.   A good framework will help with that.

-- 
Bill Moseley
[email protected]

_______________________________________________
templates mailing list
[email protected]
http://mail.template-toolkit.org/mailman/listinfo/templates