Re: encoding solutions.

Hi there,

This possibly what I'm replying to didn't get posted to the list as it was 
held up in a mailman moderation request which now seems gone. Possibly Stuart 
resent  it but I haven't seen it yet and I am replying anyway. :)

Stuart Bishop writes:
> One point is that it will be a pain to support development of
> a Formulator that uses Unicode strings for both Zope 2.5- and Zope 2.6+.
> Another point is that while Page Templates handle Unicode strings, DTML
> pages don't. I think the sanest way to do Unicode properly in Zope at
> the moment is to drop support for Zope 2.5 and DTML, but I'm sure not
> everyone has that luxury :-)

Hm, I hadn't considered the DTML issue. Dropping support for Zope 2.5 for
newer Formulator versions I am considering though, if I haven't done
it accidentally already. But oops, breaking DTML would be a more
far-reaching thing to do.

Perhaps some hackery could be in order. Let's assume Formulator content
remains in whatever encoding it is, which we'll assume is latin1. Then
I could introduce a second argument to get_value to output it as 'unicode' if
string output is happening at all. This would then return unicode.
Or perhaps it shouldn't be an argument but a form-global setting.

Of course there's also the whole XML encoding issue..

> It may be possible to make TAL.TALInterpreter.FasterStringIO handle
> mixing Unicode string and traditional strings sanely. It would involve
> replacing the inherited getvalue() method with one that catches the
> ASCII encoding exceptions, converting these strings to Unicode strings
> (by checking the RESPONSE headers for the expected character encoding,
> or using a heuristic method to guess if you're feeling brave).
> I haven't tried this yet, and fixing DTML is left as an exercise to the
> reader :-)

I'm not sure I understand what you're proposing here. Are you proposing
changing TAL? That could have a pretty far reaching impact.

Right now Zope seems to be following Python's behavior; you can't
mix traditional strings and unicode unless the traditional string only
contains ascii (first 128 characters) text. If you put in unicode at all,
all ZPT output will turn into unicode strings.

Then on the output side the output encoding setting is checked to convert
it all. I think the default behavior (without any encoding setting)
is to make latin1 out of it all, but you can set the output encoding
to UTF-8 to make it do an .encode('UTF-8'). We're doing this for Silva.

Regards,

Martijn