Re: Re: encoding solutions.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On Wednesday, September 24, 2003, at 10:51  PM, Martijn Faassen wrote:

> Stuart Bishop writes:
>> One point is that it will be a pain to support development of
>> a Formulator that uses Unicode strings for both Zope 2.5- and Zope 
>> 2.6+.
>> Another point is that while Page Templates handle Unicode strings, 
>> DTML
>> pages don't. I think the sanest way to do Unicode properly in Zope at
>> the moment is to drop support for Zope 2.5 and DTML, but I'm sure not
>> everyone has that luxury :-)
>
> Hm, I hadn't considered the DTML issue. Dropping support for Zope 2.5 
> for
> newer Formulator versions I am considering though, if I haven't done
> it accidentally already. But oops, breaking DTML would be a more
> far-reaching thing to do.

I was wrong about this. DTML under Zope 2.6 handles Unicode, and if
DTML encounters traditional strings mixed with Unicode strings, the
traditional strings are assumed to be Latin-1 and converted to Unicode.
So if dropping 2.5 support is possible, then Formulator could just 
output
Unicode strings at all times (only annoying people who are mixing 
non-Latin1
encoded output with Formulator output in DTML - Any Asian Formulator 
users?)

> Perhaps some hackery could be in order. Let's assume Formulator content
> remains in whatever encoding it is, which we'll assume is latin1. Then
> I could introduce a second argument to get_value to output it as 
> 'unicode' if
> string output is happening at all. This would then return unicode.
> Or perhaps it shouldn't be an argument but a form-global setting.

Acquired from the Form (or in my case, my Product since I'm only using
Fields).

> Of course there's also the whole XML encoding issue..

Which I don't know about, and can't find the original thread.

>> It may be possible to make TAL.TALInterpreter.FasterStringIO handle
>> mixing Unicode string and traditional strings sanely. It would involve
>> replacing the inherited getvalue() method with one that catches the
>> ASCII encoding exceptions, converting these strings to Unicode strings
>> (by checking the RESPONSE headers for the expected character encoding,
>> or using a heuristic method to guess if you're feeling brave).
>> I haven't tried this yet, and fixing DTML is left as an exercise to 
>> the
>> reader :-)
>
> I'm not sure I understand what you're proposing here. Are you proposing
> changing TAL? That could have a pretty far reaching impact.

Currently under discussion on [email protected] I've proposed a patch
which works and only impacts those pages that currently generate a
Unicode exception. If this is accepted into Zope 2.7, it may be possible
to not worry about Unicode in Formulator until Formulator v2. 
Interestingly,
if you look closely at TAL/TALInterpreter.py there is a point where it
is possible to monkey patch this solution in with full endorsement...

class TALInterpreter:
     def StringIO(self):
         # Third-party products wishing to provide a full Unicode-aware
         # StringIO can do so by monkey-patching this method.
         return FasterStringIO()

:-)

- -- 
Stuart Bishop <[email protected]> ☞ http://www.stuartbishop.net/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (Darwin)

iD8DBQE/kiDWAfqZj7rGN0oRAuqNAKCK3JMt11arivn4/9O0sZfYvBxnQQCfewNJ
7+anNAPerX/n4ksWQmG46hs=
=PVwu
-----END PGP SIGNATURE-----