Re: encoding problem

On Tue, 05 Jan 2010, Juan wrote:

> Hi, I've installed cheetah-2.4.1 and still have problems with the encoding.
> These problems showed up also in versions 2.2.2 and 2.4.0 (see previous
> messages below). I've been trying to understand/correct them.
> 
> Although I do not have experience with cheetah it looks like the problem
> is that cheetah is only catching the #encoding directive if it is the
> first line in the file (neglecting empty lines) but if there is
> anything, even a comment before it, I've got an error message pointing
> to an 'ascii' codec problem
> 
>  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 ....
> 
> I've got an small example that works (I can either "cheeta compile" or
> "cheeta fill" on it. But if we swap the first two lines from:
> #encoding UTF-8
> ## comment
> 
> to:
> ## comment
> #encoding UTF-8
> 
> it gives an encoding error.
> 
> I think the problem is in the file Compiler.py, (using re.match instead
> of re.search) but again, I don't know enough of the workings of cheeta
> to be sure.
> I attach a file with the example and the proposed patch. Please let me
> know if you need anything else,

Interesting, thanks for digging into this; I've applied your patch locally and
I will write a test or two for it this weekend and commit it.


Cheers,
-R. Tyler Ballance
--------------------------------------
 Jabber: [email protected]
 GitHub: http://github.com/rtyler
Twitter: http://twitter.com/agentdero
   Blog: http://unethicalblogger.com


> > From: Juan Fiol <[email protected]>
> > Subject: [Cheetahtemplate-discuss] encoding problem
> > To: [email protected]
> > Message-ID: <[email protected]>
> > Content-Type: text/plain; charset=us-ascii
> >
> > Hi, using versions 2.2.2 and 2.4.0 in gentoo I found that previous working templates (with version 2.0.1) do not longer work. The error point out to encoding (ascii not being good for some chars). After posting to tahchee mail list, I included encoding directives like suggested in the manual:
> >
> > #encoding: UTF-8
> >
> > but they were not recognized. It finally seems that I could identify the problem to the file Compiler.py. The line looking for the encoding is using re.match rather than re.search
> > I would think that re.match should work but in fact it does not. 
> >
> > I corrected the problem in my system by applying the simple patch that is included at the end. Is it correct? Or the problem shows from some other issue that I am not seeing?
> >
> > Another question: The behavior of cheetah has changed from version 2.0.1 . Previously I did not have any encoding defined and it just worked. Now I have to include a line with the encoding in each file to make it work. Is there another way to do it (like defining a global encoding)?
> >
> >
> > Cheers, Juan
> >
> > Info:
> > Cheetah versions: 2.2.2 and 2.4.0
> > OS: gentoo linux
> > Python: 2.6.1, 2.6.2 
> >
> >
> > Patch follows:
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > --- /root/Compiler.py	2009-11-24 23:21:24.000000000 -0300
> > +++ Compiler.py	2009-11-24 23:20:03.000000000 -0300
> > @@ -1569,7 +1569,7 @@
> >
> >          else:
> >              unicodeMatch = unicodeDirectiveRE.search(source)
> > -            encodingMatch = encodingDirectiveRE.match(source)
> > +            encodingMatch = encodingDirectiveRE.search(source)
> >              if unicodeMatch:
> >                  if encodingMatch:
> >                      raise ParseError(
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >       
> >
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Tue, 24 Nov 2009 19:03:44 -0800
> > From: [email protected]
> > Subject: Re: [Cheetahtemplate-discuss] encoding problem
> > To: Juan Fiol <[email protected]>
> > Cc: [email protected]
> > Message-ID: <[email protected]>
> > Content-Type: text/plain; charset="us-ascii"
> >
> >
> > On Tue, 24 Nov 2009, Juan Fiol wrote:
> >
> >   
> >> Hi, using versions 2.2.2 and 2.4.0 in gentoo I found that previous working templates (with version 2.0.1) do not longer work. The error point out to encoding (ascii not being good for some chars). After posting to tahchee mail list, I included encoding directives like suggested in the manual:
> >>
> >> #encoding: UTF-8
> >>     
> > Are you referring to the *compiled* templates or the uncompiled templates? If
> > it's the former than I might need to bump the latest compatible version tuple.
> > If it's the latter, are you using encoded strings? If I remember correctly
> > there should be safe-guards in place to try to perform a best effort decoding
> > of strings (as per usual, a traceback would be helpful here with relevant
> > data).
> >
> >   
> >> but they were not recognized. It finally seems that I could identify the problem to the file Compiler.py. The line looking for the encoding is using re.match rather than re.search
> >> I would think that re.match should work but in fact it does not. 
> >>
> >> I corrected the problem in my system by applying the simple patch that is included at the end. Is it correct? Or the problem shows from some other issue that I am not seeing?
> >>     
> > Looks simple enough, do you have a test case that I can add to our
> > regression tests?
> >
> >   
> >> Another question: The behavior of cheetah has changed from version 2.0.1 . Previously I did not have any encoding defined and it just worked. Now I have to include a line with the encoding in each file to make it work. Is there another way to do it (like defining a global encoding)?
> >>
> >>     
> > I'll have to get back to you on the global encoding thing, the basis of this
> > change was to resolve a number of outstanding unicode issues with previous
> > versions of Cheetah, by making all internal strings to Cheetah unicode objects,
> > instead of a mix of encoded strings and unicode objects, etc.
> >
> > Cheers,
> > -R. Tyler Ballance
> > --------------------------------------
> >  Jabber: [email protected]
> >  GitHub: http://github.com/rtyler
> > Twitter: http://twitter.com/agentdero
> >    Blog: http://unethicalblogger.com
> >
> > -------------- next part --------------
> > A non-text attachment was scrubbed...
> > Name: not available
> > Type: application/pgp-signature
> > Size: 198 bytes
> > Desc: not available
> >
> > ------------------------------
> >
> > Message: 3
> > Date: Wed, 25 Nov 2009 02:02:36 -0800 (PST)
> > From: Juan Fiol <[email protected]>
> > Subject: Re: [Cheetahtemplate-discuss] encoding problem
> > To: [email protected]
> > Cc: [email protected]
> > Message-ID: <[email protected]>
> > Content-Type: text/plain; charset=iso-8859-1
> >
> > Hi, thanks for the prompt answer. I have almost no experience with Cheetah. I am currently using it through tahchee and only for a small set of webpages. So, I am not sure even if my diagnosis is correct.
> >
> > The problem appears when I do not have the encoding line in both *compiled* templates that used accented words or words like "Espa?ol" and in *uncompiled* templates that included accented words (in spanish).
> >
> > I've tried including the encoding line in both without success until I changed re.match to re.search
> > I could send my templates, or simplified versions of them, but they are really for use with tahchee. It probably would be better if I can generate a couple of examples in cheetah templates. However, I don't even know the difference or how to use cheetah directly yet. Also, I am rather busy these days, so it can take me three or four days.
> >
> > After changed the file Compiler.py and including the encoding line in all relevant files everything worked well.
> >
> > A python trace (error message) corresponds to cheetah not finding the encoding. It follows an example:
> > +++++++++++++++++++++++++++++++++++++++++++++++
> >   File "/usr/lib/python2.6/site-packages/Cheetah/Compiler.py", line 1588, in __init__
> >     source = unicode(source)
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 84: ordinal not in range(128)
> > make: *** [local] Error 1
> > +++++++++++++++++++++++++++++++++++++++++++++++
> >
> > Cheers, Juan
> > --- On Wed, 11/25/09, [email protected] <[email protected]> wrote:
> >
> >   
> >> From: [email protected] <[email protected]>
> >> Subject: Re: [Cheetahtemplate-discuss] encoding problem
> >> To: "Juan Fiol" <[email protected]>
> >> Cc: [email protected]
> >> Date: Wednesday, November 25, 2009, 3:03 AM
> >>
> >> On Tue, 24 Nov 2009, Juan Fiol wrote:
> >>
> >>     
> >>> Hi, using versions 2.2.2 and 2.4.0 in gentoo I found
> >>>       
> >> that previous working templates (with version 2.0.1) do not
> >> longer work. The error point out to encoding (ascii not
> >> being good for some chars). After posting to tahchee mail
> >> list, I included encoding directives like suggested in the
> >> manual:
> >>     
> >>> #encoding: UTF-8
> >>>       
> >> Are you referring to the *compiled* templates or the
> >> uncompiled templates? If
> >> it's the former than I might need to bump the latest
> >> compatible version tuple.
> >> If it's the latter, are you using encoded strings? If I
> >> remember correctly
> >> there should be safe-guards in place to try to perform a
> >> best effort decoding
> >> of strings (as per usual, a traceback would be helpful here
> >> with relevant
> >> data).
> >>
> >>     
> >>> but they were not recognized. It finally seems that I
> >>>       
> >> could identify the problem to the file Compiler.py. The line
> >> looking for the encoding is using re.match rather than
> >> re.search
> >>     
> >>> I would think that re.match should work but in fact it
> >>>       
> >> does not. 
> >>     
> >>> I corrected the problem in my system by applying the
> >>>       
> >> simple patch that is included at the end. Is it correct? Or
> >> the problem shows from some other issue that I am not
> >> seeing?
> >>
> >> Looks simple enough, do you have a test case that I can add
> >> to our
> >> regression tests?
> >>
> >>     
> >>> Another question: The behavior of cheetah has changed
> >>>       
> >> from version 2.0.1 . Previously I did not have any encoding
> >> defined and it just worked. Now I have to include a line
> >> with the encoding in each file to make it work. Is there
> >> another way to do it (like defining a global encoding)?
> >>     
> >>>       
> >> I'll have to get back to you on the global encoding thing,
> >> the basis of this
> >> change was to resolve a number of outstanding unicode
> >> issues with previous
> >> versions of Cheetah, by making all internal strings to
> >> Cheetah unicode objects,
> >> instead of a mix of encoded strings and unicode objects,
> >> etc.
>   #encoding UTF-8
> ## comment
> 
> <!-- It works if there is nothing before the #encoding line, not even a comment -->
> 
> 
> #def title
> Example of using encoding
> #end def
> #def htTitle
> $title
> #end def
> 
> #def body
> <b>It seems that it works only if there is nothing before the encoding line, not even a comment</b>
> <p><b>Test:</b> Hola Amigos  ??Est??n bien? </p>
> #end def
> 
> <HTML><HEAD>
> <TITLE>$title</TITLE>
> <meta content="text/html; charset=utf-8" http-equiv="content-type" />
> </HEAD><BODY>
> <H1>$htTitle</H1>
> $body
> </BODY></HTML>

> --- /root/Compiler.py	2009-12-27 18:19:17.000000000 -0300
> +++ Compiler.py	2009-12-27 18:21:21.000000000 -0300
> @@ -1556,7 +1556,7 @@
>          
>          else:
>              unicodeMatch = unicodeDirectiveRE.search(source)
> -            encodingMatch = encodingDirectiveRE.match(source)
> +            encodingMatch = encodingDirectiveRE.search(source)
>              if unicodeMatch:
>                  if encodingMatch:
>                      raise ParseError(

------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev

_______________________________________________
Cheetahtemplate-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.12 (GNU/Linux)

iEYEARECAAYFAktDdI4ACgkQFCbH3D9R4W8TTQCfX4rmPp8/FMGevJJcfdo8cOh5
25EAn0VtLkZX1b5X0e/XxKxqtAsIgGMd
=5Ip1
-----END PGP SIGNATURE-----