Re: encoding problem
"R. Tyler Ballance" <[email protected]> Wed, 27 Jan 2010 20:44:08 -0800
| Newsgroups | gmane.comp.python.cheetah |
|---|---|
| Message-ID | <[email protected]> |
On Tue, 05 Jan 2010, R. Tyler Ballance wrote: > Interesting, thanks for digging into this; I've applied your patch locally and > I will write a test or two for it this weekend and commit it. Just to follow-up on this thread, I noticed that I hadn't committed this patch when I sat down to work on some Cheetah work tonight. I've gone ahead and added a test, committed (5d93154) and pushed the change. It will be included in Cheetah v2.4.2 Thanks for catching the issue :) > > > From: Juan Fiol <[email protected]> > > > Subject: [Cheetahtemplate-discuss] encoding problem > > > To: [email protected] > > > Message-ID: <[email protected]> > > > Content-Type: text/plain; charset=us-ascii > > > > > > Hi, using versions 2.2.2 and 2.4.0 in gentoo I found that previous working templates (with version 2.0.1) do not longer work. The error point out to encoding (ascii not being good for some chars). After posting to tahchee mail list, I included encoding directives like suggested in the manual: > > > > > > #encoding: UTF-8 > > > > > > but they were not recognized. It finally seems that I could identify the problem to the file Compiler.py. The line looking for the encoding is using re.match rather than re.search > > > I would think that re.match should work but in fact it does not. > > > > > > I corrected the problem in my system by applying the simple patch that is included at the end. Is it correct? Or the problem shows from some other issue that I am not seeing? > > > > > > Another question: The behavior of cheetah has changed from version 2.0.1 . Previously I did not have any encoding defined and it just worked. Now I have to include a line with the encoding in each file to make it work. Is there another way to do it (like defining a global encoding)? > > > > > > > > > Cheers, Juan > > > > > > Info: > > > Cheetah versions: 2.2.2 and 2.4.0 > > > OS: gentoo linux > > > Python: 2.6.1, 2.6.2 > > > > > > > > > Patch follows: > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > --- /root/Compiler.py 2009-11-24 23:21:24.000000000 -0300 > > > +++ Compiler.py 2009-11-24 23:20:03.000000000 -0300 > > > @@ -1569,7 +1569,7 @@ > > > > > > else: > > > unicodeMatch = unicodeDirectiveRE.search(source) > > > - encodingMatch = encodingDirectiveRE.match(source) > > > + encodingMatch = encodingDirectiveRE.search(source) > > > if unicodeMatch: > > > if encodingMatch: > > > raise ParseError( > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------ > > > > > > Message: 2 > > > Date: Tue, 24 Nov 2009 19:03:44 -0800 > > > From: [email protected] > > > Subject: Re: [Cheetahtemplate-discuss] encoding problem > > > To: Juan Fiol <[email protected]> > > > Cc: [email protected] > > > Message-ID: <[email protected]> > > > Content-Type: text/plain; charset="us-ascii" > > > > > > > > > On Tue, 24 Nov 2009, Juan Fiol wrote: > > > > > > > > >> Hi, using versions 2.2.2 and 2.4.0 in gentoo I found that previous working templates (with version 2.0.1) do not longer work. The error point out to encoding (ascii not being good for some chars). After posting to tahchee mail list, I included encoding directives like suggested in the manual: > > >> > > >> #encoding: UTF-8 > > >> > > > Are you referring to the *compiled* templates or the uncompiled templates? If > > > it's the former than I might need to bump the latest compatible version tuple. > > > If it's the latter, are you using encoded strings? If I remember correctly > > > there should be safe-guards in place to try to perform a best effort decoding > > > of strings (as per usual, a traceback would be helpful here with relevant > > > data). > > > > > > > > >> but they were not recognized. It finally seems that I could identify the problem to the file Compiler.py. The line looking for the encoding is using re.match rather than re.search > > >> I would think that re.match should work but in fact it does not. > > >> > > >> I corrected the problem in my system by applying the simple patch that is included at the end. Is it correct? Or the problem shows from some other issue that I am not seeing? > > >> > > > Looks simple enough, do you have a test case that I can add to our > > > regression tests? > > > > > > > > >> Another question: The behavior of cheetah has changed from version 2.0.1 . Previously I did not have any encoding defined and it just worked. Now I have to include a line with the encoding in each file to make it work. Is there another way to do it (like defining a global encoding)? > > >> > > >> > > > I'll have to get back to you on the global encoding thing, the basis of this > > > change was to resolve a number of outstanding unicode issues with previous > > > versions of Cheetah, by making all internal strings to Cheetah unicode objects, > > > instead of a mix of encoded strings and unicode objects, etc. > > > > > > Cheers, > > > -R. Tyler Ballance > > > -------------------------------------- > > > Jabber: [email protected] > > > GitHub: http://github.com/rtyler > > > Twitter: http://twitter.com/agentdero > > > Blog: http://unethicalblogger.com > > > > > > -------------- next part -------------- > > > A non-text attachment was scrubbed... > > > Name: not available > > > Type: application/pgp-signature > > > Size: 198 bytes > > > Desc: not available > > > > > > ------------------------------ > > > > > > Message: 3 > > > Date: Wed, 25 Nov 2009 02:02:36 -0800 (PST) > > > From: Juan Fiol <[email protected]> > > > Subject: Re: [Cheetahtemplate-discuss] encoding problem > > > To: [email protected] > > > Cc: [email protected] > > > Message-ID: <[email protected]> > > > Content-Type: text/plain; charset=iso-8859-1 > > > > > > Hi, thanks for the prompt answer. I have almost no experience with Cheetah. I am currently using it through tahchee and only for a small set of webpages. So, I am not sure even if my diagnosis is correct. > > > > > > The problem appears when I do not have the encoding line in both *compiled* templates that used accented words or words like "Espa?ol" and in *uncompiled* templates that included accented words (in spanish). > > > > > > I've tried including the encoding line in both without success until I changed re.match to re.search > > > I could send my templates, or simplified versions of them, but they are really for use with tahchee. It probably would be better if I can generate a couple of examples in cheetah templates. However, I don't even know the difference or how to use cheetah directly yet. Also, I am rather busy these days, so it can take me three or four days. > > > > > > After changed the file Compiler.py and including the encoding line in all relevant files everything worked well. > > > > > > A python trace (error message) corresponds to cheetah not finding the encoding. It follows an example: > > > +++++++++++++++++++++++++++++++++++++++++++++++ > > > File "/usr/lib/python2.6/site-packages/Cheetah/Compiler.py", line 1588, in __init__ > > > source = unicode(source) > > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 84: ordinal not in range(128) > > > make: *** [local] Error 1 > > > +++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > Cheers, Juan > > > --- On Wed, 11/25/09, [email protected] <[email protected]> wrote: > > > > > > > > >> From: [email protected] <[email protected]> > > >> Subject: Re: [Cheetahtemplate-discuss] encoding problem > > >> To: "Juan Fiol" <[email protected]> > > >> Cc: [email protected] > > >> Date: Wednesday, November 25, 2009, 3:03 AM > > >> > > >> On Tue, 24 Nov 2009, Juan Fiol wrote: > > >> > > >> > > >>> Hi, using versions 2.2.2 and 2.4.0 in gentoo I found > > >>> > > >> that previous working templates (with version 2.0.1) do not > > >> longer work. The error point out to encoding (ascii not > > >> being good for some chars). After posting to tahchee mail > > >> list, I included encoding directives like suggested in the > > >> manual: > > >> > > >>> #encoding: UTF-8 > > >>> > > >> Are you referring to the *compiled* templates or the > > >> uncompiled templates? If > > >> it's the former than I might need to bump the latest > > >> compatible version tuple. > > >> If it's the latter, are you using encoded strings? If I > > >> remember correctly > > >> there should be safe-guards in place to try to perform a > > >> best effort decoding > > >> of strings (as per usual, a traceback would be helpful here > > >> with relevant > > >> data). > > >> > > >> > > >>> but they were not recognized. It finally seems that I > > >>> > > >> could identify the problem to the file Compiler.py. The line > > >> looking for the encoding is using re.match rather than > > >> re.search > > >> > > >>> I would think that re.match should work but in fact it > > >>> > > >> does not. > > >> > > >>> I corrected the problem in my system by applying the > > >>> > > >> simple patch that is included at the end. Is it correct? Or > > >> the problem shows from some other issue that I am not > > >> seeing? > > >> > > >> Looks simple enough, do you have a test case that I can add > > >> to our > > >> regression tests? > > >> > > >> > > >>> Another question: The behavior of cheetah has changed > > >>> > > >> from version 2.0.1 . Previously I did not have any encoding > > >> defined and it just worked. Now I have to include a line > > >> with the encoding in each file to make it work. Is there > > >> another way to do it (like defining a global encoding)? > > >> > > >>> > > >> I'll have to get back to you on the global encoding thing, > > >> the basis of this > > >> change was to resolve a number of outstanding unicode > > >> issues with previous > > >> versions of Cheetah, by making all internal strings to > > >> Cheetah unicode objects, > > >> instead of a mix of encoded strings and unicode objects, > > >> etc. > > #encoding UTF-8 > > ## comment > > > > <!-- It works if there is nothing before the #encoding line, not even a comment --> > > > > > > #def title > > Example of using encoding > > #end def > > #def htTitle > > $title > > #end def > > > > #def body > > <b>It seems that it works only if there is nothing before the encoding line, not even a comment</b> > > <p><b>Test:</b> Hola Amigos ??Est??n bien? </p> > > #end def > > > > <HTML><HEAD> > > <TITLE>$title</TITLE> > > <meta content="text/html; charset=utf-8" http-equiv="content-type" /> > > </HEAD><BODY> > > <H1>$htTitle</H1> > > $body > > </BODY></HTML> > > > --- /root/Compiler.py 2009-12-27 18:19:17.000000000 -0300 > > +++ Compiler.py 2009-12-27 18:21:21.000000000 -0300 > > @@ -1556,7 +1556,7 @@ > > > > else: > > unicodeMatch = unicodeDirectiveRE.search(source) > > - encodingMatch = encodingDirectiveRE.match(source) > > + encodingMatch = encodingDirectiveRE.search(source) > > if unicodeMatch: > > if encodingMatch: > > raise ParseError( > > ------------------------------------------------------------------------------ > This SF.Net email is sponsored by the Verizon Developer Community > Take advantage of Verizon's best-in-class app development support > A streamlined, 14 day to market process makes app distribution fast and easy > Join now and get one step closer to millions of Verizon customers > http://p.sf.net/sfu/verizon-dev2dev > _______________________________________________ > Cheetahtemplate-discuss mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss Cheers, -R. Tyler Ballance -------------------------------------- Jabber: [email protected] GitHub: http://github.com/rtyler Twitter: http://twitter.com/agentdero Blog: http://unethicalblogger.com ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com _______________________________________________ Cheetahtemplate-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
signature.asc
(application/pgp-signature, 198 B)
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) iEYEARECAAYFAkthFhcACgkQFCbH3D9R4W/yZQCfdd9+l4I8lrQPJsSp4pw8+1KR xrMAnRsTXwWMp4ja06jNCg1sMZ5ELw/5 =rqNR -----END PGP SIGNATURE-----