[ZCM] [ZC] 2180/11 Resolve "ZPT unicode decode error rendering request values"

Issue #2180 Update (Resolve) "ZPT unicode decode error rendering request values"
 Status Resolved, Zope/bug medium
To followup, visit:
  http://www.zope.org/Collectors/Zope/2180

==============================================================
= Resolve - Entry #11 by philikon on Jan 8, 2007 5:25 pm

 Status: Accepted => Resolved

I assume this means we can close this issue now. Thanks Andreas.
________________________________________
= Comment - Entry #10 by ajung on Jan 8, 2007 4:34 pm

A customized version of zptunicode2.patch is used for the PreferredCharset resolver which will be the default for Zope 2.10.2+
________________________________________
= Comment - Entry #9 by philikon on Jan 8, 2007 1:32 pm

This issue has been, in part at least, addressed by introducing configurable UnicodeDecodeError resolvers for Zope 2.10.2 and above. Currently, it won't resolve those UnicodeDecodeErrors by itself, but you can plug in a utility of your own that allows you to handle those errors gracefully.

Whether or not this is enough to close this issue needs to be determined.
________________________________________
= Comment - Entry #8 by jmeile on Nov 24, 2006 6:41 am

Ok, I put it in the right place. It was my fault that it didn't work. Somewhere in my code I was concatenating a normal string and unicode, so the UnicodeError came.
________________________________________
= Comment - Entry #7 by jmeile on Nov 22, 2006 8:15 am

Any idea on where to include this patch in Zope 2.8.8? I can't upgrade now because I'm using a third party product which depends on this zope version.

I tried to apply this to the ZopeContext class, which is located at:
lib/python/zope/app/pagetemplate/engine.py

But the problem persists.

Best regards
Josef
________________________________________
= Assign - Entry #6 by philikon on Nov 2, 2006 12:55 pm

 Status: Pending => Accepted

 Supporters added: philikon

I'll take that one as I have provided a patch that *seems* to fix certain issues already, anyways. In Seattle, Hanno told me that the patch doesn't do enough yet, though. Hanno, can you please elaborate on that? What's still missing. Are Balasz's test cases covering them all?
________________________________________
= Comment - Entry #5 by philikon on Nov 2, 2006 12:51 pm

Note that zopeuni_newer_test.diff just contains test cases, not my fix.
________________________________________
= Comment - Entry #4 by ree on Oct 11, 2006 1:45 am


Uploaded:  "zopeuni_newer_test.diff"
 - http://www.zope.org/Collectors/Zope/2180/zopeuni_newer_test.diff/view
I have made extra unittests for Zope 2 in the form of the patch that make easier to demonstrate the problem.

All these run on 2.9 and the utf and mixed cases all fail on 2.10. I remember also that philiKON's first patch did not solve all the cases.

In any case please feel free to use them for testing a particular solution. 
________________________________________
= Comment - Entry #3 by philikon on Sep 1, 2006 7:45 am


Uploaded:  "zptunicode2.patch"
 - http://www.zope.org/Collectors/Zope/2180/zptunicode2.patch/view
The first patch did not work well with browsers that send form data encoding in something other than utf-8. The new patch acutally immitates what Zoep 3 does when it decodes form variables: loop through the preferred charsets and use the first one that works. So if it doesn't work utf-8, it will use the next encoding in line (and the browser better specify that in Accept-Charset; if not, the browser is broken).

Note to myself: We currently expect the request to be accessible as self.request. This actually comes from the base class, ZopeContext, but is only set when the expression context actually contains a 'request' variable. If this ends up being used elsewhere (e.g. the CMF's exressions) where 'request' isn't available, we'll have to check for the 'request' attribute first before using it.
________________________________________
= Comment - Entry #2 by philikon on Sep 1, 2006 4:52 am


Uploaded:  "zptunicode.patch"
 - http://www.zope.org/Collectors/Zope/2180/zptunicode.patch/view
This is a very tricky issue. I've been made aware of other cases of the same or similar issue (e.g. in Archetypes).

The problem is that the old ZPT implementation did not use unicode at all. That doesn't mean you couldn't insert unicode into your template. However, because most things in Zope 2 are still 8bit strings (such as d2m's form variable), you would always opt for inserting 8bit strings into templates, never unicode (to avoid mixing unicode and 8bit strings).

Now, Zope 3's ZPT implementation uses unicode, which makes the insertion of 8bit strings impossible, as we see here. This is a BBB foul.

My idea for solving this issue is to try and decode all 8bit strings that come into the template to unicode. While that is a bit magic, it is still predictable (it would use the request's preferred charset, with utf-8 as fallbac). Fortunately, there is a single entry point for text into ZPTs: ITALExpressionContext.evaluateText(). I've rounded up a patch that overrides this method on Zope 2's ITALExpressionContext implementation and does the automatic decoding when it encounters a 'str' object.

With the patch applied, d2m's original issue goes away. We should do more tests (e.g. with Archetypes) and provide unit tests to see if this really fixes the BBB foul.
________________________________________
= Request - Entry #1 by d2m on Aug 30, 2006 6:46 am

create a ZPT with content
  <div tal:replace="request.get('test','')" />

call it like 
  http://localhost:8080/testzpt?test=äöü

Traceback (innermost last):
  Module ZPublisher.Publish, line 115, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 41, in call_object
  Module Shared.DC.Scripts.Bindings, line 313, in __call__
  Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec
  Module Products.PageTemplates.ZopePageTemplate, line 276, in _exec
  Module Products.PageTemplates.PageTemplate, line 89, in pt_render
  Module zope.pagetemplate.pagetemplate, line 117, in pt_render
  Module zope.tal.talinterpreter, line 271, in __call__
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 754, in do_insertStructure_tal
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0: ordinal not in range(128)

alternative:
create a ZPT with content
  <div tal:replace="request/test|nothing" />

call it like 
  http://localhost:8080/testzpt?test=äöü

Traceback (innermost last):
  Module ZPublisher.Publish, line 115, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 41, in call_object
  Module Shared.DC.Scripts.Bindings, line 313, in __call__
  Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec
  Module Products.PageTemplates.ZopePageTemplate, line 276, in _exec
  Module Products.PageTemplates.PageTemplate, line 89, in pt_render
  Module zope.pagetemplate.pagetemplate, line 118, in pt_render
  Module StringIO, line 271, in getvalue
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0: ordinal not in range(128)
==============================================================