charset for FSPageTemplate

Alexis Roda <[email protected]> Thu, 06 Dec 2007 14:27:19 +0100

Newsgroups	gmane.comp.web.zope.silva.devel
Message-ID	<[email protected]>

Hi,
in a site I'm working on I keep all presentation elements (PTs and DTML) 
in a Folder called skin. Yesterday I decided to switch the skin Folder 
to a Filesystem Directory View folder (in order to manage it through 
subversion) and silva started throwing UnicodeDecodeError exceptions.

Comparing the FSPageTemplate.py bundled with Silva-2.0.1 with the 
version bundled with CMF-2.1 it seems like silva's one is missing some 
encoding detection code in the _readFile() method. I have ported the 
changes from CMF to Silva and, after defining the charset in the 
metadata file, the site is working fine again.

I don't know if this is "the right solution" but it seems to solve my 
issue. If somebody cares I have attached a patch.



Regards
Alexis Roda

_______________________________________________
silva-dev mailing list
silva-dev-IAPFreCvJWM6s/[email protected]
https://lists.infrae.com/mailman/listinfo/silva-dev

fspagetemplate.patch (text/x-patch, 2 KB)

--- /home/alex/tmp/fsdv/FileSystemSite/FSPageTemplate.py	2006-03-13 15:29:05.000000000 +0100
+++ FSPageTemplate.py	2007-12-06 13:36:55.000000000 +0100
@@ -33,8 +33,10 @@
 from FSObject import FSObject
 from utils import _setCacheHeaders, _checkConditionalGET
 from utils import expandpath
 
+from Products.PageTemplates.utils import encodingFromXMLPreamble, charsetFromMetaEquiv
+
 xml_detect_re = re.compile('^\s*<\?xml\s+(?:[^>]*?encoding=["\']([^"\'>]+))?')
 _marker = []  # Create a new marker object.
 
 
@@ -90,16 +92,33 @@
             # type is initialized as text/html by default, so we only
             # attempt further detection if the default is encountered.
             # One previous misbehavior remains: It is not possible to
             # force a text./html type if parsing detects it as XML.
+            encoding = None
             if getattr(self, 'content_type', 'text/html') == 'text/html':
                 xml_info = xml_detect_re.match(data)
                 if xml_info:
                     # Smells like xml
                     # set "content_type" from the XML declaration
                     encoding = xml_info.group(1) or 'utf-8'
                     self.content_type = 'text/xml; charset=%s' % encoding
 
+            if encoding is None:
+                charset = getattr(self, 'charset', None)
+                if charset is None:
+                    if self.content_type.startswith('text/html'):
+                        charset = charsetFromMetaEquiv(data) or 'iso-8859-15'
+                    elif self.content_type.startswith('text/xml'):
+                        charset = encodingFromXMLPreamble(data)
+                    else:
+                        raise ValueError('Unsupported content-type: %s' % self.content_type)
+
+                if not isinstance(data, unicode):
+                    data = unicode(data, charset)
+            else:
+                if not isinstance(data, unicode):
+                    data = unicode(data, encoding)
+
             self.write(data)
 
     security.declarePrivate('read')
     def read(self):