[ZCM] [ZC] 2285/ 3 Comment "StructuredText and locale"

Issue #2285 Update (Comment) "StructuredText and locale"
 Status Pending, Zope/bug medium
To followup, visit:
  http://www.zope.org/Collectors/Zope/2285

==============================================================
= Comment - Entry #3 by Charlie_X on May 10, 2007 3:46 pm

I've hit essentially the same problem with Zope 2.9. From my investigation it looks like this is down to the fact that no the comparison does not take encoding into account.

STletters has propagates a constant "letters" which while it will reflect the locale setting may have problems because of the encoding. The locale de_DE has an encoding of ISO8859-1. The search function in DocumentClass will not find any non-ascii that is encoded in utf-8 which is why the search will end at the first word with non-ascii characters.

I've tested a workaround for the letters constant:
letters = string.letters.decode("latin-1").encode("utf-8")
Obviously these values should come from the environment with the first encoding being locale.getlocale()[1]. I'm not sure where the second value can be ascertained: default_zpublisher_encoding would be the likely candidate but that is borked on my machine as zope.conf has defualt-zpublisher-encoding.

On the other hand constants for rest_input and _output encodings are available and can be defined in zope.conf so it might make sense to make use of them instead.

However, back to the function in DocumentClass - isn't this excessively restrictive being limited only to what string.letters returns? For example € symbol will always fail no matter what the locale setting but also ® and ™ and presumably other characters that won't be picked up by this simplistic scheme and in any case the same as \w which would also respond to locale?
________________________________________
= Edit - Entry #2 by ajung on Feb 27, 2007 7:26 pm

 Changes: submitter email, importance (critical => medium)
________________________________________
= Request - Entry #1 by jcea on Feb 27, 2007 7:22 pm

"letter" concept in StructuredText is not affected by the "locale" directive in "zope.conf".

This worked fine in Zope 2.9.*

Steps to reproduce:

1. Set your system locale to "C".
2. Configure your "zope.conf" to "es_ES" locale, for example
3. Create a StructuredText page with a ascii-only emphatized text, and another emphatized line with some spanish/european characters.
4. The first line will be emphatized. The second one, not!.

==============================================================