Re: PLIP suggestion : accents normalization in plone lexicon

Hi All,

I also would like to see the feature available in out-of-the-box Plone 4.x

The biggest hassle is to put together the mappings of "similar" characters
I.e., all the following unicodes could match for each other:
     0x004F: [ 
0x006F,0x00D2,0x00D3,0x00D4,0x00D5,0x00D6,0x00F2,0x00F3,0x00F4,0x00F5,0x00F6,0x00D8,0x00F8,0x014C,0x014D,0x014E,0x014F,0x0150,0x0151,0x0152,0x0153],     
# O

So if anybody implementing this wants to save some time,
I collected

I implemented an ISplitter implementation,
that I apply to application-specific instances of ZCatalog,
indexing elements not in English language
(although it should work also for english).

Note that this is running on Plone 2.5.

This allows to enter search criteria with characters of cyrilic, greek 
and latin languages,
and supports equivalences between letters plain and upper/lower case, 
accentuated/diacritical, or somehow similar.

It has been used inproduction and tested only with narrow unicode pyton 
builds (up to unicode 65535).


Available in the OSOR.eu repository forge, project gvSIG-i18n

http://forge.osor.eu/projects/gvsig-i18n/

TRASplitter.py 
<http://forge.osor.eu/plugins/scmsvn/viewcvs.php/*checkout*/trunk/gvSIGi18n/TRASplitter.py?content-type=text%2Fplain&root=gvsig-i18n>
TRAUnicode.py 
<http://forge.osor.eu/plugins/scmsvn/viewcvs.php/*checkout*/trunk/gvSIGi18n/TRAUnicode.py?content-type=text%2Fplain&rev=189&root=gvsig-i18n>
TRAUnicode_Constants.py 
<http://forge.osor.eu/plugins/scmsvn/viewcvs.php/*checkout*/trunk/gvSIGi18n/TRAUnicode_Constants.py?content-type=text%2Fplain&root=gvsig-i18n>

Antonio Carrasco Valero
[email protected]

Model Driven Development, sl
Valencia España (Spain)
www.ModelDD.org


On 20:59, Lachlan Musicman wrote:
> On Mon, Jun 6, 2011 at 23:08, thomas desvenain
> <[email protected]>  wrote:
>> Hi,
>>
>> Most users want the search to ignore accents
>>
>> where "économétrie"
>> finds "econometrie", "Econométrie", "Économétrie".
> I can understand this, if not use it. My translation students are
> capable, but they have to work on the University's locked down
> systems. As we are in Australia, the keyboard is set to US English,
> with no accents etc. Like I said, they are capable enough to discover
> work arounds, but they are also looking for workflow pace - and being
> able to search accent free would be a boon to their productivity
>
> cheers
> L.
>
>> What do you think about a PLIP to give Plone lexicon a casenormalizer
>> that would use plone.i18n stuff to normalize ZCTextIndex lexical
>> values ?
>>
>> (as lucene latin normalizer does)
>>
>> that would be the occasion to fix a bug, that "économétrie" does'nt
>> find "Économétrie" (plone.i18n stuff manages that, plone lexicon, not)
>>
>> (I can write the PLIP and implement it... but i know it is a major issue)
>>
>> Thanks
>>
>> Thomas
>>
>> --
>> Thomas Desvenain
>>
>> Téléphone : 09 51 37 35 18
>>
>> ------------------------------------------------------------------------------
>> Simplify data backup and recovery for your virtual environment with vRanger.
>> Installation's a snap, and flexible recovery options mean your data is safe,
>> secure and there when you need it. Discover what all the cheering's about.
>> Get your free trial download today.
>> http://p.sf.net/sfu/quest-dev2dev2
>> _______________________________________________
>> Plone-i18n mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/plone-i18n
>>
>
>

------------------------------------------------------------------------------
Got Input?   Slashdot Needs You.
Take our quick survey online.  Come on, we don't ask for help often.
Plus, you'll get a chance to win $100 to spend on ThinkGeek.
http://p.sf.net/sfu/slashdot-survey

_______________________________________________
Plone-i18n mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/plone-i18n