Re: [Plone-developers] PLIP suggestion : accents normalization in plone lexicon

Hi,

My PLIP draft :

Title: Plain text search ignores accents.

'''Proposer:''' Thomas Desvenain
'''Seconder:''' Vincent Fretin

== Motivation ==
Most users want the search to ignore accents.

This is a question of comfort :
a search on econometrie should found documents with term "économétrie".

And that would fix an issue, as most users don't use accents with upper case
characters :

For example, a search on 'économétrie' doesn't found a document titled as
"Econométrie"

== Assumptions ==
We will improve plone lexicon so that it normalizes indexed and searched
terms in plain text indexes (ZCTextIndex).
A document with 'Econométrie' and 'économétrie' words will be indexed for
'econometrie' term.
A search on 'économétrie' or 'econometrie' word will search for
'econometrie' index value.

The normalization will be made on the model of document ids generation in
Plone.

To avoid performance issue and extensions at lowest level than Plone,
normalization will be independent of site language.

== Proposal & Implementation ==
We have to add a new Case Normalizer named 'I18n Case Normalizer'.
This normalizer will use plone.i18n tools to generate an ascii string from
any word to normalize.

== Deliverables ==

* Code
- Add a new class in Products.CMFPlone.UnicodeSplitter, register it as 'I18n
Case Normalizer'.
- plone_lexicon will use this.
* Upgrades
- Upgrade plone_lexicon with this normalizer.
- Reindex ZCTextIndex indexes.
* Unit tests
- Test search and found documents containing words 'économétrie',
'Économétrie', 'Econométrie' with criterion 'econometrie' and 'économétrie'.
- Equivalent Unit tests with eastern language

== Risks ==
The main risks are :
- it has to work with all languages, included eastern languages.
- check consequences on general performances.
- indexes have to be updated for backward compatibility.

== Participants ==
Thomas Desvenain
I need volunteers to add tests for eastern languages.

== Progress ==

-------------------------------

NB: what do you think about an
'experimental.plonei18nlexicon' package ?


On Tue, Aug 2, 2011 at 10:30 PM, thomas desvenain <
[email protected]> wrote:

>
> On Sun, Jul 31, 2011 at 11:04 AM, Hanno Schlichting <[email protected]>wrote:
>
>> On Mon, Jun 6, 2011 at 3:08 PM, thomas desvenain
>> <[email protected]> wrote:
>> > Most users want the search to ignore accents
>> >
>> > where "économétrie"
>> > finds "econometrie", "Econométrie", "Économétrie".
>> >
>> > What do you think about a PLIP to give Plone lexicon a casenormalizer
>> > that would use plone.i18n stuff to normalize ZCTextIndex lexical
>> > values ?
>> >
>> > (I can write the PLIP and implement it... but i know it is a major
>> issue)
>>
>> +10 :)
>>
>> The framework team is accepting PLIPs for 4.3 - please get it in :)
>>
>> Great ! :)
>
> So, I will (after i come back from my holiday place).
>
> Thank you !
>
> Thomas
>
>
>> Hanno
>>
>
>
>
> --
> Thomas Desvenain
>
> Téléphone : 09 51 37 35 18
>



-- 
Thomas Desvenain

Téléphone : 09 51 37 35 18

------------------------------------------------------------------------------
Get a FREE DOWNLOAD! and learn more about uberSVN rich system, 
user administration capabilities and model configuration. Take 
the hassle out of deploying and managing Subversion and the 
tools developers use with it. 
http://p.sf.net/sfu/wandisco-dev2dev

_______________________________________________
Plone-i18n mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/plone-i18n