Re: PLIP suggestion : accents normalization in plone lexicon

On Sun, Jul 31, 2011 at 5:20 AM, Antonio Carrasco Valero on gmail
<[email protected]> wrote:
> The biggest hassle is to put together the mappings of "similar" characters
> I.e., all the following unicodes could match for each other:

Not really. We already have and ship such a list and use it as part of
plone.i18n. It depends on the http://pypi.python.org/pypi/Unidecode
library, which has a pretty comprehensive list and maps about 46000
characters from the entire Unicode range.

So all we need to do is:

from plone.i18n.normalizer.base import baseNormalize
ascii = baseNormalize('some text')

The baseNormalize function only uses the Unidecode mappings with some
upper limit - as the phonetic mappings for Asian languages aren't good
enough. This makes sense for this use-case as well, as Asian languages
need different approaches for search anyways, like not doing
whitespace delimited splitting.

But thanks for the pointer :)

Hanno

------------------------------------------------------------------------------
Got Input?   Slashdot Needs You.
Take our quick survey online.  Come on, we don't ask for help often.
Plus, you'll get a chance to win $100 to spend on ThinkGeek.
http://p.sf.net/sfu/slashdot-survey