Re: Chinese and Japanese stemmeing algorithm
Charlie Hull <[email protected]>
| Newsgroups | gmane.comp.search.snowball |
|---|---|
| Message-ID | <[email protected]> |
On 02/11/2011 10:51, Martin Porter wrote: > Miguel, > > Chinese does not have inflectional endings, so the concept of stemming > is not relevant to that language. The big problem in Chinese is the > division of text to retrieve, and queries, into "words", or at least > into units for indexing and retrieval. (Chinese is of course written > without spaces.) There is a big literature on this in IR, but my own > knowledge is now seriously out of date (*footnote). There are systems > for splitting Chinese text into words. These used to be expensive > proprietary pieces of software, and I always hoped that free versions > would eventually appear. Does anyone else contributing to snowball > discuss have more knowledge on what is currently available? > > Japanese is similar but the problem is even worse. Since there are no > spaces word splitting is required: but thereafter something like > stemming (or lemmatisation) may be applied. Again, there is a big > literature. Hi Miguel, There was a Google Summer of Code project this year on improving Chinese segmentation for the Xapian engine: http://trac.xapian.org/wiki/GSoC2011/ChineseSegmentationAnalysis Maybe this will be of help. There are some CJK segmentation algorithms for Lucene I believe; otherwise people go to commercial companies such as Basistech. Cheers Charlie > > Actually, no futher stemmers are planned at snowball, although we > might get occasional contributions. > > Martin > > (*footnote) for example, Information processing and management vol 35 > (1999) has a special section on Asian language IR pp 421-> > > On Wed, Nov 2, 2011 at 9:56 AM, Miguel Florido > <[email protected]> wrote: >> >> Dear Martin >> >> >> >> We are using one of your projects, the stemming algorithm. Due to our international expansion, we need an stemming algorithm for Chinese and Japanese. As we can see, these two algorithms are not implemented yet, and the http://snowball.tartarus.org/ shows that the last update was on Jul 2010, do you know if is planned to be implemented in a middle/short future? >> >> If not, could you tell us other source where we can get these algorithm?. >> Thanks and best regards. >> Miguel Florido >> Web Developer Junior >> [email protected] >> >> http://www.softonic.com > > _______________________________________________ > Snowball-discuss mailing list > [email protected] > http://lists.tartarus.org/mailman/listinfo/snowball-discuss -- Charlie Hull Flax - Open Source Enterprise Search tel/fax: +44 (0)8700 118334 mobile: +44 (0)7767 825828 web: www.flax.co.uk