On Tue, 03 Sep 2002 13:14:44 -0400, Jason Rennie <[email protected]>
wrote:
> [email protected] said:
>> The stoplist is apparently specific to English. It would be nice to be
>> able to use this for other languages, too (although I imagine a
>> stoplist is not even particularly useful, seeing as stop words should
>> occur about equally frequently in all categories of messages).
> That would be nice, but I don't have access to stoplists in other
> languages. Also, I don't know whether they help. I know that a stoplist
> is usually helpful for English text, hence it is turned on by default.
All I'm trying to say is that it would be useful to be able to say
"use this list of words instead of the default stoplist", if indeed
the stoplist is deemed a useful feature.
To expand on the parenthesis above, I have not done any research, but
my intuition would be that ignoring very frequent words in the actual
input is enough, and you should not need an explicit stoplist;
furthermore, typical stop words should be insignificant in any case,
meaning their distribution is basically the same in all categories, so
an explicit stoplist again probably doesn't add anything tangible to
the end result.
Another wishlist item would be alternative tokenization. I don't know
how you'd go about tokenizing e.g. Far East writing systems (wild
guess: you need an external program or library to do anything even
remotely useful) but even having control over what is considered an
"alphabetic" in your locale would be a start. (Does ifile grok the
standard POSIX locales?)
/* era */
Content in Latin-1 character set follows, sorry if your mail reader
can't cope:
$ ifile -p -k <<HERE
> Åk över älven!
> HERE
Reading message from standard input...
(1,Ék) (1,ver) (1,lven)
^
Real weird!
$ ifile -p -k -w <<HERE
> Åk över älven!
> HERE
Reading message from standard input...
(1,Ék) (1,ver) (1,lven)
This doesn't really rhyme with my interpretation of the description of
the -w option, either.
--
Too much to say to fit into this .signature anyway: <http://www.iki.fi/era/>
Join the civilized world -- ban spam like we did: <http://www.euro.cauce.org/>
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.