Re: Internationalization support (was: No effect from -h option?)

[email protected] said:
> All I'm trying to say is that it would be useful to be able to say
> "use this list of words instead of the default stoplist", if indeed
> the stoplist is deemed a useful feature. 

Oh, sorry.  Yes, I agree.  That would be a useful feature :)  In fact, 
Andrew McCallum has written code to do this and much of ifile is based on 
his libbow code.  You could probably just cut 'n paste to add this 
functionality.  See libbow at

  http://www-2.cs.cmu.edu/~mccallum/bow/

See the option "--replace-stoplist-file"

[email protected] said:
> (Does ifile grok the standard POSIX locales?)

The quick answer to your question is: I don't know.  Much of the lexing 
code I took from Andrew McCallum's libbow.  The functions that check for 
character types appear to call POSIX functions (search for ifile_isalpha 
and ifile_isgraph in lex-simple.c)

[email protected] said:
>  $ ifile -p -k <<HERE
>  > Åk över älven!
>  > HERE
>  Reading message from standard input...
>  (1,Ék) (1,ver) (1,lven)  

I don't know what's going on here (why it's converting Åk to Ék).  If you 
can figure it out, I'd be happy to apply a patch :)

[email protected] said:
>  $ ifile -p -k -w <<HERE
>  > Åk över älven!
>  > HERE
>  Reading message from standard input...
>  (1,Ék) (1,ver) (1,lven) 
> This doesn't really rhyme with my interpretation of the description of
> the -w option, either. 

-w invokes the "white" lexer.  You can find the definition at the bottom 
of lex-simple.c (search for _ifile_white_lexer).  Looks like it requires 
an alphabetic character to begin each word.  That's why the ö and ä are 
being chopped.

Jason D. M. Rennie
MIT AI Lab
[email protected]
(617) 253-5339
http://www.ai.mit.edu/~jrennie/