Re: Re: Dangerous locale patch to 1.1.3

Camillo Särs <[email protected]> writes:

> I think it's getting increasingly obvious that ifile as a whole needs several 
> different modules.  Right away, the following spring to mind:

I don't think so.

> - MIME-decoder.  "Text" parts, for your favorite definition of "text", can be 
> tokenized.  Other parts can generate features for filtering, but should not be 
> tokenized as such.
> - MIME interpreter that can flag things like character sets, languages.

... should be left to metamail and the like, which are continuously
developed and updated.

> - Lexer/tokenizer with multi-part email support.

Lexer/tokenizer, yes.  multi-part, no.

> - Database search module.
> - "Scoring" module.
> - Database modification module.

I do not see why these relatively simple things should be split up.
What do we gain from that?

For instance, a possible future feature is to also use proximity of
word.  Such an extension would affect all of the above modules in that
they would have to agree.

> - Optional client-server modules.

No, the point of using ifile in server mode should be that the
database would not have to be reparsed for each message.  Therefore,
IMHO, the tokeniser and database interacteraction should be in server
mode /a priori/, and using ifile as a one-message-only classifier
should just be a special case (maybe a special option to tell ifile
"don't hang around, I won't give you anymore").

> If you want a form that is "completely agnostic of text formatting", I think you 
> will have to implement it in Unicode, using a suitable Unicode canonicalization. 

This is a good idea.  Just my 2¢.
-- 
                                                      Jens Peter Secher
 _jpsecher get2net dk DD6A 05B0 174E BFB2 D4D9 B52E 0EE5 978A FE63 E8A1_

------------------------ Yahoo! Groups Sponsor ---------------------~-->
Get 128 Bit SSL Encryption!
http://us.click.yahoo.com/JjlUgA/vN2EAA/kG8FAA/CefplB/TM
---------------------------------------------------------------------~->

To UNSUBSCRIBE, send an empty message to
[email protected]

ifile web site: http://www.ai.mit.edu/~jrennie/ifile/ 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/