Re: Re: Dangerous locale patch to 1.1.3

Jason Rennie wrote:
> I continually wonder whether The Right Thing would be to strip down ifile
> to a form that is completely agnostic of text formatting.

"Strip down" wouldn't be the right thing.  But making ifile a bit more modular, 
with one module or library being what you describe would make sense.

I think it's getting increasingly obvious that ifile as a whole needs several 
different modules.  Right away, the following spring to mind:

- MIME-decoder.  "Text" parts, for your favorite definition of "text", can be 
tokenized.  Other parts can generate features for filtering, but should not be 
tokenized as such.
- MIME interpreter that can flag things like character sets, languages.
- Lexer/tokenizer with multi-part email support.
- Database search module.
- "Scoring" module.
- Database modification module.
- Optional client-server modules.

If you want a form that is "completely agnostic of text formatting", I think you 
will have to implement it in Unicode, using a suitable Unicode canonicalization. 
The only alternative approach I can come up with is to determine which character 
sets can be "naïvely" tokenized and still yield valid tokens.  I think that most 
character sets work fairly well, provided that the lexer works on 8-bit data and 
knows what characters are not word-parts.

( However, Unicode is an example of an encoding that allows alternative forms 
for many characters.  The following example is not really "useful", but may 
illustrate my point.

These two characters have different Unicode codes:

Å: 0x00C5 - Latin Capital Letter A With Ring Above
Å: 0x212b - Angstrom Sign

Most users would still simply write 0x00c5 instead of 0x212b, as the two 
actually are represented by the same glyph.)

Camillo