Re: Re: Dangerous locale patch to 1.1.3

Jens Peter Secher wrote:
>>- MIME-decoder.  "Text" parts, for your favorite definition of "text", can be 
>>tokenized.  Other parts can generate features for filtering, but should not be 
>>tokenized as such.
>>- MIME interpreter that can flag things like character sets, languages.
> 
> ... should be left to metamail and the like, which are continuously
> developed and updated.

Agreed.  Only the interface between the different programs need to be well 
defined to get reliable lexing.  For instance, some representations are not 
suitable for "ifile tokenization" at all.  These cases must be handled 
gracefully.

BTW; can anyone show me how to reliably use metamail to "un-base64" email? 
What I mean is actually "how to use metamail to turn any 'reasonable' 
textual format into something that ifile can tokenize".  Or any other 
similar application for that matter.

My current experience with metamail is that it fails badly on many spam 
messages that intentionally use weird encodings.  I can't have a MTA that 
keeps crashing - it's just begging for someone to find a buffer overflow and 
exploit it.

Also, configuring metamail so that it never tries something stupid requires 
care.  It has a tendency to dig out mailcap files from weird places and 
start spawning viewers all over the place.

>>- Lexer/tokenizer with multi-part email support.
> 
> 
> Lexer/tokenizer, yes.  multi-part, no.

Please elaborate.  If an email contains multiple MIME parts, would you 
expect the "metamail-equivalent" to output just a single body from the 
message?  I would really like ifile to be able to work on the entire 
contents, which means all "printable" mime-parts.

>>- Database search module.
>>- "Scoring" module.
>>- Database modification module.
> 
> I do not see why these relatively simple things should be split up.
> What do we gain from that?

The searching and modification functions of the db are just two facets of 
the same object.  However, in practical terms it often makes sense to 
technically implement a "read-only" search interface.  Many optimizations 
are then possible.

Scoring should be independent of the underlying database architecture, and 
thus is a module (object) of its own.  It should only depend on the database 
interface, not on the implementation.

> No, the point of using ifile in server mode should be that the
> database would not have to be reparsed for each message.

There are many file-based database formats out there that do not require a 
full reparsing of the file every time.  They also implement fast searches. 
The client-server architecture serves systems that implement ifile widely, 
but comes with many complications.

I feel strongly for also keeping a fast command version of ifile around.

>>If you want a form that is "completely agnostic of text formatting", I think you 
>>will have to implement it in Unicode, using a suitable Unicode canonicalization. 
> 
> This is a good idea.  Just my 2¢.

Unfortunately I think that comperehensive unicode libraries are hard to 
find.  Libraries that would need to be cross-platform as well.  But I may be 
wrong.

Camillo
-- 
Camillo Särs <[email protected]>             Aim for the impossible and you
http://www.iki.fi/ged                   will achieve the improbable


------------------------ Yahoo! Groups Sponsor ---------------------~-->
Get 128 Bit SSL Encryption!
http://us.click.yahoo.com/JjlUgA/vN2EAA/kG8FAA/CefplB/TM
---------------------------------------------------------------------~->

To UNSUBSCRIBE, send an empty message to
[email protected]

ifile web site: http://www.ai.mit.edu/~jrennie/ifile/ 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/