Re: Proposed work for Universal Charset Detector

John Gardiner Myers wrote:
> In my corpus of Japanese spam, there is a substantial portion of 
> messages with an incorrect MIME charset label.

I forgot to add that this is probably quite more frequent with spam, and 
IMO would make a valid rule to highen slightly the spam assassin score.

> [...] Using the charset label 
> instead of the detected charset in a spam filtering application actually 
> encourages such incorrect labeling, as it provides the spammers an 
> effective mechanism to obscure their text from the spam filter.

Well, yes, but to avoid that you don't need to change the current code ?
You just need to run the UCD for all messages, even with a label.

>> So I don't think your algorithm where you use MIME charset should be 
>> in the Mozilla code.
> 
> I don't see that there is a substantial disadvantage to Mozilla of 
> having the detector class support a feature that Mozilla itself does not 
> use.

There's a philosophy of not including code that is not used, and I 
thought the alternative was viable. But after all, if what you add to 
nsUniversalDetector is neglectable.

> The disadvantage to Mozilla of refusing this feature is that it makes it 
> much more likely that I will fork the charset detector code.

I'm presenting you arguments against your option, but it's only a i18n 
peer that could take that decision. Go ahead and ask if you want to 
include it.

>> But nsUniversalDetector is a virtual class, it's member 
>> mCharSetProbers that has the detailled results before the selection of 
>> one charset is protected not private, therefore couldn't you put that 
>> modification in your class that instanciate nsUniversalDetector and be 
>> free to do what you want without changing what Mozilla uses ?
> 
> I don't consider that a tenable approach.  The top-level and group 
> prober classes would have to be substantially duplicated, using 
> implementation knowledge that really shouldn't be so visible.

Hum, I'm beginning to wonder if I understood what you want to do.
IMO the knowledge that you would need is only GetConfidence and 
GetCharSetName from nsCharSetProber and that is public already in 
nsCharSetProber.h.

I'm thinking of an alternative.
What if nsCharSetProber just returns the best charset and it's 
probability in the form of a double ? Is that enough for your needs ?

>> I don't see a problem if that code gets in the non-xpcom replacement 
>> for UniversalChardetTest.
> 
> I don't see exposing the caller-supplied charset feature through xpcom 
> until Thunderbird or some other xpcom client asks for it.

OK.

If you answer, note that I will be gone until tuesday.