Re: Proposed work for Universal Charset Detector

Jean-Marc Desperrier wrote:

> John Gardiner Myers wrote:
>
>> 5) Extend class nsUniversalDetector to take an optional charset label 
>> from its caller.  In email applications, the caller would pass in a 
>> value derived from the MIME charset label.  Should the confidence of 
>> the detection be sufficiently low, this label could affect the 
>> eventual report.  (In email, it is not uncommon for the MIME charset 
>> label to be incorrect.)
>
>
> I think that point is not something really interesting for Mozilla.

I could see Thunderbird wanting to use it.

>
> First, in my experience it's quite rare nowadays that the MIME charset 
> label is present and incorrect, second doing something like that 
> encourages people not to care about correctly labeling email and helps 
> propagating that error, when the ultimate aim is that everybody puts 
> corrects labels on mail and web pages.

For a long time it has been common for Korean e-mail to be in the EUC-KR 
charset but with a MIME label of ISO-8859-1.  This derived from the use 
of metamail, which decoded MIME encoded-words but ignored the label and 
encoded with a default label of ISO-8859-1.

In my corpus of Japanese spam, there is a substantial portion of 
messages with an incorrect MIME charset label.  Using the charset label 
instead of the detected charset in a spam filtering application actually 
encourages such incorrect labeling, as it provides the spammers an 
effective mechanism to obscure their text from the spam filter.

>
> So I don't think your algorithm where you use MIME charset should be 
> in the Mozilla code.

I don't see that there is a substantial disadvantage to Mozilla of 
having the detector class support a feature that Mozilla itself does not 
use.

The disadvantage to Mozilla of refusing this feature is that it makes it 
much more likely that I will fork the charset detector code.  Mozilla 
would then miss out on the bug fixes and other improvements I make to 
the detection features it does use.

>
> But nsUniversalDetector is a virtual class, it's member 
> mCharSetProbers that has the detailled results before the selection of 
> one charset is protected not private, therefore couldn't you put that 
> modification in your class that instanciate nsUniversalDetector and be 
> free to do what you want without changing what Mozilla uses ?

I don't consider that a tenable approach.  The top-level and group 
prober classes would have to be substantially duplicated, using 
implementation knowledge that really shouldn't be so visible.

>
> I don't see a problem if that code gets in the non-xpcom replacement 
> for UniversalChardetTest.

I don't see exposing the caller-supplied charset feature through xpcom 
until Thunderbird or some other xpcom client asks for it.

>> I will also be creating and submitting to CPAN a Perl module that 
>> exposes the universal charset detector.
>
>
> I foresee it will become popular :-)

It's now available as Encode::Detect.