Re: Proposed work for Universal Charset Detector

John Gardiner Myers wrote:
> 1) Further separate the base charset detector code from the xpcom glue. 
>  The base code would be compiled into a static library which would then 
> be linked together with the xpcom glue into a shared library.

Great :-)

> 2) Modify UniversalChardetTest so that it links against the static 
> library, removing the dependency on xpcom.

Could you do another version ? Keeping UniversalChardetTest as an 
example of how to do it with xpcom and have another file with a direct 
link ?

> 3) Create the framwork for a regression test suite.

Shanjian had samples files thta could be used to test regressions.
Simon Montagu  in Mozilla's current i18n team who has a contact with him 
(see https://bugzilla.mozilla.org/show_bug.cgi?id=86999#c23) and I think 
I remember he talked about trying to get them.

> 4) Fix the outstanding detection bugs.  The list I have is 168526, 
> 178495, 181344, 177505, 285435, 301915, 306224, and 306272.

Good list. If you find more, can you add them as dependencies of bug 
264871 ?

> 5) Extend class nsUniversalDetector to take an optional charset label 
> from its caller.  In email applications, the caller would pass in a 
> value derived from the MIME charset label.  Should the confidence of the 
> detection be sufficiently low, this label could affect the eventual 
> report.  (In email, it is not uncommon for the MIME charset label to be 
> incorrect.)

I think that point is not something really interesting for Mozilla.

First, in my experience it's quite rare nowadays that the MIME charset 
label is present and incorrect, second doing something like that 
encourages people not to care about correctly labeling email and helps 
propagating that error, when the ultimate aim is that everybody puts 
corrects labels on mail and web pages.

So I don't think your algorithm where you use MIME charset should be in 
the Mozilla code.

But nsUniversalDetector is a virtual class, it's member mCharSetProbers 
that has the detailled results before the selection of one charset is 
protected not private, therefore couldn't you put that modification in 
your class that instanciate nsUniversalDetector and be free to do what 
you want without changing what Mozilla uses ?

I don't see a problem if that code gets in the non-xpcom replacement for 
UniversalChardetTest.

> I will also be creating and submitting to CPAN a Perl module that 
> exposes the universal charset detector.

I foresee it will become popular :-)