Proposed work for Universal Charset Detector

John Gardiner Myers <[email protected]> Thu, 20 Oct 2005 14:07:39 -0700
Newsgroups gmane.comp.mozilla.internationalization
Organization Another Netscape Collabra Server User
Message-ID <[email protected]>
I need to adapat the Universal Charset Detector for use in an email 
application, SpamAssassin.  I am proposing to do the follwing work:

1) Further separate the base charset detector code from the xpcom glue. 
  The base code would be compiled into a static library which would then 
be linked together with the xpcom glue into a shared library.

2) Modify UniversalChardetTest so that it links against the static 
library, removing the dependency on xpcom.

3) Create the framwork for a regression test suite.

4) Fix the outstanding detection bugs.  The list I have is 168526, 
178495, 181344, 177505, 285435, 301915, 306224, and 306272.

5) Extend class nsUniversalDetector to take an optional charset label 
from its caller.  In email applications, the caller would pass in a 
value derived from the MIME charset label.  Should the confidence of the 
detection be sufficiently low, this label could affect the eventual 
report.  (In email, it is not uncommon for the MIME charset label to be 
incorrect.)

I will also be creating and submitting to CPAN a Perl module that 
exposes the universal charset detector.