Re: new patch for bug 86999 - Hebrew support for Universal (All) Autodetect

Shy Shalom wrote:
> I have added a new patch for bug 86999 featuring full hebrew identification, 
> logical and visual.

> currently I am looking for people to review my patch and make comments and 
> suggestions.
> thanks to anyone willing to help

Great job. It would excellent if we would be able to simply duplicate 
some the job done and add also arabic support (bug 265030).

I might be slightly unrelated, but I'm thinking about implementing some 
changes related to FilterWithoutEnglishLetters.
That kind of filtering is OK for single byte encodings, but it clearly 
causes very annoying case of mis-detection of multi-byte encodings.
I'm referring to bug 168526 "Windows-1252 detected as Shift_JIS", as 
well as bug 181344 "Universal auto-detector detects western page as 
gb18030" and  probably also bug 285435 "ISO-8859-1 page detected as BIG-5".

Also the code for the auto-detection looks to me really unefficient, 
some filtering is done several times (but changing it to something 
really effective would mean very significant changes in the 
auto-detector architecture).
There are significant hints that as a result auto-detection adds 10 to 
15% time to pageload which is clearly more than should be :
https://bugzilla.mozilla.org/show_bug.cgi?id=248304#c30
and
https://bugzilla.mozilla.org/show_bug.cgi?id=234683

I have locally too a micro patch to UniversalChardetTest.cpp to make it 
work again, so you can test the result of changes to universal detector 
with just a small command line utility, and not restarting a browser 
session.
The patch leaves the coding in UniversalChardetTest.cpp utterly outdated 
but at least with the change it runs.