Re: Dangerous locale patch to 1.1.3

Jason Rennie wrote:
> If I understand correctly, setlocale(LC_CTYPE, "") will look to the 
> environment variable LC_CTYPE for the character encoding instead of the 
> default, "C".  Is this right?

Yes.

> What happens if you don't have LC_CTYPE set?

If LC_CTYPE is not set, LANG is tried.  If LANG is not set, I assume setlocale 
fails.  And according to the manual page: "If setlocale fails, a null pointer is 
returned and the program's locale is not changed."

This is why I took the exceptional step of not checking the return value.  If 
the locale is set, things will work out.  If it is not, "C" will remain as the 
locale.

A smarter implementation would do some checking on the result of setlocale, but 
what exactly "some checking" boils down to is unclear to me.

The can of worm I am poking my fingers into right now is actually the matter of 
interpreting encoded text.  The curren tokenizer returns consistent results for 
most encodings, but it is pretty "dumb".  A smarter tokenizer would decode 
different transfer encodings before tokenizing.  However, any decoding of emails 
adds complexity and introduces security risks.  Also, what is the "right" 
decoding?  Many emails are incorrectly encoded, spam in particular.  The 
tokenizer should generate special features for filing in such cases, as 
incorrect encodings probably are strong features for categorization.

In my particular case, as noted before, I receive 8bit encoded emails mostly 
from valid sources.  Being able to tokenize such emails is an advantage. 
However, the setting of "LC_CTYPE" would ideally depend on information from the 
actual email, not [only] on my locale settings.

For now, though, I'm considering simply taking my patched version into use 
without even rebuilding my .idata file.  The results would probably improve 
detection of certain nonspam categories of email.  7-bit encoding such as 
quoted-printable and base64 would not be affected, because of my choice of 
LC_CTYPE locale.

I happen to have a good argument against taking my patched version into use - I 
get hardly any false positives anyway.  Improving nonspam detection is thus not 
a really big priority.  Detecting the [increasing number of] base64 encoded spam 
messages I get would be a much higher priority!

> Let us know how it goes.  I think we should understand what's going to 
> happen before applying the patch.

I understand what happens in my case.  The problem is, I cannot speak for all 
the different locales available.  Under some locales, it is conceivable that 
"isalpha" behavior is radically affected.  Although I do believe that the 
iso-8859 locales all would behave "decently", as they include ascii as the <127 
subset.

Another really problematic issue is the .idata file.  Currently it is strictly 
interpreted in the "C" locale.  Any change to LC_CTYPE would make .idata depend 
on the locale in question.  The effects are potentially as far-reaching as that 
of changing the lexing options.  That is, a complete rebuild of .idata is 
required.  Most users don't have that option.  I, for one, don't archive all my 
email, only selected parts of it.  Any rebuild of .idata yields a similar, but 
still different, version of .idata.

Camillo