Re: Problem with -8 option

Thomas Schürger wrote:
> One question remains: as can be seen from my database dump, 8 bit
> characters are not converted to lowercase. For example, the German
> uppercase Umlaut characters ÄÖÜ remain like that instead of using the
> lowercase variants äöü. There are other 8 bit uppercase characters in
> the latin1 character set that could require similar lowercasing (e.g.,
> some French characters with accents).
> 
> Would it be a big change to support such non-ASCII lowercasing? Or is
> the current implementation a design philosophy to stay more or less
> independent of the character encoding?

I think this must have something to do with the LANG environment 
variable.  I ran a test tokenizing your email with and without the LANG 
environment variable and seemed to get the correct results.

$ ./spamprobe -8 tokenize ~/sampleemail.txt
  ... characters were upper case
$ env LANG=de_DE.iso88591 ./spamprobe -8 tokenize ~/sampleemail.txt
  ... characters were lower case

Dmitry Koteroff posted to sourceforge that procmail does not pass the 
value of the LANG environment variable to SP.  Maybe that's the cause of 
your problem?

<http://sourceforge.net/tracker/index.php?func=detail&aid=905363&group_id=61201&atid=496457>


Really SP should determing the right character encoding based on the 
headers in each email so that it can properly handle multilingual email. 
  That's something I'd like to add but for now it's best just to ensure 
that LANG is set appropriately for your own primary language.

All the best,
++Brian


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_idt77&alloc_id492&op=click