Re: isspace() breaks encoding in rfc2047.c on some systems

On Tue, Mar 16, 2010 at 08:40:33PM +0100, Dennis Preiser wrote:
> I encountered the following issue:
> 
> On my system (MacOSX 10.4, powerpc-apple-darwin8.11.0) isspace()
> returns TRUE for '0xa0' (I use an UTF-8 locale). 0xa0 can be the second
> byte of a two byte UTF-8 character, for instance:
> 
> ??	0xc5 0xa0
> 
> This causes the isbetween() macro in rfc2047.c to return TRUE and the
> various encoding functions fail. For quoted-printable and the character
> above in the subject it looks like this:
> 
> Subject: =?UTF-8?Q?=C5?=
> 
> followed by 0xA0. (The internal inews complains about unencoded 8-bit
> chars, which is correct.)
> 
> Here some code to check the behaviour of isspace() (returns TRUE with an
> UTF-8 locale on my system):
> 
> #include <stdio.h>
> #include <ctype.h>
> #include <locale.h>
> 
> int
> main(
> 	void)
> {
> 	unsigned char ch = 0xa0;
> 
> 	setlocale(LC_ALL, "");
> 	printf("isspace(0xa0) = %s\n", isspace(ch) ? "TRUE" : "FALSE");
> 	return 0;
> }
> 
> According to google (quick check with 'isspace(0xa0)'), there seems to
> be some systems whith such a behaviour. MacOSX, (older?) BSD and linux
> with an older glibc version.
> 
> I think we have to decide whether it is worse to be fixed and if so, how.

I'd use a configure check for it (e.g. your code above, but with UTF-8 as
locale in setlocale) to check for broken isspace(). at least when not
crosscompiling this should give us exact results. Thomas? 

> As a preliminary fix, I use the patch below. It replaces the isspace()
> inside isbetween() whith a check for the common white-space characters.

limiting that to the case where the local charset is UTF-8 might be an
option as I don't know what other whitespace chars do exist in other
charsets.

urs
-- 
"Only whimps use tape backup: _real_ men just upload their important stuff
 on ftp, and let the rest of the world mirror it ;)" - Linus