Re: First strawman for UTF-8 headers proposal

>>> If we're talking about "trivial changes to existing code", then yes,  
>>> the change is trivial: You add 128-255 to comment, atom, and  
>>> quoted-string (or more specifically in 2822, atext, ctext, dtext,  
>>> qtext, and text) and you're done. You can still treat the field  
>>> contents as octets. And in fact, if your code is just looking for  
>>> specials and has an 'else' clause for all the other octets, it might  
>>> need no coding changes at all.
>>
>> yes, this will work in some cases, though you might get bitten if  
>> some kinds of atoms (or atext, whatever) can contain utf-8 and other  
>> kinds cannot.  it even appears to work for gb18030.
>
> No, it does not at all work for gb18030. GB uses virtually all of the
> US-ASCII bytes not only for denoting US-ASCII characters, but also in
> second (or fourth) position of when encoding other characters.
> See e.g. the section "Structure" in
> http://www-106.ibm.com/developerworks/unicode/library/u-china.html? 
> dwzone=unicode

you're right - I misread this earlier.  the four octet sequences are  
okay because the 2nd and 4th positions are in the range 30-39 (ascii  
digits).  but the two octet sequences can have values from 40-7e as  
their 2nd octet, which includes several 2822 specials.

even if you're scanning utf-8 you can't just scan individual octets if  
any of the characters outside the repertoire have special meaning -  
e.g. if full width at sign is taken as an equivalent for '@' then the  
scanner needs to be able to recognize this as a multiple octet  
character.