Re: First strawman for UTF-8 headers proposal

On 1/3/04 at 11:05 AM -0500, Keith Moore wrote:

>On Jan 3, 2004, at 10:33 AM, Keld J=F8rn Simonsen wrote:
>
>>Well, UTF-8 is made so that all characters in the 7-bit ASCII range
>>  have the same codes as in ASCII, so if your grammar only has ASCII
>>meta-characters, then you can parse the UTF string as ASCII.
>
>no you can't, because none of the octets in a multiple-octet utf-8=20
>character are valid components of an atom.

What? Your original statement to which Keld was responding was:

At 20:35 03/11/30 -0500, Keith Moore wrote:

>One simple example.  Bernstein and others have pointed out that it's=20
>easier to parse header fields with address lists from the right to=20
>the left rather than from the left to the right, because this=20
>requires less lookahead.  It's still possible to do this with UTF-8=20
>(particularly if you do lexical analysis left-to-right and parsing=20
>right-to-left), but it's probably not a trivial change to existing=20
>code.

If we're talking about "trivial changes to existing code", then yes,=20
the change is trivial: You add 128-255 to comment, atom, and=20
quoted-string (or more specifically in 2822, atext, ctext, dtext,=20
qtext, and text) and you're done. You can still treat the field=20
contents as octets. And in fact, if your code is just looking for=20
specials and has an 'else' clause for all the other octets, it might=20
need no coding changes at all.

Of course, if you want to treat the field as characters, you'll need=20
some buffering for UTF-8, but you can still parse right-to-left: Any=20
octet with '10' in the high two bits is a trailing octet in a UTF-8=20
sequence, and any octet with '11' in the high two bits is the first=20
octet in a UTF-8 sequence. That change is non-trivial, but that's not=20
what you asked.

pr
--=20
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102