Re: First strawman for UTF-8 headers proposal

On Fri, Jan 02, 2004 at 09:53:31PM -0500, Keith Moore wrote:
> 
> 
> On Jan 2, 2004, at 5:20 PM, Martin Duerst wrote:
> 
> >Hello Keith,
> >
> >
> >At 20:35 03/11/30 -0500, Keith Moore wrote:
> >
> >>One simple example.  Bernstein and others have pointed out that it's 
> >>easier to parse header fields with address lists from the right to 
> >>the left rather than from the left to the right, because this 
> >>requires less lookahead.  It's still possible to do this with UTF-8 
> >>(particularly if you do lexical analysis left-to-right and parsing 
> >>right-to-left), but it's probably not a trivial change to existing 
> >>code.
> >
> >Can you give more details?
> 
> yes.  when parsing ASCII you can look at one octet at a time.  so when 
> parsing
> 
> To: Martin Duerst <[email protected]>
> 
> right to left the parser sees ">" then "g", then "r", etc.  as soon as 
> the parser sees ">" it knows that this is a production of the form
> 
> [ phrase ] "<" addr-spec ">"
> 
> (forgive me for using 822 rather than 2822 - I have never memorized the 
> latter)
> 
> if you're parsing utf-8 then you can't look at one octet at a time - 
> you first have to parse octets into characters.  you can do it, but 
> it's more of a pain - for instance, you have to do more checking for 
> boundary conditions.  it's certainly not as simple as something like
> 
> if (ptr <= bufstart)
>    break;
> c = *ptr--;
> 
> i.e. it's not a trivial change to code written to assume that a 
> character is a fixed width and fits into a single octet.

Well, UTF-8 is made so that all characters in the 7-bit ASCII range
 have the same codes as in ASCII, so if your grammar only has ASCII
meta-characters, then you can parse the UTF string as ASCII. 
This was a design goal for UTF-8. 

Best regards
Keld