Re: Internationalization

Thanks for clarifying that terminators can be used.

Probably the largest reason for using a length is 
that we need lengths for arrays and for 
octetstrings (binary strings) anyway.  So we have 
the encoding mechanism already in the protocol.

The other nice thing, I think, is that FEs in 
general do not actually have to understand UTF-8, 
or know how to construct valid (or invalid) 
multi-byte characters.  Since the FE is not 
actually responsible for displaying the strings, this seems advantageous.

Yours,
Joel

PS: There are a lot of contexts where terminators 
would probably be more sensible, and it is 
helpful to be reminded of how to allow them.


At 03:55 AM 10/1/2006, tom.petch wrote:
>To address the question you specifically asked, 
>I have no problems with using a
>length although my personal preference, based on time spent coding and
>debugging, is for termination characters.
>
>It is possible to use termination characters with UTF-8 and some protocols do;
>At the risk of going back to basics:-
>
>If you want to use give eg £ a special meaning in a message, distinct from its
>ue as user data, then all you need is an escape character, eg %, so that £ on
>its own has the special meaning while %£ stands 
>for user data £.  Then of course
>user data % needs to be taken care of, eg as %%, 
>and then user data %% needs to
>become %%% or %%%% and so on.  All very logical and widely used, in some shape
>or form.
>
>More generally, what I/you/we are doing is choosing a character set (eg
>Unicode), giving each member of the set a corresponding numerical code and
>deciding how to encode that code (eg UTF-8) into 
>octets (or whatever).  But the
>mechanisms of storage and transmission may give 
>special meaning to some octets,
>in which case you then apply a transfer encoding 
>syntax (eg to the UTF-8) which
>differentiates the special use from the character encoding syntax use.  Escape
>characters is one technique; another, used by 
>SMTP, is base64, whereby a string
>of three UTF-8 octets is divided into four 
>groups of six bits, each of which is
>then encoded into an octet for transmission, using 64 of the possisble 256
>combinations, leaving the other 192 free for special use.
>
>So we can have UTF-8 and a special terminating 
>character, as long as there is a
>suitable transfer encoding syntax.  Worth it?  I await rough consensus.
>
>Tom Petch
>
>----- Original Message -----
>From: "Joel M. Halpern" <[email protected]>
>To: <[email protected]>
>Sent: Thursday, September 28, 2006 8:41 PM
>Subject: Internationalization
>
>
> > The Forces LFB definition allows LFBs to include strings.  That seems
> > pretty useful.
> > However, that seems to raise the i18n question.  We probably need to
> > say something about this.
> > For the properties section, I am declaring that the string length is in
>octets.
> >
> > Currently, the actual definition says that 
> strings are null terminated ASCII.
> > That is liable to get us in trouble.
> > But allowing multi-byte characters, and other things, is going to get
> > us into termination problems.
> >
> > Now, string are always shipped with lengths.  So we could get rid of
> > the null termination.
> > And we could define them to be UTF-8, and declare that the length is
> > the maximum number of octets, not the maximum number of characters.
> >
> > I don't know if that is sufficient.  But it would probably help.
> >
> > Feedback ASAP.  This will affect progression of the document.  There
> > are folks out there who watch very carefully for this sort of issue.
> >
> > Yours,
> > Joel