Re: Internationalization

To address the question you specifically asked, I have no problems with using a
length although my personal preference, based on time spent coding and
debugging, is for termination characters.

It is possible to use termination characters with UTF-8 and some protocols do;
At the risk of going back to basics:-

If you want to use give eg £ a special meaning in a message, distinct from its
ue as user data, then all you need is an escape character, eg %, so that £ on
its own has the special meaning while %£ stands for user data £.  Then of course
user data % needs to be taken care of, eg as %%, and then user data %% needs to
become %%% or %%%% and so on.  All very logical and widely used, in some shape
or form.

More generally, what I/you/we are doing is choosing a character set (eg
Unicode), giving each member of the set a corresponding numerical code and
deciding how to encode that code (eg UTF-8) into octets (or whatever).  But the
mechanisms of storage and transmission may give special meaning to some octets,
in which case you then apply a transfer encoding syntax (eg to the UTF-8) which
differentiates the special use from the character encoding syntax use.  Escape
characters is one technique; another, used by SMTP, is base64, whereby a string
of three UTF-8 octets is divided into four groups of six bits, each of which is
then encoded into an octet for transmission, using 64 of the possisble 256
combinations, leaving the other 192 free for special use.

So we can have UTF-8 and a special terminating character, as long as there is a
suitable transfer encoding syntax.  Worth it?  I await rough consensus.

Tom Petch

----- Original Message -----
From: "Joel M. Halpern" <[email protected]>
To: <[email protected]>
Sent: Thursday, September 28, 2006 8:41 PM
Subject: Internationalization


> The Forces LFB definition allows LFBs to include strings.  That seems
> pretty useful.
> However, that seems to raise the i18n question.  We probably need to
> say something about this.
> For the properties section, I am declaring that the string length is in
octets.
>
> Currently, the actual definition says that strings are null terminated ASCII.
> That is liable to get us in trouble.
> But allowing multi-byte characters, and other things, is going to get
> us into termination problems.
>
> Now, string are always shipped with lengths.  So we could get rid of
> the null termination.
> And we could define them to be UTF-8, and declare that the length is
> the maximum number of octets, not the maximum number of characters.
>
> I don't know if that is sufficient.  But it would probably help.
>
> Feedback ASAP.  This will affect progression of the document.  There
> are folks out there who watch very carefully for this sort of issue.
>
> Yours,
> Joel