Re: Internationalization

Ok, that is fine; I suggest including somewhere
'...SHOULD be in shortest form'
If you need a reference, then I suggest

         Davis, M. and M. Suignard, "UNICODE Security Considerations",
         July 2005, <http://www.unicode.org/reports/tr36/tr36-3.html>.

What about the string ending in an incomplete multi-byte sequence?  I think the
possibility should be mentioned, even if the I-D does not go as far as requiring
receivers to do any checking of what they store.

Tom Petch

----- Original Message -----
From: "Joel M. Halpern" <[email protected]>
To: <[email protected]>
Sent: Sunday, October 01, 2006 10:59 PM
Subject: Re: Internationalization


> The only Comparison's the ForCES documents have to talk about are
> those that are done between fields in FEs.  populated by CEs.
> Now, I realize that different CEs could generate the strings differently.
> But a given FE is managed only by a single CE.
>
> Would it be sensible to say that the CE SHOULD normalize the UTF-8
> strings, so that they can be properly utilized if received by another CE
later?
> There may be LFBs which have strings for display to users.  But it is
> up to that LFB definition to constrain the string to be valid UTF-8.
>
> My concern is that if we do more, we will be "requiring" FEs do have
> a significant amount of logic that not only has no benefit, it will
> never get exercised (thus making it likely to be wrong.)
>
> Yours,
> Joel
>
> At 03:02 PM 10/1/2006, tom.petch wrote:
> >Ok, but life with UTF-8 is not simple, so:
> >
> >- I would prefer two data types, one for 'referents', limited to
> >ASCII, and the
> >other for human readable, UTF-8 (although I have yet to work out
> >just what that
> >means for ForCES; it's that XML again:-(.  This is a well-trodden
> >road for SNMP
> >where the SMI species a number different data types, such as DisplayString or
> >SnmpAdminString, for different puruposes; unfortunately, this
> >predates i18n and
> >is based on ASCII but I like the principle of a DDL allowing for such
> >differences.
> >
> >- I think we MUST have legal UTF-8 eg a string of any length that ends with
> >x'C0' is not valid UTF-8 and SHOULD be discarded (silently?)
> >
> >- I think that comparison does require normalisation, ie shortest
> >form; UTF-8 is
> >a character encoding scheme (Transformation Format) that turns a character
> >string into a binary representation.  Unfortunately, there are many possible
> >(countably infinite?) different binary representations of the same character
> >string; it only becomes unique if you require shortest form, and
> >this has become
> >well known because the use of other than shortest form creates a security
> >exposure with some applications.  So for ForCES, are you comparing
> >the character
> >string or are you comparing the way in which a string source has chosen to
> >encode a particular string in UTF-8?  Hopefully, all benign software
> >has got the
> >message and will only generate shortest form but I think we should require it
> >although, as with invalid octet sequences, this could require the
> >receiver of a
> >string to do more than just store it.
> >
> >Tom Petch