Re: Internationalization

The only Comparison's the ForCES documents have to talk about are 
those that are done between fields in FEs.  populated by CEs.
Now, I realize that different CEs could generate the strings differently.
But a given FE is managed only by a single CE.

Would it be sensible to say that the CE SHOULD normalize the UTF-8 
strings, so that they can be properly utilized if received by another CE later?
There may be LFBs which have strings for display to users.  But it is 
up to that LFB definition to constrain the string to be valid UTF-8.

My concern is that if we do more, we will be "requiring" FEs do have 
a significant amount of logic that not only has no benefit, it will 
never get exercised (thus making it likely to be wrong.)

Yours,
Joel

At 03:02 PM 10/1/2006, tom.petch wrote:
>Ok, but life with UTF-8 is not simple, so:
>
>- I would prefer two data types, one for 'referents', limited to 
>ASCII, and the
>other for human readable, UTF-8 (although I have yet to work out 
>just what that
>means for ForCES; it's that XML again:-(.  This is a well-trodden 
>road for SNMP
>where the SMI species a number different data types, such as DisplayString or
>SnmpAdminString, for different puruposes; unfortunately, this 
>predates i18n and
>is based on ASCII but I like the principle of a DDL allowing for such
>differences.
>
>- I think we MUST have legal UTF-8 eg a string of any length that ends with
>x'C0' is not valid UTF-8 and SHOULD be discarded (silently?)
>
>- I think that comparison does require normalisation, ie shortest 
>form; UTF-8 is
>a character encoding scheme (Transformation Format) that turns a character
>string into a binary representation.  Unfortunately, there are many possible
>(countably infinite?) different binary representations of the same character
>string; it only becomes unique if you require shortest form, and 
>this has become
>well known because the use of other than shortest form creates a security
>exposure with some applications.  So for ForCES, are you comparing 
>the character
>string or are you comparing the way in which a string source has chosen to
>encode a particular string in UTF-8?  Hopefully, all benign software 
>has got the
>message and will only generate shortest form but I think we should require it
>although, as with invalid octet sequences, this could require the 
>receiver of a
>string to do more than just store it.
>
>Tom Petch