Re: Internationalization

<inline>
Tom Petch

----- Original Message -----
From: "Joel M. Halpern" <[email protected]>
To: <[email protected]>
Sent: Friday, September 29, 2006 10:10 PM
Subject: Re: Internationalization


> Thanks, I think.  (Actually, let me be clear that I very much
> appreciate this clarification.  It raises a number of questions that
> I did not know enough to ask. Thanks!)
>
> I was hoping for a simpler approach in this context.  But it may well
> be insufficient.
>
> 1) Strings are UTF-8
> 2) FEs store whatever CEs give them
> 3) Whenever Strings are used for correlating information across LFBs,
> exact match is required (no equivalence checks are required of either
> the CE or the FE for this purpose.)
> 4) Any entity that is displaying or using a stored string is
> responsible for verifying its correctness.
> 5) Specific LFB Class definitions may restrict the string contents in
> any way needed for their usage.  (For example, if the string must
> contain a DNS name, etc.)
>
> In order to have different rules for different contexts, we would
> have to introduce two different data types, once for ASCII strings
> and one for UTF-8 strings.
> The other alternative is to just make everything octetstring, and use
> comments to indicate when this is expected to contain a human
> readable string, and how it should be processed.  I would prefer not
> to do that.
>
> Does this spec need to cover issues of subsetting and representation
> selection?  Except for the exact match mentioned above, the ForCES
> protocol does not work with these strings in any meaningful way.
>
Ok, but life with UTF-8 is not simple, so:

- I would prefer two data types, one for 'referents', limited to ASCII, and the
other for human readable, UTF-8 (although I have yet to work out just what that
means for ForCES; it's that XML again:-(.  This is a well-trodden road for SNMP
where the SMI species a number different data types, such as DisplayString or
SnmpAdminString, for different puruposes; unfortunately, this predates i18n and
is based on ASCII but I like the principle of a DDL allowing for such
differences.

- I think we MUST have legal UTF-8 eg a string of any length that ends with
x'C0' is not valid UTF-8 and SHOULD be discarded (silently?)

- I think that comparison does require normalisation, ie shortest form; UTF-8 is
a character encoding scheme (Transformation Format) that turns a character
string into a binary representation.  Unfortunately, there are many possible
(countably infinite?) different binary representations of the same character
string; it only becomes unique if you require shortest form, and this has become
well known because the use of other than shortest form creates a security
exposure with some applications.  So for ForCES, are you comparing the character
string or are you comparing the way in which a string source has chosen to
encode a particular string in UTF-8?  Hopefully, all benign software has got the
message and will only generate shortest form but I think we should require it
although, as with invalid octet sequences, this could require the receiver of a
string to do more than just store it.

Tom Petch

> Yours,
> Joel
>
> At 02:37 PM 9/29/2006, tom.petch wrote:
> >Some thoughts
> >
> >1) Truncation; you should specify what a receiver SHOULD do when it
> >receives an
> >incomplete multi-byte sequence; discard the message? use as much as it can?
> >
> >2) Shortest form; characters can be encoded in more than one way and this
> >generates a security exposure so shortest fom SHOULD always be generated and
> >messages not in shortest form SHOULD be discarded.
> >
> >3) Subset; Unicode is big and mostly unwanted, positively unhelpful
> >even.  Some
> >RFC have specified a subset but I not seen any one gain
> >acceptance.  I suggest a
> >recommendation that it be restricted to U+0020..U+007F unless there is a good
> >reason not to.
> >
> >4) Reference; if you want a reference for Unicode, you have a choice of the
> >website or the Addison-Wesley book
> >
> >5) Which strings can be in UTF-8?  As fewe as possible IMHO, ie only where
the
> >string
> >needs to make sense in another language, not in names or other short strings
> >that are used as a handle, a reference, for which the meaning and the
language
> >usually becomes academic.
> >
> >While the IETF mandates UTF-8, I believe it has not done enough to
> >support those
> >writing I-Ds;  I did raise this on the main list last December and did not
> >change my views as a result.  There is expertise but often we are left to
> >founder; I
> >think that one of the better bits of floundering recently is in
> >http://www.ietf.org/internet-drafts/draft-ietf-syslog-protocol-17.txt
> >
> >Tom Petch
> >
> >----- Original Message -----
> >From: "Joel M. Halpern" <[email protected]>
> >To: <[email protected]>
> >Sent: Thursday, September 28, 2006 8:41 PM
> >Subject: Internationalization
> >
> >
> > > The Forces LFB definition allows LFBs to include strings.  That seems
> > > pretty useful.
> > > However, that seems to raise the i18n question.  We probably need to
> > > say something about this.
> > > For the properties section, I am declaring that the string length is in
> >octets.
> > >
> > > Currently, the actual definition says that strings are null
> > terminated ASCII.
> > > That is liable to get us in trouble.
> > > But allowing multi-byte characters, and other things, is going to get
> > > us into termination problems.
> > >
> > > Now, string are always shipped with lengths.  So we could get rid of
> > > the null termination.
> > > And we could define them to be UTF-8, and declare that the length is
> > > the maximum number of octets, not the maximum number of characters.
> > >
> > > I don't know if that is sufficient.  But it would probably help.
> > >
> > > Feedback ASAP.  This will affect progression of the document.  There
> > > are folks out there who watch very carefully for this sort of issue.
> > >
> >Yup, it stirred me from my slumbers.
> > > Yours,
> > > Joel