Re: Internationalization

A final thought below

Tom Petch

----- Original Message -----
From: "Joel M. Halpern" <[email protected]>
To: <[email protected]>
Sent: Friday, September 29, 2006 10:10 PM
Subject: Re: Internationalization


> Thanks, I think.  (Actually, let me be clear that I very much
> appreciate this clarification.  It raises a number of questions that
> I did not know enough to ask. Thanks!)
>
>
> I was hoping for a simpler approach in this context.  But it may well
> be insufficient.
>
> 1) Strings are UTF-8
> 2) FEs store whatever CEs give them
> 3) Whenever Strings are used for correlating information across LFBs,
> exact match is required (no equivalence checks are required of either
> the CE or the FE for this purpose.)
> 4) Any entity that is displaying or using a stored string is
> responsible for verifying its correctness.
> 5) Specific LFB Class definitions may restrict the string contents in
> any way needed for their usage.  (For example, if the string must
> contain a DNS name, etc.)
>
My thinking in proposing a subset as well as a full range of Unicode characters
was that it would be more controlled than allowing each Class definition to
choose its own, and that we might be able to guess in advance a small number,
two or not many more, which would meet most requirements; and that this would
simplify things, but that is probably a matter of opinion.:-)


> In order to have different rules for different contexts, we would
> have to introduce two different data types, once for ASCII strings
> and one for UTF-8 strings.
> The other alternative is to just make everything octetstring, and use
> comments to indicate when this is expected to contain a human
> readable string, and how it should be processed.  I would prefer not
> to do that.
>
> Does this spec need to cover issues of subsetting and representation
> selection?  Except for the exact match mentioned above, the ForCES
> protocol does not work with these strings in any meaningful way.
>
> Yours,
> Joel
>
> At 02:37 PM 9/29/2006, tom.petch wrote:
> >Some thoughts
> >
> >1) Truncation; you should specify what a receiver SHOULD do when it
> >receives an
> >incomplete multi-byte sequence; discard the message? use as much as it can?
> >
> >2) Shortest form; characters can be encoded in more than one way and this
> >generates a security exposure so shortest fom SHOULD always be generated and
> >messages not in shortest form SHOULD be discarded.
> >
> >3) Subset; Unicode is big and mostly unwanted, positively unhelpful
> >even.  Some
> >RFC have specified a subset but I not seen any one gain
> >acceptance.  I suggest a
> >recommendation that it be restricted to U+0020..U+007F unless there is a good
> >reason not to.
> >
> >4) Reference; if you want a reference for Unicode, you have a choice of the
> >website or the Addison-Wesley book
> >
> >5) Which strings can be in UTF-8?  As fewe as possible IMHO, ie only where
the
> >string
> >needs to make sense in another language, not in names or other short strings
> >that are used as a handle, a reference, for which the meaning and the
language
> >usually becomes academic.
> >
> >While the IETF mandates UTF-8, I believe it has not done enough to
> >support those
> >writing I-Ds;  I did raise this on the main list last December and did not
> >change my views as a result.  There is expertise but often we are left to
> >founder; I
> >think that one of the better bits of floundering recently is in
> >http://www.ietf.org/internet-drafts/draft-ietf-syslog-protocol-17.txt
> >
> >Tom Petch
> >
> >----- Original Message -----
> >From: "Joel M. Halpern" <[email protected]>
> >To: <[email protected]>
> >Sent: Thursday, September 28, 2006 8:41 PM
> >Subject: Internationalization
> >
> >
> > > The Forces LFB definition allows LFBs to include strings.  That seems
> > > pretty useful.
> > > However, that seems to raise the i18n question.  We probably need to
> > > say something about this.
> > > For the properties section, I am declaring that the string length is in
> >octets.
> > >
> > > Currently, the actual definition says that strings are null
> > terminated ASCII.
> > > That is liable to get us in trouble.
> > > But allowing multi-byte characters, and other things, is going to get
> > > us into termination problems.
> > >
> > > Now, string are always shipped with lengths.  So we could get rid of
> > > the null termination.
> > > And we could define them to be UTF-8, and declare that the length is
> > > the maximum number of octets, not the maximum number of characters.
> > >
> > > I don't know if that is sufficient.  But it would probably help.
> > >
> > > Feedback ASAP.  This will affect progression of the document.  There
> > > are folks out there who watch very carefully for this sort of issue.
> > >
> >Yup, it stirred me from my slumbers.
> > > Yours,
> > > Joel