Re: encoding and content in YML files

Luckily for YAML we don't have to deal with graphemes. That's a horribly
thorny issue, as you describe. AFAIK all languages that "support Unicode"
will actually deal with code points, and let handling graphemes be done at
a higher level.

There's _some_ justification to leave these out of the language itself
because the main reason to think about graphemes is to think in terms of
displayed text, and when you start thinking about that, you have to think
about right-to-left vs. left-to-right, and the fact that certain graphemes
require different glyphs depending on what is next to them, and all sort of
other horribleness that comes from dealing with glyphs and fonts.

The only reason that non-display-oriented code needs to deal with something
like graphemes is for capitalization, sorting and comparison. I think
requires just the normalization rules, which are defined in terms of code
points only. These rules are simpler, but still pretty complicated. You'd
expect every self-respecting language that "supports Unicode" to provide
normalization, and you'd be disappointed ;-)

Good luck with your Java library,

Oren.


On Fri, Oct 16, 2015 at 1:30 AM, Zenaan Harkness <[email protected]> wrote:

> On 10/15/15, Oren Ben-Kiki <[email protected]> wrote:
> > Doesn't work that way :-)
> >
> > "Code point" is a "platonic ideal". \u1234, UTF8, UTF16LE, UTF16BE,
> > UTF32LE, UTF32BE, etc. are all different ways to encode it.
> >
> > Think of the number three. You can't put the number three into a file.
> You
> > can put the byte 0b00000011 into a file, encoding it as a 1-byte integer
> > (uint8_t). Or you can write the ascii string 't' 'h' 'r' 'e' 'e' into a
> > file. You you could put the ascii character '3' into a file. Or any of a
> > zillion other ways. All these are _encodings_. The number 3 itself is
> > neither of them. It is a platonic ideal (the successor of the successor
> of
> > the successor of the zero element, if you go by Peano's axioms - and the
> > previous sentence is yet another "encoding").
> >
> > A "code point" is like that. You can only put an _encoding_ of a code
> point
> > into a file.
>
> Very well described. Thank you Oren.
>
> Here is a little project, with some Javadoc describing Java's Unicode
> limitations, for those working in Java:
> http://zenaan.github.io/zen/javadoc/zen/lang/string.html
>
> I haven't touched this since May, and may well not for another year or
> two by the looks of it. Still, the Javadoc may be useful for those
> trying to wrap their heads around all things Unicode, and also, and in
> particular, with respect to Java programming. It certainly did my head
> in, a few times.
>
> Here was my original post to stringtemplate-discussion mailing list,
> "a coder's lament on the paucity of java.lang.String functionality":
>
> https://groups.google.com/forum/?_escaped_fragment_=msg/stringtemplate-discussion/jJ_gZrF8SKg/ir_cuPRx1JsJ#!msg/stringtemplate-discussion/jJ_gZrF8SKg/ir_cuPRx1JsJ
>
> I also posted that email to a semi-private google group, and we
> engaged in a substantial discussion, which may be useful to those who
> still are struggling after reading the Javadoc posted above - if so, I
> am willing to repost the key exchanges here (with emails/names
> redacted) - just ask.
>
> Regards
> Zenaan
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Yaml-core mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/yaml-core
>

------------------------------------------------------------------------------

_______________________________________________
Yaml-core mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/yaml-core