Re: Dylan 2016: Evolving Dylan
Bruce Hoult <[email protected]> Sat, 19 Apr 2014 09:19:12 +1200
| Newsgroups | gmane.comp.lang.dylan.gwydion.devel |
|---|---|
| Message-ID | <CAMU+Ekwbju7v+xkGAbT15_Ychnp4hKC8d0uNujBXuLcKCn9rdQ@mail.gmail.com> |
--===============0726254866== Content-Type: multipart/alternative; boundary=089e0122f3327b9afe04f757b354 --089e0122f3327b9afe04f757b354 Content-Type: text/plain; charset=UTF-8 It's not "aside" from anything. I don't have a clue how a simple Dylan programmer (or C programmer for that matter) who indexes into an array of code points (or extracts one from a UTF-8 or UTF-16 stream) should know that the code point she is looking at needs to be treated as a unit with other code points before or after it because of either combining marks or grapheme clusters. It is common knowledge, for example, that if you are indexing into a UTF-8 string and find an octet with the high bit clear then then have the entire code point already. If you find the two high bits set then you have the first octet of a multi-octet code point (and the total number of octets is the same as the number of high bits set). If you find the two high bits are "10" then you need to scoot over to the left until you find both high bits set to find the start of the code point. What are the corresponding rules for combining code points into characters? No doubt there is a standards document somewhere, but I for one have never seen an explanation of how to do it, only that it sometimes needs to be done. And that NSString does it automagically for you so don't worry :) On Sat, Apr 19, 2014 at 3:15 AM, Dustin Voss <[email protected]> wrote: > On Apr 18, 2014, at 2:25 AM, Bruce Hoult <[email protected]> wrote: > > > I absolutely agree that UTF-16 as used by Windows, C#, Java is wrong. > And that UTF-8 is best in files and streams and in plain C programs. > > > > What I don't see addressed there (or anywhere I've seen) is how one > recognises a combining character or grapheme cluster. > > You mean aside from the properties in the Unicode code tables? > > > It seems to me that if such things can be recognised on decoding an > input file/stream then a representation in which a character is the same > size as a pointer allows complex beasts to be implemented as a pointer to a > sequence of code points, interned and shared by all strings using them. > Simpler characters (the vast vast majority) can be stored as their code > point munged in the traditional dynamic language way to distinguish > pointers from immediate values. > > This is much like the idea I have, and have proposed before, for > supporting Unicode text: <character> as a sequence of one or more > <code-point>, <string> as a sequence of <character>. <Character> can then > be an abstract (encodable) character or a grapheme cluster. (I can reply or > CC the list with more detail, if you like.) > _______________________________________________ > hackers mailing list > [email protected] > https://lists.opendylan.org/mailman/listinfo/hackers > --089e0122f3327b9afe04f757b354 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">It's not "aside" from anything. I don't = have a clue how a simple Dylan programmer (or C programmer for that matter)= who indexes into an array of code points (or extracts one from a UTF-8 or = UTF-16 stream) should know that the code point she is looking at needs to b= e treated as a unit with other code points before or after it because of ei= ther combining marks or grapheme clusters.<div> <br></div><div>It is common knowledge, for example, that if you are indexin= g into a UTF-8 string and find an octet with the high bit clear then then h= ave the entire code point already. If you find the two high bits set then y= ou have the first octet of a multi-octet code point (and the total number o= f octets is the same as the number of high bits set). If you find the two h= igh bits are "10" then you need to scoot over to the left until y= ou find both high bits set to find the start of the code point.</div> <div><br></div><div>What are the corresponding rules for combining code poi= nts into characters?</div><div><br></div><div>No doubt there is a standards= document somewhere, but I for one have never seen an explanation of how to= do it, only that it sometimes needs to be done. And that NSString does it = automagically for you so don't worry :)</div> </div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Sat,= Apr 19, 2014 at 3:15 AM, Dustin Voss <span dir=3D"ltr"><<a href=3D"mail= to:[email protected]" target=3D"_blank">[email protected]</a>></span> wrote:<br><b= lockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px = #ccc solid;padding-left:1ex"> <div class=3D"">On Apr 18, 2014, at 2:25 AM, Bruce Hoult <<a href=3D"mai= lto:[email protected]">[email protected]</a>> wrote:<br> <br> > I absolutely agree that UTF-16 as used by Windows, C#, Java is wrong. = And that UTF-8 is best in files and =C2=A0streams and in plain C programs.<= br> ><br> > What I don't see addressed there (or anywhere I've seen) is ho= w one recognises a combining character or grapheme cluster.<br> <br> </div>You mean aside from the properties in the Unicode code tables?<br> <div class=3D""><br> > It seems to me that if such things can be recognised on decoding an in= put file/stream then a representation in which a character is the same size= as a pointer allows complex beasts to be implemented as a pointer to a seq= uence of code points, interned and shared by all strings using them. Simple= r characters (the vast vast majority) can be stored as their code point mun= ged in the traditional dynamic language way to distinguish pointers from im= mediate values.<br> <br> </div>This is much like the idea I have, and have proposed before, for supp= orting Unicode text: <character> as a sequence of one or more <cod= e-point>, <string> as a sequence of <character>. <Charact= er> can then be an abstract (encodable) character or a grapheme cluster.= (I can reply or CC the list with more detail, if you like.)<br> <div class=3D"HOEnZb"><div class=3D"h5">___________________________________= ____________<br> hackers mailing list<br> <a href=3D"mailto:[email protected]">[email protected]<= /a><br> <a href=3D"https://lists.opendylan.org/mailman/listinfo/hackers" target=3D"= _blank">https://lists.opendylan.org/mailman/listinfo/hackers</a><br> </div></div></blockquote></div><br></div> --089e0122f3327b9afe04f757b354-- --===============0726254866== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ hackers mailing list [email protected] https://lists.opendylan.org/mailman/listinfo/hackers --===============0726254866==--