Re: Thoughts on upsizing Unicode

--00000000000043f3b6064e9c67c9
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Apr 3, 2026 at 8:54=E2=80=AFPM S=C5=82awomir Osipiuk <sosipiuk@gmai=
l.com> wrote:

> It's also wrong to "steal" PUA code points to cater to this wonky
> encoding. It's bad enough that people already perceive PUA characters
> as second-class and are reluctant to use them. The solution should be
> restricted to the existing set of surrogates only.
>

There is no "stealing" going on. The PUA code points still exist, only now
they are encoded as U+110000 and above would be. (Just like when converting
from Latin-1 to UTF-8, U+0080 to U+00FF still exist, only now they are
encoded as U+0100 and above are.) And this solution does only use the
existing set of surrogates.

Maybe there's some confusion over code units vs code points. Both of our
solutions are proposing changing how UTF-16 code *units* work; neither of
them are proposing changing code *points* in any way (besides allowing code
points beyond U+10FFFF).

Any solution is going to be wonky, as UTF-16 itself is a wonky encoding
to begin with. Having multiple advantages such as error detection,
substring matching, and self-synchronization is more important than whether
the solution is "simple" versus "wonky."

> If a Hi-Lo-Lo sequence is truncated, it appears as a valid Hi-Lo sequence
> for a different character, and the error is impossible to detect.
>
> I think this is a minor problem, and the most acceptable one.
>

If this were an actual call for proposals this would definitely be a major
problem. It complicates text rendering, string validation, security,
substring matching, collation, and a host of other things that are much
easier to deal with when one valid code sequence can't contain another.


> If we're talking about security, a
> truncated stream should be raising an alarm anyway.
>

That's hard to do when you can't detect it.

--00000000000043f3b6064e9c67c9
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_quote gmail_quote_container"><div dir=
=3D"ltr" class=3D"gmail_attr">On Fri, Apr 3, 2026 at 8:54=E2=80=AFPM S=C5=
=82awomir Osipiuk &lt;<a href=3D"mailto:[email protected]">sosipiuk@gmail.=
com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e=
x">It&#39;s also wrong to &quot;steal&quot; PUA code points to cater to thi=
s wonky<br>
encoding. It&#39;s bad enough that people already perceive PUA characters<b=
r>
as second-class and are reluctant to use them. The solution should be<br>
restricted to the existing set of surrogates only.<br></blockquote><div><br=
></div><div>There is no &quot;stealing&quot; going on. The PUA code points =
still exist, only now they are encoded as U+110000 and above would be. (Jus=
t like when converting from Latin-1 to UTF-8, U+0080 to U+00FF still exist,=
 only now they are encoded as U+0100 and above are.) And this solution does=
 only use the existing set of surrogates.</div><div><br></div><div>Maybe th=
ere&#39;s some confusion over code units vs code points. Both of our soluti=
ons are proposing changing how UTF-16 code <i>units</i> work; neither of th=
em are proposing changing code <i>points</i> in any way (besides allowing c=
ode points beyond U+10FFFF).</div><div><br></div><div>Any solution is going=
 to be wonky, as UTF-16 itself is a wonky encoding to=C2=A0begin=C2=A0with.=
 Having multiple advantages such as error detection, substring matching, an=
d self-synchronization is more important than whether the solution is &quot=
;simple&quot; versus &quot;wonky.&quot;</div><div><br></div><blockquote cla=
ss=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid =
rgb(204,204,204);padding-left:1ex">
&gt; If a Hi-Lo-Lo sequence is truncated, it appears as a valid Hi-Lo seque=
nce for a different character, and the error is impossible to detect.<br>
<br>
I think this is a minor problem, and the most acceptable one.<br></blockquo=
te><div><br></div><div>If this were an actual call for proposals this would=
 definitely be a major problem. It complicates text rendering, string valid=
ation, security, substring matching, collation, and a host of other things =
that are much easier to deal with when one valid code sequence can&#39;t co=
ntain another.</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" styl=
e=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddin=
g-left:1ex">If we&#39;re talking about security, a<br>
truncated stream should be raising an alarm anyway.<br></blockquote><div><b=
r></div><div>That&#39;s hard to do when you can&#39;t detect it.</div><div>=
<br></div></div></div>

--00000000000043f3b6064e9c67c9--