Re: Thoughts on upsizing Unicode

--00000000000032cdfe064e995652
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Apr 3, 2026 at 3:48=E2=80=AFAM Dominikus Dittes Scherkl via Unicode=
 <
[email protected]> wrote:

> For  UTF-16 simply use the now forbidden surrogate sequences as already
> suggested (3 surrogates Hi-Hi-Lo or Hi-Lo-Lo each encoding 10 bits + one
> bit decided by using Hi or Lo in the middle for 31bits at all).
>

The problem with using a Hi-Lo-Lo sequence like this is that you then
cannot tell if a Hi-Lo sequence is the beginning of a Hi-Lo-Lo sequence
without looking ahead to the next code unit. If a Hi-Lo-Lo sequence is
truncated, it appears as a valid Hi-Lo sequence for a different character,
and the error is impossible to detect.

A small tweak to this system can solve this issue, however. Instead of
Hi-Lo-Lo and Hi-Hi-Lo sequences, use Hi-Hi-Lo and Hi-Hi-Hi sequences. Then
no valid sequence can appear at the beginning of another valid sequence,
and if a three-surrogate sequence is truncated, it appears as a Hi-Hi
sequence, which is invalid and can be detected.

Hi-Lo sequence (2^20 codepoints):
0xD800 0xDC00 =3D> U+00010000
0xDBFF 0xDFFF =3D> U+0010FFFF

Hi-Hi-Lo sequence (2^30 codepoints):
0xD800 0xD800 0xDC00 =3D> invalid (overlong encoding of U+00000000)
0xD810 0xDBFF 0xDFFF =3D> invalid (overlong encoding of U+0010FFFF)
0xD811 0xD800 0xDC00 =3D> U+00110000
0xDBFF 0xDBFF 0xDFFF =3D> U+3FFFFFFF

Hi-Hi-Hi sequence (2^30 codepoints):
0xD800 0xD800 0xD800 =3D> U+40000000
0xDBFF 0xDBFF 0xDBFF =3D> U+7FFFFFFF

Even then there are still issues. If your text consists entirely of code
points U+40000000 and above, you lose self-synchronization.

-- Rebecca Bettencourt

--00000000000032cdfe064e995652
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_quote gmail_quote_container"><div dir=
=3D"ltr" class=3D"gmail_attr">On Fri, Apr 3, 2026 at 3:48=E2=80=AFAM Domini=
kus Dittes Scherkl via Unicode &lt;<a href=3D"mailto:[email protected]=
rg">[email protected]</a>&gt; wrote:<br></div><blockquote class=3D"g=
mail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204=
,204,204);padding-left:1ex">For=C2=A0 UTF-16 simply use the now forbidden s=
urrogate sequences as already<br>
suggested (3 surrogates Hi-Hi-Lo or Hi-Lo-Lo each encoding 10 bits + one <b=
r>
bit decided by using Hi or Lo in the middle for 31bits at all).<br></blockq=
uote><div><br></div><div>The problem with using a Hi-Lo-Lo sequence like th=
is is that you then cannot tell if a Hi-Lo sequence is the beginning of a H=
i-Lo-Lo sequence without looking ahead to the next code unit. If a Hi-Lo-Lo=
 sequence is truncated, it appears as a valid Hi-Lo sequence for a differen=
t character, and the error is impossible to detect.<br><br>A small tweak to=
 this system can solve this issue, however. Instead of Hi-Lo-Lo and Hi-Hi-L=
o sequences, use Hi-Hi-Lo and Hi-Hi-Hi sequences. Then no valid sequence ca=
n appear at the beginning of another valid sequence, and if a three-surroga=
te sequence is truncated, it appears as a Hi-Hi sequence, which is invalid =
and can be detected.<br><br>Hi-Lo sequence (2^20 codepoints):<br>0xD800 0xD=
C00 =3D&gt; U+00010000<br>0xDBFF 0xDFFF =3D&gt; U+0010FFFF<br><br>Hi-Hi-Lo =
sequence (2^30 codepoints):<br>0xD800 0xD800 0xDC00 =3D&gt; invalid (overlo=
ng encoding of U+00000000)<br>0xD810 0xDBFF 0xDFFF =3D&gt; invalid (overlon=
g encoding of U+0010FFFF)<br>0xD811 0xD800 0xDC00 =3D&gt; U+00110000<br>0xD=
BFF 0xDBFF 0xDFFF =3D&gt; U+3FFFFFFF<br><br>Hi-Hi-Hi sequence (2^30 codepoi=
nts):<br>0xD800 0xD800 0xD800 =3D&gt; U+40000000<br>0xDBFF 0xDBFF 0xDBFF =
=3D&gt; U+7FFFFFFF<br><br>Even then there are still issues. If your text co=
nsists entirely of code points U+40000000 and above, you lose self-synchron=
ization.<br><div dir=3D"ltr"><div><div dir=3D"ltr" class=3D"gmail_signature=
"><br>-- Rebecca Bettencourt</div></div><br></div></div></div></div>

--00000000000032cdfe064e995652--