Re: Markus King's "UTF-8 decoder capability and stress test"

--0000000000001559980653fd9cd3
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

So my first thought is to write a test to verify how Raku handles each of
these cases, but the file itself says this isn't a conformance test.

You can see raku's specification suite (including unicode) here:
https://github.com/raku/roast - but that's everything, and doesn't have a
section specifically about, e.g. malformed UTF8.

It should be very straightforward to write a test in raku for these various
items and see what the result is. Might be a good idea for a blog post!

On Wed, Jun 10, 2026 at 1:50=E2=80=AFAM William Michels via perl6-users <
[email protected]> wrote:

> https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt
>
> Unicode/UTF-8 Question:
>
> Does anyone know if Rakudo has ever been tested against Markus
> King's "UTF-8 decoder capability and stress test"?
>
> From the intro:
>
> "This test file can help you examine, how your UTF-8 decoder handles
> various types of correct, malformed, or otherwise interesting UTF-8
> sequences."
>
>
> Any info appreciated,
>
> Thx, Bill.
>

--0000000000001559980653fd9cd3
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>So my first thought is to write a test to verify how =
Raku handles each of these cases, but the file itself says this isn&#39;t a=
 conformance test.</div><div><br></div><div>You can see raku&#39;s specific=
ation suite (including unicode) here:=C2=A0<a href=3D"https://github.com/ra=
ku/roast">https://github.com/raku/roast</a> - but that&#39;s everything, an=
d doesn&#39;t have a section specifically=C2=A0about, e.g. malformed UTF8.<=
/div><div><br></div><div>It should be very straightforward to write a test =
in raku for these various items and see what the result is.=C2=A0Might be a=
 good idea for a blog post!</div></div><br><div class=3D"gmail_quote gmail_=
quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Jun 10, 2026=
 at 1:50=E2=80=AFAM William Michels via perl6-users &lt;<a href=3D"mailto:p=
[email protected]">[email protected]</a>&gt; wrote:<br></div><blockquo=
te class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-widt=
h:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-le=
ft:1ex"><div style=3D"line-break:after-white-space"><a href=3D"https://www.=
cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt" target=3D"_blank">https://=
www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt</a><div><br></div><div>=
Unicode/UTF-8 Question:</div><div><br></div><div>Does anyone know if Rakudo=
 has ever been tested against Markus King&#39;s=C2=A0&quot;UTF-8 decoder ca=
pability and stress test&quot;?</div><div><br></div><div><div>From the intr=
o:</div><div><br></div></div><blockquote style=3D"margin:0px 0px 0px 40px;b=
order:medium;padding:0px"><div><div>&quot;This test file can help you exami=
ne, how your UTF-8 decoder handles</div></div><div><div>various types of co=
rrect, malformed, or otherwise interesting UTF-8</div></div><div><div>seque=
nces.&quot;</div></div></blockquote><div><br></div><div>Any info appreciate=
d,=C2=A0</div><div><br></div><div>Thx, Bill.</div></div></blockquote></div>

--0000000000001559980653fd9cd3--