Re: OOB reading when using strings and surrogates

On Wed Jun 24, 2026 at 11:49 PM CEST, Peter McGoron via Chicken-users wrote=
:
>
> I was experimenting with ways to break it, and found this using=20
> valgrind. I don't have specific buggy behavior associated with this, but=
=20
> the OOB reads might cause one in another scenario.

This is a problem of the UTF-8 decode which we use. The code is mostly
branchless, and thus performs a lookahead, regardless of the number of
remaining bytes left in the string. At the end of a string, the '\0'
byte will cause a decoding error anyway, but doesn't prevent the decoder
looking still one or two bytes further.=20

This is not ideal, but I'm not too eager fiddling with the decoder, some
attempts to do so made tests fail that ensure that even incorrectly encoded
text passes cleanly through the decoding/encoding processes. I have adjuste=
d
the buffer sizes for some character-reading operations to at least avoid
the warning in normal operation at the repl caused by the reader.=20

Still, when using strings holding incorrect encodings, the warnings will
occur, I have no idea of how to address this without replacing the decoder.


cheers,
felix