Re: Undefined behavior in utf8_decode()

This program looks to be equivalent:

utf8.len'\xD9\xBD\x01\xFF\x8F\x8F\x8F\x8F\x8F\x8F\x8F\x00\x00'

And reduced to remove the valid parts of the string:

utf8.len'\xFF\x8F\x8F\x8F\x8F\x8F\x8F\x8F'

The issue appears to be:

for (; c & 0x40; c <<= 1) {

110XXXXX: read 1 continuation byte
1110XXXX: read 2 continuation bytes
11110XXX: read 3 continuation bytes (max possible in valid unicode)
111110XX: read 4 continuation bytes
1111110X: read 5 continuation bytes (max possible with lax)
11111110: read 6 continuation bytes (impossible normally)
11111111: read 7 continuation bytes (impossible normally)

With 6 or 7 continuation bytes the line:

res = (res << 6) | (cc & 0x3F);

Might even wrap around, though that appears to not matter. The lines after 
the loop:

res |= ((l_uint32)(c & 0x7F) << (count * 5)); /* add first byte */
if (count > 5 || res > MAXUTF || res < limits[count])
return NULL; /* invalid byte sequence */

Appear to be the actual issue. count*5 specifically when count is seven 
will be greater than or equal to 32, making the shift undefined behavior. A 
simple fix here is to just move the line to add the first byte after the if:

if (count > 5 || res > MAXUTF || res < limits[count])
return NULL; /* invalid byte sequence */
res |= ((l_uint32)(c & 0x7F) << (count * 5)); /* add first byte */
On Monday, February 16, 2026 at 10:05:03 AM UTC-5 Sergey Bronnikov wrote:

> Hello,
>
> there is an issue with undefined behavior in lutf8lib.c because shift 
> exponent may be too large for 32-bit type 'l_uint32':
>
> lutf8lib.c:67:34: runtime error: shift exponent 35 is too large for 32-bit 
> type 'l_uint32' (aka 'unsigned int')
>     #0 0x56308fd6face in utf8_decode 
> /src/testdir/build/lua-master/source/lutf8lib.c:67:34
>     #1 0x56308fd6f4fd in utflen 
> /src/testdir/build/lua-master/source/lutf8lib.c:99:22
>     #2 0x56308fd05fe9 in precallC 
> /src/testdir/build/lua-master/source/ldo.c:663:7
>     #3 0x56308fd0687c in luaD_precall 
> /src/testdir/build/lua-master/source/ldo.c
>     #4 0x56308fd40413 in luaV_execute 
> /src/testdir/build/lua-master/source/lvm.c:1729:22
>     #5 0x56308fd06ce3 in ccall 
> /src/testdir/build/lua-master/source/ldo.c:774:5
>     #6 0x56308fd03b69 in luaD_rawrunprotected 
> /src/testdir/build/lua-master/source/ldo.c:166:3
>     #7 0x56308fd07cd8 in luaD_pcall 
> /src/testdir/build/lua-master/source/ldo.c:1096:12
>     #8 0x56308fcfaaa0 in lua_pcallk 
> /src/testdir/build/lua-master/source/lapi.c:1097:14
>     #9 0x56308fcf1435 in LLVMFuzzerTestOneInput 
> /src/testdir/tests/capi/luaL_loadbuffer_test.c:34:3
>
> Steps to reproduce:
>
> make MYCFLAGS=-fsanitize=undefined MYLDFLAGS=-fsanitize=undefined
> echo "dXRmOC5sZW4n2b0B/4+Pj4+Pj48AACc=" | base64 --decode | ./lua -
>
> Lua version: c6b48482
>
> Sergey
>

-- 
You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion visit https://groups.google.com/d/msgid/lua-l/a94c71d5-428a-48d7-a8c7-d1c7ec757222n%40googlegroups.com.