Re: Undefined behavior in utf8_decode()

On Monday, February 16, 2026 at 1:13:12 PM UTC-5 Halalaluyafail3 wrote:

This program looks to be equivalent:

utf8.len'\xD9\xBD\x01\xFF\x8F\x8F\x8F\x8F\x8F\x8F\x8F\x00\x00'

And reduced to remove the valid parts of the string:

utf8.len'\xFF\x8F\x8F\x8F\x8F\x8F\x8F\x8F'

The issue appears to be:

for (; c & 0x40; c <<= 1) {

110XXXXX: read 1 continuation byte
1110XXXX: read 2 continuation bytes
11110XXX: read 3 continuation bytes (max possible in valid unicode)
111110XX: read 4 continuation bytes
1111110X: read 5 continuation bytes (max possible with lax)
11111110: read 6 continuation bytes (impossible normally)
11111111: read 7 continuation bytes (impossible normally)

With 6 or 7 continuation bytes the line:

res = (res << 6) | (cc & 0x3F);

Might even wrap around, though that appears to not matter. The lines after 
the loop:

res |= ((l_uint32)(c & 0x7F) << (count * 5)); /* add first byte */
if (count > 5 || res > MAXUTF || res < limits[count])
return NULL; /* invalid byte sequence */


Appear to be the actual issue. count*5 specifically when count is seven 
will be greater than or equal to 32, making the shift undefined behavior. A 
simple fix here is to just move the line to add the first byte after the if:

if (count > 5 || res > MAXUTF || res < limits[count])
return NULL; /* invalid byte sequence */
res |= ((l_uint32)(c & 0x7F) << (count * 5)); /* add first byte */


Hmm it appears that the if does use res, which I did not notice before. It 
should be:

if (count > 5)
return NULL; /* invalid byte sequence */
res |= ((l_uint32)(c & 0x7F) << (count * 5)); /* add first byte */
if (res > MAXUTF || res < limits[count])
return NULL; /* invalid byte sequence */
 

On Monday, February 16, 2026 at 10:05:03 AM UTC-5 Sergey Bronnikov wrote:

Hello,

there is an issue with undefined behavior in lutf8lib.c because shift 
exponent may be too large for 32-bit type 'l_uint32':

lutf8lib.c:67:34: runtime error: shift exponent 35 is too large for 32-bit 
type 'l_uint32' (aka 'unsigned int')
    #0 0x56308fd6face in utf8_decode 
/src/testdir/build/lua-master/source/lutf8lib.c:67:34
    #1 0x56308fd6f4fd in utflen 
/src/testdir/build/lua-master/source/lutf8lib.c:99:22
    #2 0x56308fd05fe9 in precallC 
/src/testdir/build/lua-master/source/ldo.c:663:7
    #3 0x56308fd0687c in luaD_precall 
/src/testdir/build/lua-master/source/ldo.c
    #4 0x56308fd40413 in luaV_execute 
/src/testdir/build/lua-master/source/lvm.c:1729:22
    #5 0x56308fd06ce3 in ccall 
/src/testdir/build/lua-master/source/ldo.c:774:5
    #6 0x56308fd03b69 in luaD_rawrunprotected 
/src/testdir/build/lua-master/source/ldo.c:166:3
    #7 0x56308fd07cd8 in luaD_pcall 
/src/testdir/build/lua-master/source/ldo.c:1096:12
    #8 0x56308fcfaaa0 in lua_pcallk 
/src/testdir/build/lua-master/source/lapi.c:1097:14
    #9 0x56308fcf1435 in LLVMFuzzerTestOneInput 
/src/testdir/tests/capi/luaL_loadbuffer_test.c:34:3

Steps to reproduce:

make MYCFLAGS=-fsanitize=undefined MYLDFLAGS=-fsanitize=undefined
echo "dXRmOC5sZW4n2b0B/4+Pj4+Pj48AACc=" | base64 --decode | ./lua -

Lua version: c6b48482

Sergey

-- 
You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion visit https://groups.google.com/d/msgid/lua-l/9e163dac-607f-4eee-b2a7-d2b87506b91bn%40googlegroups.com.