Re: [RFC PATCH] Fix up cpp_check_xid_property (used by rust)

On 6/11/26 7:48 AM, Jakub Jelinek wrote:
> Hi!
> 
> I believe cpp_check_xid_property (function added to libcpp for the Rust
> FE) is incorrect.
> The bits are documented
>    /* Valid in a C++23 identifier?  */
>    CXX23 = 32,
>    /* Valid in a C++23 identifier, but not as the first character?  */
>    NXX23 = 64,
> So, characters which have both the Unicode XID_Start and XID_Continue
> derived properties have just CXX23 bit set, and characters which
> have just XID_Continue derived property and not XID_Start have
> CXX23|NXX23 bits set.
> So, I believe the function (except for the ASCII fast path which is correct)
> incorrectly returns CPP_XID_START | CPP_XID_CONTINUE for any characters
> which have XID_Continue derived property, and only ever returns just
> CPP_XID_CONTINUE in the ASCII [0-9_] cases.

Sounds right to me.

> The following untested patch should fix that, I guess it would be nice to
> have a testcase, pick some character in XID_Continue and not in XID_Start,
> random choice e.g. SUNDANESE CONSONANT SIGN PASANGAN WA, and check if
> it can be used in the second+ character of identifier (probably it can and
> should continue to do so) and as the first character of identifier
> (likely incorrectly accepted right now).
> 
> 2026-06-11  Jakub Jelinek  <[email protected]>
> 
> 	* charset.cc (cpp_check_xid_property): Return
> 	CPP_XID_START | CPP_XID_CONTINUE only if CXX23 bit is set and NXX23
> 	is not.  If both are set, return CPP_XID_CONTINUE.
> 
> --- libcpp/charset.cc.jj	2026-04-21 18:24:44.122033404 +0200
> +++ libcpp/charset.cc	2026-06-11 13:33:51.356096060 +0200
> @@ -1366,9 +1366,9 @@ cpp_check_xid_property (cppchar_t c)
>   
>     unsigned short flags = ucnranges[mn].flags;
>   
> -  if (flags & CXX23)
> +  if ((flags & (CXX23 | NXX23)) == CXX23)
>       return CPP_XID_START | CPP_XID_CONTINUE;
> -  if (flags & NXX23)
> +  if ((flags & (CXX23 | NXX23)) == (CXX23 | NXX23))
>       return CPP_XID_CONTINUE;
>     return 0;
>   }
> 
> 	Jakub
>