Re: [PHP-DEV] [Concept] declare(strict_identifiers=1)

 > A couple of people have touched on this, but I don't think it's been
 > directly addressed, so I'll ask it more explicitly: what do these terms
 > mean?
 > - UAX31-R1-2
 > - the standard Default-Ignorable Exclusion Profile
 > - NFC

In plain terms:

**NFC.** Unicode can write some characters more than one way. An o-umlaut
is either one character, U+00F6, or two: a plain o followed by a combining
diaeresis. Normalization Form C is the form that uses the single character
wherever one exists. Requiring NFC means one spelling per name.

**UAX #31.** The Unicode annex that says which characters a programming
language should allow in identifiers: letters, digits, marks and connecting
punctuation, but not general punctuation, symbols or formatting characters.
"R1-2" only means "we follow it with a stated list of changes" rather than
"we follow it exactly".

**Default-Ignorable Exclusion Profile.** One of those stated changes, and a
standard one that Unicode itself defines rather than something I made up:
drop the characters that are invisible by design -- zero-width joiners,
variation selectors and the like.

 > what is the summary that goes into the manual and error messages?
 > "Class name doesn't meet UAX31-R1-2" would be about as meaningful to most
 > people as the infamous "Unexpected T_PAAMAYIM_NEKUDOTAYIM".

You are right that this is the actual deliverable, and I had not written
it. Attempting it, with the standard's name appearing nowhere:

     Identifier contains U+00A0 NO-BREAK SPACE, which is not allowed in a
     name

     Identifier is not valid UTF-8 (invalid byte 0xA9 at offset 0)

     Identifier "gro<U+0308>sse" is not in Unicode normalization form C;
     write it as "gr<U+00F6>sse"

The third one bears on your second question.

 > The other thing that I'm not entirely clear on is how much of this is or
 > should be about *rejecting* names, and how much about *normalising* them
 > - bearing in mind we already perform some normalisation in the form of
 > ASCII case folding.

That is the sharpest thing anyone has said in this thread, and I did not
have it clear in my own head. Sorting the four things I have been bundling
together along that axis:

   not valid UTF-8     only rejectable; there is nothing to normalise to
   invisible chars     only rejectable; removing them would change meaning
   not NFC             either -- and because the engine knows the composed
                       spelling, a rejection can print it, which gets most
                       of the benefit of normalising without the engine
                       quietly editing your source
   ASCII case folding  already normalisation, and incomplete:
                       Stra<U+00DF>e and STRA<U+00DF>E are one class,
                       Stra<U+1E9E>e is another, Strasse is a third

They also have four different costs, which I can now put numbers to, from
4,863 packages and 520,802 files:

   not valid UTF-8     2 identifiers
   invisible chars     68, and not all accidental -- math-php spells
                       variables like 
<U+27EE>1<U+00A0><U+2212><U+00A0>p<U+27EF><U+02E3>,
                       with U+00A0 inside the name, on purpose
   not NFC             0, across 627,515 files in both corpora
   case divergence     19, in case-insensitive positions

Bundling four rules with four cost profiles behind one mechanism was the
mistake. Larry, Claude Pache and you have each pushed on a different corner
of the same thing.

 > Perhaps we need to step back and define the *problem statement* more
 > clearly, rather than starting with a *solution* and trying to define its
 > benefits?

Yes. I started from a mechanism and have been arguing backwards from it all
week. Trying it the other way round:

**PHP's identifier rule is expressed in bytes and says nothing about
characters.** Three things follow. Two identifiers a reader cannot tell
apart may be distinct to the engine. An identifier may be text that is not
well-formed in any encoding. And the engine's own case-insensitive
matching, which is a normalisation, is defined over a 26-letter subset of
what an identifier may contain.

Underneath that sits a question nobody has answered: **are non-ASCII
identifiers a supported feature of PHP?** The manual says they are not, and
explains that they work because of how UTF-8 happens to encode. 1,447 of
them, in 25 of the 5,000 most-installed packages, say otherwise -- Vincenty
geodesy in mjaschen/phpgeo, Latte's U+029F prefix in tracy/tracy, Russian
schema accessors in wsdltophp/packagegenerator, and mathematical formulae
as variable names in markrogoyski/math-php.

I do not think any one of those four items justifies a language change on
its own, and I would rather say so than keep hunting for an argument that
makes it sound bigger. Together they say that PHP's identifier rule was
never designed, only inherited, and that the ecosystem has quietly built on
it anyway. Whether that is worth fixing, and in which direction, is a
question for the list rather than for me.

One observation, and then I will stop reaching for mechanisms. What falls
out of your reject/normalise split is not one feature but three, with three
different audiences: a diagnostic for invisible characters, which needs no
opt-in and is exactly what Claude Pache described wanting; a rule about
well-formedness, which has to reckon with symfony/cache; and a conformance
rule about which characters are permitted at all, which is the only part
that would break math-php 888 times and therefore the only part that
plausibly needs opting into. NFC costs nothing either way and can ride
along with whichever of those happens.

But take the problem statement first. I owe the thread that much before
proposing anything further.

Tooling, raw scanner output and per-identifier CSVs, if anyone wants to
check the numbers rather than take them: https://github.com/Otzie2023/PHP

Regards,
Luca