Re: Normalisation and matching

Dan Oscarsson <[email protected]> wrote:

>      An IMA is NOT case folded.  An IMA is required to preserve case.
>
>      Equivalence of local parts is defined in terms of the dequoted
>      form (see above) and case-insensitive matching.

I can think of two ways to make local parts case-insensitive and
case-preserving.

One way is to exclude the case-folding step from Nameprep, and to
define the comparison procedure as caseInsensitiveUnicodeCompare(
ToUnicode(X), ToUnicode(Y) ), as opposed to the current procedure which
is caseInsensitiveASCIICompare( ToASCII(X), ToASCII(Y) ). The drawback
of the UnicodeCompare approach is that it's not compatible with existing
mail servers.  Users could not create internationalized local parts
until the server was updated to perform the new comparison.  With the
ASCIICompare approach, that's the same operation that existing mail
servers already perform, so users could create internationalized local
parts immediately.

The other way to make local parts case-insensitive and case-preserving
is to leave the case-folding step in Nameprep, but use mixed-case
annotation to allow it to be undone.  There are a handful of special
cases that this wouldn't cover (sharp s, titlecase), but it would be
compatible with existing mail servers.  This can always be added later
as an option.  But it's fairly complex, which is a reason to make it
optional rather than required.  Why is it so important to preserve
case in identifiers that are case-insensitive?  Even if case is not
preserved, the identifier will still work, and still be recognizable to
humans.  Case preservation seems desirable, but not essential.

>      >An "internationalized local part" (ILP) is anything that satisfies
>      >both of the following conditions: 
>          >(1) It conforms to the same
>          >syntax as a non-internationalized local part except that non-ASCII
>          >Unicode characters are allowed wherever ASCII letters are allowed.
>          (2) It is normalised using NFKC and have no forbidden
>          characters (see NAMEPREP).
>          (3) Maximum length is the same number of characters as
>          in legacy ASCII local parts. Though it is recommended that
>          the limit to be so large that it will never matter.
>      The ToASCII operation must aways succeed on the above.
>      (and Punycode cannot not have a 59 character limit)

(3) is implied by (1), so there's no need to state it.

The 59-character limit applies to encoded segments of the local part,
not to the entire local part.  That issue is in the could-be-reopened
category, because it hasn't been discussed much.  If we raise the limit
on Punycode output size, then some existing implementations of Punycode
won't work for IMAA.  If the limit is much larger than 59, then the
published Punycode algorithm becomes inappropriate, because it's an
O(n^2) algorithm.  I think there exists a more complex algorithm that
is O(n log n).  Once again, there is a tradeoff: is support for long
segments worth the added complexity?

>      An IMA is always normalised using NFKC.  In free form text
>      there may be e-mail addresses using wide characters or other
>      unnormalised forms.

And those unnormalized forms are not IMAs?  But they must be accepted by
application user interfaces and converted to IMAs?  Then why not just
call them IMAs?  If it looks like a duck and quacks like a duck...

>      The character SHARP S (es-zet) does not match "ss". That is:
>      Lasse do NOT match La<sharp s>e.

But does LASSE match La<sharp-s>e?  The former is the proper all-caps
form of the latter, so they should match.  Does LASSE match lasse?  Yes,
they are all-ASCII strings that already match according to existing
rules.  Therefore lasse matches La<sharp-s>e, unless matching is
non-transitive, which would be screwy (that would prevent the matching
from being implemented as exactCompare(Canonical(X),Canonical(Y)), which
is how matching is customarily implemented).

Besides, the Unicode Consortium already defines how to do
case-insensitive matching of Unicode strings; what makes us think we
know better?

>      >An "IMA-aware mail address slot"...
>      As an IMA always is normalised the slot will always contain
>      a well defined, easy to handle, e-mail address.

I don't see why we should impose that constraint.  An IMA-aware slot
can impose as many or as few additional constraints as it likes.  For
example, it could require normalized UTF-8, or it could require ASCII
(using ACE when necessary), or it could allow non-normalized UTF-8, or
it could allow non-Unicode charsets (like iso-2022-jp or Big5)
which are implicitly non-normalized because normalization is defined
only for Unicode.

Also, IDNA does not impose this constraint, so what makes IMAA different
in this respect?

AMC