Re: Normalisation and matching

Adam M. Costello wrote:


>The other way to make local parts case-insensitive and case-preserving
>is to leave the case-folding step in Nameprep, but use mixed-case
>annotation to allow it to be undone.  There are a handful of special
>cases that this wouldn't cover (sharp s, titlecase), but it would be
>compatible with existing mail servers. 

Does titlecase exist after a text has been normalised using NFKC?
I would expect NFKC to remove titelcase as it is a purely typographic matter.

sharp s should not be matched to ss (one of the reasons is just the
fact that you cannot know if uppercase ss is SS or sharp s. Unicode
should include a "lower case sharp s character code" if that is wanted).



> Why is it so important to preserve
>case in identifiers that are case-insensitive?  Even if case is not
>preserved, the identifier will still work, and still be recognizable to
>humans.  Case preservation seems desirable, but not essential.

Case is for many a part of the identity. As a e-mail address will be shown
to many people, you want it to be shown as you write it.
Matching is case-insensitive - everybody do not know what case you prefer
on your e-mail address (and case is not spoken over a phone when you
give your address).

If we cannot handle ACE->UCS with case preservation we could as well use
a one way hash for the ASCII world.

>
>>      >An "internationalized local part" (ILP) is anything that satisfies
>>      >both of the following conditions: 
>>          >(1) It conforms to the same
>>          >syntax as a non-internationalized local part except that non-ASCII
>>          >Unicode characters are allowed wherever ASCII letters are allowed.
>>          (2) It is normalised using NFKC and have no forbidden
>>          characters (see NAMEPREP).
>>          (3) Maximum length is the same number of characters as
>>          in legacy ASCII local parts. Though it is recommended that
>>          the limit to be so large that it will never matter.
>>      The ToASCII operation must aways succeed on the above.
>>      (and Punycode cannot not have a 59 character limit)
>
>(3) is implied by (1), so there's no need to state it.
OK. But that may break you 59 character limit.
An ASCII local part can contain more than 59 characters.

>>      An IMA is always normalised using NFKC.  In free form text
>>      there may be e-mail addresses using wide characters or other
>>      unnormalised forms.
>
>And those unnormalized forms are not IMAs?  But they must be accepted by
>application user interfaces and converted to IMAs?  Then why not just
>call them IMAs?  If it looks like a duck and quacks like a duck...

No, we must separate what is used in protocols and what is used
in user interfaces. In a protocol a well defined simple format is very
much desired. IMAA should define how non-ASCII e-mail addresses are
used in protocols, not how a user interface should handle them.
By having IMAs being normalised using NFKC in protocols makes
everything much simpler to handle while not removing anything vital.

>
>>      The character SHARP S (es-zet) does not match "ss". That is:
>>      Lasse do NOT match La<sharp s>e.
>
>But does LASSE match La<sharp-s>e? 

NO

> The former is the proper all-caps
>form of the latter, so they should match. 

Only in some cases in germany. The discussions on the IDN list have shown
that it is not the general case even in Germany.

They do not match in Swedish.


>Besides, the Unicode Consortium already defines how to do
>case-insensitive matching of Unicode strings; what makes us think we
>know better?

They do not say that ss and sharp s should match. They have a
single to single matching that is good and easy to use.
They have a selection of special casing of which the sharp s handling
in one. You are not required to use it.
And I have seen so many examples of problems with sharp s and ss, that 
it would be bad to include it.

>>      >An "IMA-aware mail address slot"...
>>      As an IMA always is normalised the slot will always contain
>>      a well defined, easy to handle, e-mail address.
>
>I don't see why we should impose that constraint.  An IMA-aware slot
>can impose as many or as few additional constraints as it likes.  For
>example, it could require normalized UTF-8, or it could require ASCII
>(using ACE when necessary), or it could allow non-normalized UTF-8, or
>it could allow non-Unicode charsets (like iso-2022-jp or Big5)
>which are implicitly non-normalized because normalization is defined
>only for Unicode.

An IMA-aware mail address slot must use UCS normalised NFKC to make
things simple. You do not want to spend resources on matching @ with "wide @".
Slots with other encodings are something else
and should not be recommended. How you encode UCS is up to the protocol.

>
>Also, IDNA does not impose this constraint, so what makes IMAA different
>in this respect?

IDNA failed with this. It is one reason why I think the IDNA RFC is
unclear.

-
Maybe the question is: what is IMAA good for?
I want a RFC defining a clear standardised simple format to use
for e-mail addresses with non-ASCII characters together with clear
and simple rules of how they are to be compared. It should also
define how they are to be encoded when transmitted over legacy
e-mail systems.

I do not want something that says that in an international context
you can transmitt characters unnormalised or in many character sets.
That does not make interoperability work.

    Dan