Re: Normalisation and matching

Dan Oscarsson <[email protected]> wrote:

> Does titlecase exist after a text has been normalised using NFKC?

Good question!  Nope, NFKC-normalized text never contains titlecase
characters, so titlecase isn't a problem afterall.

> Case is for many a part of the identity.  As a e-mail address will be
> shown to many people, you want it to be shown as you write it.

Yes, we want that, but there are other things we want too, and sometimes
we have to make choices.

> Matching is case-insensitive - everybody do not know what case you
> prefer on your e-mail address (and case is not spoken over a phone
> when you give your address).
>
> If we cannot handle ACE->UCS with case preservation we could as well
> use a one way hash for the ASCII world.

Do you see what you're saying?  Given:

 1) José
 2) josé
 3) kmcsi5csxy

you're saying that if you can't show (1), then it doesn't matter whether
you show (2) or (3), because neither is any better than the other.  Are
you serious?

We already went through all this for IDNA.  Traditionally domain names
are case-insensitive and case-preserving, but the case-preserving
part was relaxed for IDNs because it wasn't deemed worth the
additional required complexity.  We can add an optional mechanism for
case-preservation later.

We could repeat the same arguments for IMAA, but I see no reason why it
would play out any differently than it did for IDNA.

> An ASCII local part can contain more than 59 characters.

Yes, and the current draft supports that, it just doesn't support more
than 63 characters (59 + ACE infix) in any single encoded segment.

For example,

0iesg1n8jok5ay5dzabd5bym9f0cm5685rrjetr6pdxa-0iesg1b1abfaaepdrnnbgefbaDotcwatmq2g4l

is an 83-character local part that is valid under imaa-02, because neither
of its two segments exceeds the limit.  But

0iesg1989aomsvi5e83db1d2a355cv1e0vak1dwrv93d5xbh15a0dt30a5jpsd879ccm6fea98c

would be invalid, even though it's shorter, because it's all one
segment.

> IMAA should define how non-ASCII e-mail addresses are used in
> protocols, not how a user interface should handle them.

Just the opposite.  Initially, non-ASCII mail addresses will not appear
in protocols at all (only ASCII address will), but they will appear in
user interfaces.  The primary issue that IMAA needs to address is how
to bridge the gap between non-ASCII user interfaces and existing ASCII
protocols.  Non-ASCII mail addresses won't appear in protocols until
new protocols are defined, and each new protocol can then specify how
non-ASCII mail addresses are to be represented in that protocol.  If
normalized Unicode is the best way, then that's what they'll specify,
but there's no need for us to prescribe that choice now.

> You do not want to spend resources on matching @ with "wide @".

Obviously *you* don't.  When you define an IMA-aware protocol, you are
welcome to restrict the syntax so that receivers don't need to bother
checking for fullwidth @.

But there may be some protocol designers who believe that it's important
to preserve not only mixed case, but also other presentational
details that would be destroyed by normalization (like a fullwidth
@, or a superscript 2).  They might decide that it's better to
transmit/display/store the mail address exactly as it was originally
typed, and perform the normalization only momentarily for the purpose of
matching it to the appropriate mailbox, and then discard the normalized
form.

There is no need for us to decide this question now for all future
protocols.  As long as the equivalence relation is standard, each
protocol can use whatever equivalent form it thinks is best.  Gateways
between protocols will of course need to respect the requirements of
each protocol, and perform any necessary translations, but that's the
nature of a gateway.

And again, I see no reason why IMAA and IDNA should take different
courses on this issue.  IDNA did not prescribe any particular form for
IDN-aware slots.

> Slots with other encodings are something else and should not be
> recommended.

They are neither recommended nor discouraged.

> > Besides, the Unicode Consortium already defines how to do
> > case-insensitive matching of Unicode strings; what makes us think we
> > know better?
>
> They do not say that ss and sharp s should match.  They have a
> single to single matching that is good and easy to use.  They have a
> selection of special casing of which the sharp s handling in one.  You
> are not required to use it.

SpecialCasing.txt is for case mapping, not caseless matching.  The
relevant file for caseless matching is CaseFolding.txt.  It contains the
following comment:

    Note that where they can be supported, the full case foldings are
    superior: for example, they allow "MASSE" and "Maße" to match.

That makes it very clear that SS and sharp-s are supposed to match.
They both get folded to ss.

There is nothing locale-specific about that mapping.  The only
locale-specific mappings relate to Turkish i.

AMC