Re: standardized local-part structuring

Dave Crocker <[email protected]> wrote:

> This evening's clarification -- that imaa wants to modify basic
> Internet mail parsing rules

IMAA does not modify the parsing rules in any existing context.  In
message headers and SMTP commands, IMAA does not alter the syntax of
mail addresses; they continue to be ASCII-only, and the old parsing
rules still work.

But there need to be some contexts where a new syntax is used, a syntax
that allows non-ASCII characters.  If there are no such contexts, then
we haven't internationalized anything.

Where does this new syntax get used?  In new user interfaces of new
internationalized mail applications.  And maybe in new protocols (IMAA
neither encourages nor discourages the introduction of new protocols
that use non-ASCII mail addresses).

When a user types a non-ASCII mail address into a user agent, the agent
needs to parse the address into a non-ASCII local part and a non-ASCII
domain name, and then parse the latter into non-ASCII labels, before it
can perform the encoding.  Obviously, the old parsing rules won't work,
because those rules accept only ASCII characters.  Therefore we must
have new parsing rules for use in this new context.

When the new parsing rules are applied to ASCII-only mail addresses,
they degenerate into the old parsing rules.  Therefore a new application
doesn't actually need to switch between two sets of parsing rules.  The
new rules are a backward-compatible extension of the old rules.

Dave> Doing segmentation at protected boundaries makes the imaa mechanism
Dave> significantly more complicated.

AMC> True, but remember that dividing a local part into segments
AMC> for separate encoding/decoding is not significantly more
AMC> complex than dividing the domain part into labels for separate
AMC> encoding/decoding.

Dave> For domains, the syntactic rules are global, rigid, and
Dave> well-specified.  For local-part, they are varied, unspecified.
Dave> And, of course, they are local.

That's true, but you're changing the subject.  You said segmentation was
complex, and I responded to that criticism by pointing out that we're
already doing an operation of nearly identical complexity on the other
side of the at-sign.

> Internet mail has gotten quite a bit of benefit from avoiding global
> knowledge about local-part internals.  Please do not mess with that
> strategic benefit.

IMAA does not expect anyone to know how local parts are structured.  It
merely uses an encoding that avoids throwing a wrench into the works
whenever possible.

> > ToUnicode step 1 is not strictly necessary, but it avoids
> > gratuitously lowercasing ASCII letters.
>
> IMAA had better not lowercase ASCII, whether gratuitously or not.
> Local-part is defined as being case sensitive.  IMAA needs to work
> within that reality.

More precisely, ASCII local parts are defined to have the following
tricky properties regarding case sensitivity:

    Local parts MAY be case-sensitive, and therefore MUST be treated as
    case-sensitive by anyone who doesn't know for sure; however, the
    authoritative servers who finally decide the issue are discouraged
    from being case-sensitive.  [RFC 2821]

For non-ASCII local parts, we found that this model was just too
tricky to pull off without undue complexity in the spec.  (You think
segmentation is complex?  You should have seen this...)  When defining a
new class of local parts (non-ASCII local parts), we had the opportunity
to use a simpler case-sensitivity model, and we did.  We could have
either defined non-ASCII local parts to be always case-sensitive,
or always case-insensitive.  We chose the latter because it is more
consistent with actual practice and with user expectations.  IMAA makes
no change to the case-sensitivity model for ASCII local parts.

> > Another reason for having this bypass in ToUnicode is to mimic the
> > same bypass from ToASCII, where the the bypass is needed to make
> > sure that pure ASCII strings are never altered in any way.
>
> Parsing/encoding algorithms that need these sorts of special-case,
> look-ahead processing invite mis-implementation.  They certainly
> suggest excessive complexity for a task that is already plenty
> complicated.

The fact that ASCII local parts and non-ASCII local parts use two
different case-sensitivity models (a simple one for non-ASCII local
parts and a trickier one for ASCII local parts) causes ToASCII to need
to check whether its input is pure ASCII, and avoid Nameprep (which
includes case-folding) if it is.

The IDNA ToASCII also contains this bypass, even though domain names are
always case-insensitive and doing case-folding on ASCII labels would not
have altered the domain.  Still, why squash the case if you don't have
to?

By the way, another approach, rather than having the bypass in ToASCII,
would have been to define Nameprep to fold non-ASCII letters but leave
ASCII letters unfolded.  I suggested this, but was overruled.  As I
recall, the main argument against this idea was that deviating from the
Unicode case-folding algorithm in any way would open up a can of worms
for the numerous proposed tweaks and fixes of Unicode case-folding and
normalization, and we'd never reach consensus.

> > Punycode was designed for domain names, where introducing hyphens
> > was not a problem.  It uses all 37 LDH characters to maximize
> > efficiency.  We could define a new encoding very similar to Punycode
> > that uses "9" instead of "-" (at a slight cost in efficiency), but
> > I thought it would be simpler to put a wrapper around Punycode that
> > removes/restores the hyphen.
>
> So, Punycode is not a general-purpose module, but you can hack around
> it to adapt it to the more complex requirements of mail local-part, by
> making things even more complex...

Right.  But I still think it was the simplest of the options, which
were:

  * Put a wrapper around Punycode.
  * Create a slight variant of Punycode.
  * Introduce an entirely new encoding algorithm.

Given that a Punycode implementation is already needed for the domain
part of the address, I think the wrapper idea is the simplest.

> > Also, the grammar above doesn't recognize fullwidth characters in
> > the ACE prefix, but ToUnicode does (because it performs Nameprep
> > before looking for and removing the prefix).
>
> Oh, good.  There is more than one way to do the prefix, too?

Yes.  One of the guiding principles in the design of IDNA (and therefore
IMAA) is that if two strings are equivalent Unicode strings, they had
better be treated the same.  Unicode defines two kinds of equivalence:
canonical equivalence, and compatible equivalence.  It was decided that
compatible equivalence is what we wanted.

"xn--jos-dma" and fullwidth "xn--jos-dma" are equivalent Unicode
strings.  Therefore, if the former gets displayed as "josé", the latter
had better get displayed as "josé" too.

We hope users won't need to type ACE forms, but occasionally they will.
Curiously, while CJK users tend to be quite careful about upper case
versus lower case (much more so than most English speakers), they tend
to be quite careless about fullwidth versus regular width.  (At least,
that's my observation from Japanese web pages.)  I think they would be
mystified if the fullwidth version of the ACE form didn't work.

> > People will naturally expect to be able to send mail to
> > josé[email protected].
>
> The extent to which random users can expect to generate a segmented
> local-part for a particular recipient is entirely outside the current
> scope for existing Internet mail.

If example.net has an existing policy of accepting mail for user+tag and
delivering it to user, then José himself might expect to be able to tell
his friends to send mail to josé+tag, or at least might be disappointed
to realize that he can't.

Also, if a third party already corresponds with several users at
example.net, and is therefore familiar with the user+tag convention
at example.net, they will be astonished that it doesn't work for
josé@example.net.

> "Crosstalk"?

Leakage between two channels.  For example, if you're having a
conversation on an analog cell phone, and you hear some other cell phone
conversation, that's crosstalk.  In the context of encoding strings, if
FOO1 gets encoded as bar1, and FOO2 gets encoded as bar2, and FOO1-FOO2
gets encoded as bar1-bar2, then there has been no crosstalk across
the boundary marked by the hyphen.  But if FOO1-FOO2 gets encoded as
something other than bar1-bar2, that means information must somehow have
leaked across the boundary during the encoding process.

> To the extent that there is consensus to have the local-part be a
> mixture of ace-encoding and classic ascii, then define the ace-encoded
> strings with left/right framing.
> 
> Something like:
>
>     local-part   =   1*(ascii-local / ace-local)
>     ascii-local  =   dot-atom / quoted-string / obs-local-part
>     ace-local    =   ima-prefix  ace-encoded-unicode  ima-suffix
> 
> will do the trick.

I considered that.  It was a precursor to the current IMAA encoding.

Let's explore this idea further.  First, we need to fix it up a bit.
The above grammar allowes multiple quoted strings, which is no longer
allowed except in obsolete syntax.  Let's ignore obsolete syntax for
this discussion.

Since we can't have multiple quoted-strings, the ACE parts are going
to need to go inside the single quoted-string.  It's too hard to write
a grammar for that unless we view the quoting as a separate layer.  In
other words, we defined a grammar for a dequoted-ascii-local-part, and
then quote that as necessary.

We don't want the grammar to be ambiguous, since the purpose is to
indicate how to parse a string, not merely to decide whether a string
is valid, right?  The above grammar is ambiguous unless dot-atom is
restricted to not contain ima-prefix and ima-suffix.

Here's a grammar that is intended to be the same in spirit as the one
you proposed, while dealing with the above concerns:

    dequoted-ascii-local-part = empty-string / 1*segment
    segment = literal-segment / encoded-segment
    literal-segment = 1*ascii-char ; must not contain prefix or suffix
    encoded-segment = prefix nonascii-encoded-as-alphanumeric-ascii suffix

("encoded" refers to more than just Punycode.  It's Nameprep and
Punycode.)

What is the nice feature of this syntax?  The encoded form is likely
to have the same "structure" as the original form, with a one-to-one
correspondence between the encoded "components" and the original
"components", with each "component" encoded independently of the others
(no crosstalk).  Even though we don't know any details of what the
structure or the components might be, we get this nice feature because
the encoding does not muck around with ASCII characters.  It doesn't
delete them, reorder them, or introduce them.  Therefore, as long as the
structure is based on ASCII delimiters, the encoding hasn't interfered.

Oh wait--the encoding *does* introduce some ASCII characters, namely
alphanumeric ASCII characters.  Therefore we do interefere if the
unknown structure uses alphanumeric ASCII characters as delimiters.  But
we don't interfere if only non-alphanumeric ASCII characters are used as
delimiters.

Well, as long as we're mucking with alphanumeric ASCII characters by
introducing them, we might as well reorder them too, if it improves the
encoding.  Indeed, we can make the encoding more compact by tweaking the
grammar to lump alphanumeric ASCII characters together with non-ASCII
characters:

    dequoted-ascii-local-part = empty-string / 1*segment
    segment = protected-segment / unprotected-segment
    protected-segment = 1*nonalphanumeric-ascii-char
    unprotected-segment = 1*alphanumeric-ascii-char

An unprotected-segment might be literal or might be encoded; you just
try to decode it and see what happens.  As before, "encoding" and
"decoding" refer to more than just Punycode; this time they include not
only Nameprep and Punycode but also to a special substring.  The first
step of decoding is looking for a special substring; if it's not there
then the decoding fails and the segment is literal.  Even if the special
substring is present, the decoding might still fail, indicating that the
segment is literal (and misleading, and therefore discouraged).  If the
decoding succeeds, then the segment was encoded.

In the previous grammar, the special substring (prefix/suffix combo)
served two purposes: (1) it marked the boundaries of the segments, and
(2) it made it unlikely that a string intended to be literal would
accidentally be interpreted as an encoded string.

In the revised grammar, the special substring serves only the second
purpose.  We don't need it to mark the boundaries of the segments,
because the segment boundaries are already marked by the adjacency of an
alphanumeric character and a nonalphanumeric character.  Therefore we
are free to use an infix rather than a prefix or suffix, which turns out
to be convenient.

The revised grammar yields a more compact encoding than the previous
grammar, for Latin-based local parts.  For example, "résumé" would have
needed two prefixes, two suffixes, and 6 characters of Punycode, whereas
now it needs only one infix and 4 characters of Punycode.

> > Suppose the manager of the aliases file at example.net creates
> > an ACE alias that displays as niño and expands to multiple
> > addresses.  The IMA-unaware MTA, whenever it expands the ACE alias,
> > automatically looks for a companion alias owner-ACE to use as the
> > envelope From address.  That address will display as owner-niño
> > if there is no crosstalk across protected characters, but it will
> > display as ASCII garbage if there is crosstalk.
>
> 1. Alias expansion is a function of list processing, not classic MTA
> processing.

Sendmail is a classic MTA, and I'm pretty sure it has been performing
alias expansion as far back as I have been using email (14 years).  All
MTAs I've ever heard of perform alias expansion.

Until recently, majordomo was one of the most popular mailing list
processors, and it is still in fairly widespread use.  It has never
performed alias expansion; it has always relied on the MTA to do it.

> Yes it is useful and popular, but let's be clear about where it
> fits in the architecture.  Let's not confuse architecture with
> implementation.

Even if we make the distinction between MTA functions and list
processing functions, they're both infrastructure functions, as opposed
to user-agent functions.  IMAA's goal is to work well even without
changes to infrastructure.

AMC