Re: Problems of Internationalized Mail Address eXtensions (IMAX)

Paul Hoffman / IMC <[email protected]> writes:

> At 1:50 PM +0100 2/25/03, Simon Josefsson wrote:
>>>> Punycode decoding is not optional if the MTA wants to support
>>>> non-ASCII.
>>> Where in the IMAA document does it say that? I believe you are
>>>  completely wrong here.
>>
>>Are you saying that if I implement a MTA and want to support non-ASCII
>>mail addresses in the places where MTAs use ASCII mail addresses
>>today, that MTA need not implement punycode decoding?
>
> You are (again) confusing the protocol with the implementation. The
> protocol does not require these things; the implementation might.

Right, I was talking about the implementation, I tried to make that
clear by saying "the MTA" rather than "the specification".  Isn't (one
of) the goal of the IMAA protocol to make it possible for MTA
implementations to support non-ASCII?  Then whether the implementation
or the specification is generating the requirement seems like an
academic point.  The end result is that punycode decoding is required
in the implementation, which is what I consider the problem.  If a
solution that didn't involve encoding techniques such as punycode
could be developed, I think that should be preferred.

>>If so, how would you translate an incoming punycoded string into
>>non-ASCII data that is stored in the log file, for instance?
>
> MTA implementations that want to write into log files already need
> Punycode decoding for the host names. Your complaint here is invalid.

Obviously we are interpreting IMAX differently, or you wouldn't say
that.  Now that you write this I would agree that IMAX is unclear on
one thing: does IMAX make the RHS of the email address a (in IDNA
terminology) a IDN-aware domain name slot?  I think it should.  It
doesn't make sense to negotiate non-ASCII and then simply don't take
advantage of that and use IDNA for the RHS, treating it as a
IDN-unaware domain name slot.

IMAX authors, perhaps add an example (and text to go with it) that
illustrates non-ASCII RHS too.

MAIL FROM:<UTF-8=E4=E8@=E6=E96=E87.com>

if this is what you intend?  The alternative would be

MAIL FROM:<[email protected]>

but then IMHO the whole point of IMAX falls: that you can support
non-ASCII using raw charset encodings instead of application specific
encodings.

I interpreted IMAX as providing a IDN-aware domain name slot for the
RHS too, where you could send non-punycoded data.

>>If you are saying that the MTA should put the IMAA encoded mail
>>address in the log file, I'd say then that MTA doesn't support
>>non-ASCII.
>
> You are free to say that. Others would disagree. In the case of IMAX,
> what would you want in your log file. All UTF-8? That means you need
> converters from every accepted charset to UTF-8. Careful sysadmins
> would probably want to know *exactly* what came in, not some converted
> form, but that means that their log file would have multiple charsets
> in it, which would make display a mess. A reasonable option is to
> store the addresses as ACE and to have a log-file viewer that converts
> on display (and has an option for not converting).
>
> Again, this is an implementation issue, not a protocol issue.

Yes.  But it is an important point.  A internationalization solution
that doesn't consider these practical issues is of only theoretical
value.

I would want the log file to contain characters that can be read
without special IDNA/IMAA/IMAX aware programs.  I.e., if the system
uses UTF-8 as the system encoding, I'd want the log file to be in
UTF-8.  If the system uses ISO-8859-1, the log file should be in
ISO-8859-1 (and the application must cope with data that can't be
represented somehow).

Yes, the application must know how to convert alien (but charset
tagged) data into the system charset.  But IDNA and IMAA have the same
characteristic: it require the application to convert Unicode (which
is the only charset IDNA/IMAA accept) to the system charset.  So I
cannot see where the big difference lies?

I agree careful sysadmins want to see exactly what came in.  The only
way to represent that, unless the system uses the same charset as the
data that came in, is to print the charset of the incoming data and
the byte sequence.  The same is true today on a ISO-8859-1 system that
receives Unicode via IDNA.

It seems we disagree that it is reasonable to require users to use
special applications to view log files, or edit configuration files,
etc.  Personally, I don't use applications that have configuration
files or log files that can't be manipulated using text operations.  I
do suppose many Microsoft Windows users would find your approach
acceptable though, since that's what they are accustomed to.  IMHO a
solution must be able to accomodate both users.

>>   An essential feature of supporting non-ASCII is to make it
>>possible for the user of the application to actually see the
>>characters.  ASCII encoding them and displaying them to the user
>>doesn't make the application support non-ASCII in practice.  It would
>>be like claiming to support Unicode in a terminal emulator when it
>>only displayed Base64 encoding of the UTF-8 encoded Unicode code
>>points.
>
> IMAA describes in detail when and how to display the Unicode form to
> the user; IMAX mostly glosses over this.

Yes, IMAX is not a final document so this isn't surprising.  Although
for IMAX, those issues are simpler since IMAX allows implementations
to use charsets that the system already support natively.

>>  >>A punycode encoder is required if the MTA handle non-ASCII data in
>>>>decoded, normal, format.  Like in the user interface for /etc/aliases,
>>>>/etc/mail/virtusertable etc.
>>>
>>>  Neither of those are controlled by the MTA. This is getting pretty silly.
>>
>>That was not a generic example, it was an example for one MTA
>>implementation: Sendmail.  It uses and control those files.
>
> And, again, you are mixing up protocols with implementations.

I'm sorry, I'll try to make it more clear when I talk about the
implementation or the specification.  If you are saying that we should
simply ignore all implementation related aspects in a proposed
solution, then I guess I simply don't agree with that.  I'll continue
to relate a proposal to the real world.

>>  >>   If it doesn't handle non-ASCII in normal
>>>>format, it might as well not support non-ASCII at all since the user
>>>>would never notice the different.
>>>
>>>  You are mixing up the MTA and the MUA.
>>
>>I wasn't clear.  I meant the user of the MTA, i.e., the administrator.
>>Administrators have non-ASCII requirements too.
>
> Correct, and IMAA describes when and how to convert for display.

Right.  This is what cause the dependence on punycode decoding.  Since
administrators not only view non-ASCII but input non-ASCII too,
punycode encoding is required too.

>>MTA implementations, nor internationalization solutions for MTAs,
>>exist in a vacuum.  If it is impossible to implement an
>>internationalized product and being compliant, the specification has a
>>problem.
>
> Of course. Nothing in IMAA makes it "impossible to implement an
> internationalized product".

Cool.  Then, perhaps, what we have is two solutions that can implement
an internationalized product.  I'm trying to convince myself which of
them is the better approach.

>>  > But you keep talking about the need to handle fallback. Handling two
>>>  protocols is not easier than handling one in any universe.
>>
>>True.  Yes, the fallback is a problem.  Hm.  Perhaps those interested
>>in non-ASCII need to require the use of modern software at the
>>receiver and the sender, then implementations doesn't need to
>>implement the fall back case.
>
> That's not what the IMAX document says.

Right, I proposed something new.

> If you want to propose a ESMTP extension with no fallback, either
> change IMAX or create your own Internet Draft. In either case, you
> will have to say explicitly how this will interact with SMTP servers
> that do not support the new protocol, how bounces would be handled,
> how users would know if they could send a message, and so on. I
> think when you write that, if you do so honestly, you will see that
> it would be silly to propose such a solution.

Discarding it as silly seems a bit premature to me.  Having such a
proposal, that discusses all the consequences you mention seems like a
valuable contribution to this discussion.  But I guess it is easier to
advocate one solution if the competition are discarded early on...

>>  >>  > Clean is in the eye of the beholder. You and I like UTF-8, but many
>>>>   > people don't. Forcing them to use our preferred charset isn't a good
>>>>   > practice if it can be avoided.
>>>>
>>  >>I agree completely.  This is one of my problems with IDNA and IMAA, it
>>>>forces Unicode on everyone.
>>>
>>>  Unicode is not a charset.
>>
>>I'm not sure if you genuinely missed my point due to this
>>misunderstanding, but assuming you did, let me correct myself: replace
>>"Unicode" with "Any charset encoding format of Unicode".
>
> I think I hear you saying that you think that the protocols should
> allow any repertoire and any encoding of those repertoires. If so, we
> certainly disagree. The IETF is not very keen on creating protocols
> for which there would be limited and unpredictable
> interoperability. Other standards group might not be so picky.

That is stretching it a bit, I think.  I believe that a solution worth
its salt should consider existing habits, and whether we like it or
not there is more than charset used on the Internet.  MIME appears to
acknowledge this and is rather successful.  HTML acknowledge this and
is rather successful.  Same for HTTP.  Come to think of it, I can't
recall any successful internationalization product the IETF has
produced to counter my examples, can you help me?

If you are speaking for IETF, I find it interesting that RFC 2277
"IETF Policy on Character Sets and Languages" says that protocols MAY
allow use of any repertoire.  It doesn't say that it is a bad idea to
allow more than one charset.  I agree with that document, let's
require the use of UTF-8 in protocols, but allow negotiation of other
charsets to smooth transition and deployment.