Re: Bidi issues

Roy Badami <[email protected]> wrote:

> I think I am starting to get a handle on at least part of the
> STRINGPREP bidi restrictions.  (Though with the disclaimer that I'm no
> bidi expert, so the examples below could be wrong.)

It looks to me like you're becoming one, seeing you find all these
corner cases.  :)

> If you allow labels to mix LTR and RTL characters, you get effects
> such as the following:
>
>       abCD.EFgh.com (logical order)
> 	abFE.DCgh.com (display order)
>
> I think it was reasonble for the authors of IDNA to conclude that this
> is just too confusing.

I don't know about the other authors, but I don't remember being aware
of this particular pitfall (an individual label being split apart).  I
remember being aware that the bidi step was needed in Stringprep in
order to prevent the existence of distinct labels that get displayed the
same.  (But now you have discovered that it doesn't quite accomplish
that goal.)

> > If we don't want the bidi check to apply to the whole local part,
> > but rather to individual segments, then I think we'll need to break
> > Nameprep into two halves.
>
> The problem with this is that it elevates the segmentation mechanism
> from an internal mechanism of ToASCII/ToUNICODE to a user-visible
> feature of IMAs.

Well, it increases the number of user-visible effects of the
segmentation mechanism.  Already, if users (or programmers) want to
add/remove strings to/from local parts (like "owner-" or "-request"
or "+tag"), the segmentation mechanism makes it safe to perform the
operation on either the ASCII form or the non-ACE form, provided that
the boundary is between segments, or within a protected segment.

With the split-Nameprep idea, there would be another user-visible
effect:  Mixing LTR and RTL characters would be allowed within local
parts, provided they are not mixed within any individual segment.

> I think there are three basic choices here (from most restrictive to
> least restrictive):
>
> 1.  Apply the stringprep bidi check to the entire localpart.
>
> 2.  Apply a different, (presumably less restrictive) bidi check to the
>     local part.
>
> 3.  Use stringprep without the bidi restrictions (nameprep would
>     still prohibit the directional formating codes, though).  Make
>     a recommendation that care be taken when mixing LTR and RTL
>     characters in e-mail addresses intended for human use.
>
> Option 3 is, as I understand it, essentially what IRI currently does.

It looks to me like the IRI draft says that the bidi restrictions
(roughly the same as in Stringprep) apply to all bottom-level components
of all IRIs intended for human use, although the restrictions might be
unenforceable in some cases.

> Option 1 is the status quo in the draft.  If option 2 were chosen, we
> may well end up with what Adam proposes, but I don't think the nature
> of the bidi restrictions should be driven by the current design of the
> ToASCII algorithm.

I made an intuitive leap without considering the full design space.

The model suggested by the IRI draft is that complex identifiers (like
IRIs and IMAs) be thought of as made up of components, which may in turn
be thought of as made up of smaller components, and so on, eventually
bottoming out.  The bidi restrictions are applied to the bottom-level
components.

The model does not suggest that any special bidi restrictions be
invented for special types of components (though it doesn't explicitly
discourage such invention either).  Nor does it suggest that some
components might be exempt from the bidi restrictions.  (It concedes
that the restrictions might be unenforceable in some cases, but they are
quite naturally enforceable for IMA local parts.)  The only degree of
freedom highlighted by the IRI model is how to define the components.

For local parts, there are three ways we could define the bidi
components:

  * The local part is all one bidi component (as in the current draft).

  * Each unprotected segment is a bidi component (and the protected
    segments are the delimiters between the bidi components).  (This
    would require reordering the algorithm.)

  * Something else.  (This would require an addition to the algorithm.)

It's possible that there is a compelling reason use a different
segmentation rule for the bidi check than for the encoding.

> Option 1 worries me a bit, because if legacy software is going to
> construct addresses of the form
>
>       owner-<ace-local-part>@domain
>
> indiscriminately, that's an open invitation to MUA authors to violate
> the standard and produce a non-conformant ToUnicode in an attempt to
> be 'helpful' to users by displaying such domains in a comprehensible
> form.

It's not going to be very "helpful" when users discover that those
comprehensible names don't actually work as mail destinations (because
they are rejected by any conformant ToASCII).

But I share your concern that with the current draft, FOO can be valid
while owner-FOO is invalid.  That's asking for trouble.

> But note that applying the stringprep bidi restrictions to individual
> segments is not strictly less restrictive than option 1.  Consider,
> for instance the localpart A1-2B (where A and B are RTL characters).
> This is valid under option 1 (and indeed as a domain label under
> IDNA).  The entire localpart satisfies the stringprep rules, but the
> individual segments don't (because they don't each begin and end with
> an RTL character).

Good observation.

> I think the bidi restrictions in stringprep don't actually achieve
> their goal of ensuring that you can't have two different labels that
> render the same.
>
> Consider the labels:
>
>       A-123,456B
>
> and
>
>       A456,-123B
>
> Here, A is HEBREW LETTER ALEF, B is HEBREW LETTER BET (or any
> characters of bidi class R that you like, but *not* arabic letters,
> which are class AL) and the comma is actually ARABIC COMMA U+060C (or
> any character of class CS or ES).
>
> As far as I can tell these both pass nameprep with UseSTD13ASCIIRules
> set, and they both render identically under bidi as:
>
>       B-123,456A
>
> If you don't care about UseSTD13ASCIIRules, you can replace ARABIC
> COMMA with COMMA, SOLIDUS or COLON.

That's rather alarming.

[By the way, it's UseSTD3ASCIIRules, not UseSTD13ASCIIRules.]

> There are also other real nasties with this display model:
>
>         ARABIC.3com.com (logical order)
>         3.CIBARAcom.com (display order)
>
>         ARABIC.3-com.com (logical order)
>         3.CIBARA-com.com (display order)
>
>         HEBREW.3-com.com (logical order)
>         3-.WERBEHcom.com (display order)

For the sake of argument, do you see a simple tweak to the bidi
restrictions that would improve these corner cases?  For example,
expanding LCat and/or RandALCat to include more bidi classes?

(I'm just talking about fixing the cases where an invidual component
gets split apart.  For the cases where whole components get ordered
ambiguously, Martin explained why we're stuck with them.)

Hmmm, I just noticed that the IRI draft talks about "left-to-right
characters" and "right-to-left characters" without refering to any
specific bidi classes.  Are those phrases precisely defined anywhere?
If not, is the imprecision deliberate?

> In the case of IMA we need to warn people (at least) against addresses
> such as:
> 
> 	[email protected] (logical order)
> 	[email protected] (logical order)
> 
> both of which display as 
> 
> 	[email protected] (display order)
> 
> but I think there are other cases as well.

If we do this, then we should likewise add a warning to IDNA about
123.ABC.com and ABC.123.com.

As for the "other cases", do they all involve non-directional
components (components that contain no left-to-right characters and
no right-to-left characters)?  Maybe a general warning about such
components would be helpful.

AMC