Re: Bidi issues

Roy Badami <[email protected]> wrote:

>  > It's possible that there is a compelling reason use a different
>  > segmentation rule for the bidi check than for the encoding.
>
> I'm not sure there is a compelling reason, but there are many other
> possibilities.  Two that immediately spring to mind:
>
>  * Segment only on dot.  This means that the rules for what is allowed
>    on the LHS and RHS are the same.

Recall that the motivation for the current segmentation is to support
arbitrary existing structure conventions.  For example, if foo is
a local part, then from it one might construct owner-foo, foo+tag,
foo!bar, bar.foo, etc.  We wanted such conventions to continue to work
for IMAs in a way that was compatible with legacy software, so we made
sure that foo was encoded separately from the other pieces, so that
software operating on the ACE form and humans operating on the non-ASCII
form don't confuse each other.

You recently pointed out a hole in this architecture:  The bidi check
is currently applied to the entire local part, which means that even if
foo is valid, owner-foo might be invalid.  Likewise, foo+tag, foo!bar,
bar.foo, etc. might be invalid too.

If we want the bidi check to be as tolerant of existing structure
conventions as the encoding is, then we need the bidi components to be
no larger than the segments.

That still leaves open the question of whether the bidi components
should be smaller than the segments, as in the other possibility you
mentioned:

>  * Segment on _all_ punctuation, not just ASCII puncuation.
>    (Spefically, segment on bidi categories ET, ES, CS and ON)

This is not necessary for supporting existing structure conventions,
because there cannot be any existing conventions based on non-ASCII
punctuation.  As far as I can see, the only motivation for this would
be to allow more fine-grained mixing of LTR and RTL text within a local
part.  I'm skeptical that users would value this added flexibility
enough to warrant the added complexity in ToASCII/ToUnicode.

> I think I have rules which would prevent the A-123,456B case and the
> 123.ABC.com case, but they're not very pretty.  I can tidy them up a
> bit and post them if there's any interest.

I'd be interested in a bidi check similar to the Stringprep bidi check,
but which avoids allowing two distinct strings that display the same,
while still allowing all pure-ASCII strings.

Such a check could take care of single-component anomalies (like
A-123,456B) but not multiple-component anomalies (like 123.ABC.com).
One way to avoid the latter is to generally avoid components that don't
begin and end with strong directional characters, like 123, 3com,
u2.  But that seems too onerous.  Another idea is to check the whole
identifier, rather than individual components.  For example:

Let strong directional characters be opaque walls: white for LTR (LCat),
black for RTL (RandALCat).  Other characters are transparent voids.  The
beginning and end of the whole identifier are opaque white walls.  If
you can stand inside a void inside a component (not between components)
and see both black and white walls, there is the potential for trouble
(ambiguity or component-splitting or both); otherwise you are safe.  Is
that true?

I'd be extremely hesitant to require (MUST) or even recommend (SHOULD) a
multi-component check, because sometimes there are separate constraints
on separate components and introducing interactions between them would
be a mess.  But it might be good to describe a multi-component check
that people could use as a guideline for creating bidi-safe identifiers.

> I think there should be a warning in IDNA.
>
> But is there any scope for adding such a warning to IDNA in the near
> future, now that the IDN WG has been disbanded?

The IDNA RFCs will be reissued when they advance to Draft Standard,
sometime after multiple interoperable implementations are demonstrated.
We already have lists of editorial clarifications to make at that time.
I doubt there would be any difficulty in adding a helpful warning about
tricky cases.  Adding a new SHOULD recommendation would be a technical
change though; I don't know the policy on that.

AMC