Re: URL internationalization!

On Tue, 11 Mar 1997, Masataka Ohta wrote:

> > > > There are some problems when directly applying
> > > > it to editing the same text with some markup, e.g. HTML or
> > > > LaTeX. For HTML, this has been discussed on the relevant list,
> > > 
> > > Finite-state bi-directionality, whose state can be encoded in
> > > stateless character code, is necessary at the plain text level
> > > and has nothing to do with HTML.
> > 
> > If you have a solution for this, which is more than just
> > "use visual order", I would appreciate a pointer to it.
> 
> What do you mean "visual order"?

If you don't know that, I'm afraid you don't understand much
about bidirectionality, or at least you live in a world with
terminology completely different from what is commonly used.

To shortly explain that, assume that German would be
written right to left and with upper-case characters.
Then the following is logical order from left to right
(sequence as you would read it):

	She said: "GUTEN TAG." and turned away.

The following is visual order:

	She said: ".GAT NETUG" and turned away.

For display, the only thing that we need is visual order
(that's why it is called visual!). For internal representation
(backing store) both systems that use visual order and systems
that use logical order exist. Systems that use logical order
need a more complicated display machinery. Systems that use
visual order are good for processing strictly line-oriented
data. You can easily write a program to change the fifth
from the left in each line to something else. Systems that
use logical order in backing store can handle all kinds of
more content-related processing better.


> The only possible way to statelessly handle bidirectional text is
> to encode directionality state into character codes.
> 
> The only reasonable way to do so is to use a single bit for left/right
> directionality. The result is that English words are spelled backward
> in the environment where line begins from right mergin.
> 
>    Masataka Ohta, "On Plain Text", Proceedings of International
>    Symposium on Multilingual Information Processing, pp 65-67,
>    March 1996.
> 
> should be helpful to you to understand what is the absolute
> requirement.

I am invited to the 1997 issue of this Symposium. I look forward
to meet you and discuss these issues with you. It would be nice
if you can send me (by fax or electronically) a copy of your
paper, so that I can read it beforehand.


> But, it is not a matter, becasue regular expressions spelled backward
> are still regular expressions and all the finite-stateness is preserved.
> 
> Anyway, you can't deny that ISO 10646/UNICODE is stateful.

It is stateful only in a few cases, namely when instead of
only having one citation with different directionality as
in my example above, you have two or more levels of embedding.
Two or more levels of embedding combined with logical order
in the backing store can't be handled without some state.
But they are very rare, and we definitely will rule them
out for URLs.
Everything else is strictly a problem of the rendering process
and confined to within a single line.


> > > What a fair attitude.
> > 
> > I agree that it is not completely fair. And if URLs were
> > just something inside the computer, we could say that it
> > is a user interface issue, have everybody use his/her
> > preferred way of displaying it, and ignore it in this
> > discussion.
> > 
> > However, URLs are also written on paper.
> 
> That's why URLs must be ASCII only.
> 
> > Now it is clear that whatever way to display bidi URLs we
> > finally choose, in their internal representation and their
> > canonical representation (%HH), the sequence of the parts
> > of the generic syntax has to be in logical order if we
> > don't want all the existing software to be rewritten
> > beyond limits.
> 
> Martin, we can't abandon finitestateness, regular expressions, grep,
> sed, awk and all the tools for plain text processing, only because
> high-level tools for mark-up languages allow for PDA grammers.

We don't have to do that. Defining that URLs are stored
in logical order means that all programs that work for example
to do calculations on relative URLs, to extract parts of URLs,
and so on, will work the same way as before. The programs have
to work with all 8 bits, of course. Please note that if we
would use ISO-2022, major changes would be necessary to
avoid that bytes identical to reserved characters are
misinterpreted. Of course, everything also works if we
limit ourselves to "canonical form". The only thing
that will be different is that if the regular expression
for "any character" (usually ".") is intended to be applied
to actual characters and not just bytes, it has to be
written differently, or has to be expanded on a lower
level. But this applies to any multibyte encoding whatsoever.


Regards,	Martin.