Re: URL internationalization!

On Tue, 18 Mar 1997, Masataka Ohta wrote:

> Martin;
> 

> > > What do you mean "visual order"?
> > 
> > If you don't know that, I'm afraid you don't understand much
> > about bidirectionality, or at least you live in a world with
> > terminology completely different from what is commonly used.
> 
> I have no time to be bothered by those terminologies created
> by someone who does not understand bidirectinality.

I know a lot of people in Israel and in countries using
Arabic use these terms, and I have good reasons to assume
that they know more about bidirectionality than you do.
And interestingly enough, you use it yourself below.


> > To shortly explain that, assume that German would be
> > written right to left and with upper-case characters.
> > Then the following is logical order from left to right
> > (sequence as you would read it):
> > 
> > 	She said: "GUTEN TAG." and turned away.
> > 
> > The following is visual order:
> > 
> > 	She said: ".GAT NETUG" and turned away.
> 
> No. Unless you know the line length of the display, it is
> important to distinguish
> 
> 	".GAT NETUG"
> 
> and
> 
> 							.GAT NETUG

If you have a list of file names/URLs, and some of them contain
RTL characters, an average user would probably prefer to have them
all aligned on the same side of the screen.

At the Multilingual symposium in Tsukuba last week, to which I
was invited, the following question came up during a demo in one
of my talks: If you have the sentence

	She said: ".GAT NETUG" and turned away.

on your screen, put the cursor between ".GAT" and "NETUG", and
hit return, how should the result look? shuld it look like

	She said: ".GAT
	NETUG" and turned away.

or should it look like

	She said: "NETUG
	.GAT" and turned away.

I showed that my system (based on the distinction between
visual and logical ordering) did the later. People from
Alis and Apple (both with longstanding experience in the
Middle East markets) confirmed that this was what their
systems did. If they didn't do the right thing, they would
not be in that market anymore. Ken'ichi Handa, the leader
of the mule (multilingual emacs) team, who had asked the
question, agreed to implement it that way if that was
what the users needed.

I wonder what your understanding of bidirectionality
would have produced.


> > to meet you and discuss these issues with you. It would be nice
> > if you can send me (by fax or electronically) a copy of your
> > paper, so that I can read it beforehand.
> 
> I will.

Got it. Also got a paper copy of it, in the proceedings of
last year's workshop. Terse and criptic writing, as always
in your case. And rather blatant inconsistencies, such as
on the one hand saying that grep is very important, and on
the other hand proposing iso 2022. Interestingly enough,
things such as Boyer-Moore work extremely well with UTF-8,
but fail quickly for iso 2022.


> > > > Now it is clear that whatever way to display bidi URLs we
> > > > finally choose, in their internal representation and their
> > > > canonical representation (%HH), the sequence of the parts
> > > > of the generic syntax has to be in logical order if we
> > > > don't want all the existing software to be rewritten
> > > > beyond limits.
> > > 
> > > Martin, we can't abandon finitestateness, regular expressions, grep,
> > > sed, awk and all the tools for plain text processing, only because
> > > high-level tools for mark-up languages allow for PDA grammers.
> > 
> > We don't have to do that. Defining that URLs are stored
> > in logical order means that all programs that work for example
> > to do calculations on relative URLs, to extract parts of URLs,
> > and so on, will work the same way as before.
> 
> Wrong. Conversion between logical and visual order needs
> push down automaton.

That is fine. It only affects one step, namely the display of
URLs. And it actually helps all the rest, because if we can
use logical order internally, things such as searching, relative
URL processing, sorting, and so on, will work nicely without any
changes. If you use sorting on visual order, it will be wrong.


> > The programs have
> > to work with all 8 bits, of course. Please note that if we
> > would use ISO-2022, major changes would be necessary to
> > avoid that bytes identical to reserved characters are
> > misinterpreted.
> 
> Huh? You are confusing the internal and external representations.

Are you assuming that all URL-handling programs will be
rewritten to do conversion from ISO-2022 external representation
to whatever suitable internal representation? Have you hopes
that somebody will volunteer for this work? Or do you want to
do all of it by yourself? You would better start very soon
if you want to complete it before the end of the NEXT century.
And I guess it's fair to call this major changes.

For UTF-8, there is no need to do any external/internal
conversion for comparison, you can use a lot of your
favorit UNIX tools on it directly.


> Anyway, URL can't be expanded beyond some character set with
> soft hyphens or non-breaking spaces.

I don't know exactly what you want to say with this, but if
you want to say that the use of SHY and NBSP in URLs is
dangerous, I agree with you. It is about as dangerous as
using 0O0O0O00O.html.


Regards,	Martin.