Re: URL internationalization!

Martin;

> > The paper assumes that the readers have some knowledge on the
> > difference between finite state automaton and push down automaton.
> 
> Yet it doesn't give a single reference on these topics!
> Nor does it mention such an assumption.

Of course. It's not a textbook, but a research paper.

> > > Interestingly enough,
> > > things such as Boyer-Moore work extremely well with UTF-8,
> > > but fail quickly for iso 2022.
> > 
> > "Fail quickly"? What do you mean?
> 
> I don't mean the theoretical difficulties, but the practical
> difficulties of setting up an expression for example to
> capture a ü (in HTML notation) in a general ISO 2022
> text (with bells and whistles such as designations and
> invocations, and several coded character sets that contain
> that character).

Handling of designations and invocations is a lot more easy 
than that of combining characters.

You can't search 'A with umlaut' easily.

> > Boyer-Moore is just determinization of finite-state automaton
> > and works perfectly fine over any text with finite-state
> > structure, including ISO 2022 based ones.
> > 
> > Boyer-Moore does not work at all with structured text with
> > nesting.
> 
> If you want to stay on the theoretical side, then please
> note that the BIDI nesting structure in Unicode is limited
> to 16 levels.

Thus, Unicode is completely stateful.

And, 16 or any finite number is NOT enough, of course.

Moreover, just a single level of nesting is already too bad,
requiring a string reversing operation for output, which is
NOT finite state.

> It is easy to construct an FSA, and possible
> (although tedious) to construct a regular expression, for
> limited levels of nesting. It is only for unlimited levels
> of nesting that you need a push down automaton.

Theoretically, yes. But, it is one thing that you can construct
a regular expression and it is another thing that you can type in
the regular expression. You can't.

> In addition, as long as you are searching logically
> (which is what a user usually does), you can completely
> ignore the BIDI nesting structure.

No. You just can't input logically ordered regular expression.

> Also, it is similar to the situation where you search
> for e certain variable name (or whatever) in a C program.
> C can have a lot of nesting, but it is irrelevant for
> all kinds of practical searching.

C is not bidirectional.

> So on average, a regular expression to find a Hebrew
> or Arabic word or character will be shorter for UTF-8
> than for ISO 2022.

If you extract full capability of ISO 2022 to support fully nested
BIDI, yes.

If you carefully profile ISO 2022, no. It's a lot simpler and shorter
than Unicode's way of supporting BIDI.

> > > > Wrong. Conversion between logical and visual order needs
> > > > push down automaton.
> > > 
> > > That is fine.
> > 
> > No.
> 
> Why not? What's so bad about it when we have to do a lot
> of other processing for Arabic and Indic scripts anyway?

When you can't serach, there is no other meaningful processing.

> And if we really stay with it, as long as there is a bound
> on the line length (such as 80 or 132 positions),

That a line has 80 positions for fiexed-width ASCII characters means
nothing for characters such as 'A with Umlaut' with combining
characters, zero width characters and Arabic characters.

> the conversion
> from logical to visual order could indeed be processed by
> a finite series of replacing regular expressions,

If you limit the line length below 80 bytes, you only need 2^640
(just a finite number) regular expressions to do any operation on the line.

But, so what?

> even
> including what you call styled text, i.e. text that
> doesn't fit on a line and where linebreaks have to be
> found.

Wrong. Interline processing needs a lot more complex operations.

> And here is another interesting tidbit from the recent workshop
> in Tsukuba: Mark Crispin told us about an extension to the IMAP
> protocol to do search in messages. When asked who this would be
> implemented, he told us about one (at the moment the only)
> implementation, which converted everything to UTF-8. Obviously,
> with real implementations, practical facts count more than
> theoretical ramblings.

There is no point to have real implementations on broken
specifications.

> > > For UTF-8, there is no need to do any external/internal
> > > conversion for comparison, you can use a lot of your
> > > favorit UNIX tools on it directly.
> > 
> > Martin, you are biting yourself.
> 
> What do you mean? You stated how important it is to use
> these tools. I explained that with UTF-8, it is actually
> easier than with ISO 2022. The home of most of the ideas
> on UNIX tools, the Bell Labs, is actually also the home
> of UTF-8.

Plan9 of Bell Labs is an abandoned variant of UNIX.

> If they wouldn't have been able to use the tools
> they cared and liked, they wouldn't have introduced UTF-8.

Surely, they shouldn't.

That UTF-8 works perfectly well on text containing only Latin1
characters does not mean UTF-8 is good.

> > > in your case. And rather blatant inconsistencies, such as
> > > on the one hand saying that grep is very important, and on

> Sorry for generalizing a little bit with "grep is very important".
> You never generalize, don't you :-)?

I never generalize someone else's opinion to say it wrong,
of course.

							Masataka Ohta