Re: [Fresco-devel] Babylon Strings

On Fri, Jul 18, 2003 at 12:52:49PM +0200, Tobias Hunger wrote:
> Unicode specifies that you store the characters in the sequence
> you type them. This is called the memory order. On the other hand
> you might need to map this memory order to a visual order when
> displaying: If you have a hebrew-only text (hebrew is written from
> right to left) all you need to do is render the last character in
> memory order first, then the one before the last, etc. If you have

This is simplifying it a little; vowel markings, for example, are
combining characters that follow the consonant.  Would the visual
order rearrange those, too?

(In bidi books I've seen the convention of using UPPERCASE for letters
that are in a typically RTL language.)
So if you have, in memory order: test KETEB words

Would the visual order be: hello BETEK words?  In that case, those "e"s
would be combining characters that modify the character *following*
them, which is backwards from the way it normally works.  Similarly,
when joining Arabic letters you'd have to join them in reverse; the
alef-lam ligature would have to be used when the "visual" (that is,
left-to-right) order is lam-alef.

> What do you think? Is this what we need at the Babylon level of the
> design? Stefan thinks we only need the memory-oder, but then I
> understood him to argue from a fresco application developer's point
> of view. I agree with him that somebody developing an application
> for Fresco shouldn't need to bother with visual orderings, but I
> want the bidir handling in Babylon (that's where it belongs IMHO)
> and since berlin (our implementation of the fresco display server)
> needs the visual order we need Babylon to export this information
> in some way.

There are many hard problems with bidi that simply having the visual
order won't solve; for example, line-breaking.

Memory order:
  this is a sentence MAKTUB KETEB with words.
Let's say around here      ^ is where we need the line break.
Visual order:
  this is a sentence BETEK BUTKAM with words.
The correct output is:
  this is a sentence BUTKAM
  BETEK with words.

Notice how the words must swap positions, for the word that comes first
(in memory order, which you could also consider "reading" order-- the
order in which they are read) to come before the second word.


Another case study: cut and paste.  Suppose I have this sentence again
and I click and drag the mouse to make a selection from left to right
along the region underlined:
  this is a sentence BETEK BUTKAM with words.
  ^^^^^^^^^^^^^^^^^^^^^^^^^
What I should get in the copy buffer, and what should be selected
visually, is this:
  this is a sentence BETEK BUTKAM with words.
  ^^^^^^^^^^^^^^^^^^       ^^^^^^
(Keep in mind this corresponds to this memory ordering, which is also
the order of the words in a mental sense and is the correct region to
select:)
  this is a sentence MAKTUB KETEB with words.
  ^^^^^^^^^^^^^^^^^^^^^^^^^
Because when I paste that text between the English and the Arabic, I
expect this memory ordering:
  this is a sentence this is a sentence MAKTUB MAKTUB KETEB with words.
which produces this visual order:
  this is a sentence this is a sentence BETEK BUTKAM BUTKAM with words.


To summarize: bidi is really hard to do right.  Most applications
shouldn't need to know the visual order, and the code that will need to
to know it (such as the text layout engine of the text widget) will need
to know more than just the visual order.

(Apologies in advance if I got either the language or the bidi algorithm
wrong; I just read about this recently and I'm going from memory.)

-- 
Evan Martin
[email protected]
http://neugierig.org