Re: Proposal: Introduction of Neutral Override Character

On Thu, 22 Jul 2004, Dov Grobgeld wrote:

> Hi,
>
> After reading Behdad's excellent summary of the current state of
> BiDi in Gtk, I realized that the problem that Behdad describes
> on Slide 11 could easily be solved by the introduction of a new

Seems like I forgot to CC the fribidi list.  Here are the slides,
focusing the slide Dov is referring to:

http://behdad.org/download/Presentations/bidi-layouts/#12

> unicode control character "Neutral Override". This new unicode
> character should have similar functionality to:
>
>    202D;LEFT-TO-RIGHT OVERRIDE;Cf;0;LRO;;;;;N;;;;;
>    202E;RIGHT-TO-LEFT OVERRIDE;Cf;0;RLO;;;;;N;;;;;
>
> And I suggest that for the time being we give the new character
> the definition:
>
>    20EB;NEUTRAL OVERRIDE;Cf;0;NO;;;;;N;;;;;
>
> The effect of the character is terminated by PDF. The effect
> of the character should be different than LRO and RLO though,
> as it will not change the properties of the characters within
> its range, but will only prevent the use of the range for
> the purpose of seeking a base direction.

Interesting proposal.  I suggest that it adds 2 to the embedding
level of the characters within, pushing them upper such that they
are *isolated* from the surrounding characters a bit too.  I
think this proposed character of yours is quite related to what I
was proposing to Asmus Freytag last month in Unicode Technical
Committee meeting in Toronto.  Mine is not an Override character,
but an Embedding.  So I like to push your character to 20EC and
put mine in 20EB, resulting in:

   20EB;NEUTRAL EMBEDDING;Cf;0;NE;;;;;N;;;;;
   20EC;NEUTRAL OVERRIDE;Cf;0;NO;;;;;N;;;;;

Now what my character tries to do is th embed sub-paragraphs in
your paragraph.  The idea is that instead of writing:

  You said: %s\n

you write:

  You said: <NE>%s<PDF>\n

and the text in the sub-paragraph, will get it's own paragraph
direction, falling back o the higher level direction if all
inside is neutral characters.  The details can be written down
easily.

But the problem with what you suggest is that...

> If we implement this addition in pango and fribidi, we get an
> acceptable solution for Gtk2. All that is needed then in order
> to solve the layout problem of slide 11 is to wrap the name in
> a NO/PDF pair as follows :
>
>    <NO>Behdad<PDF>:

Many engines, including Pango and Gtk+, find the paragraph
direction themselves, by looking up the first letter in the
paragraph themselves, they just don't leave it to FriBidi.  And
with your suggestion paragraph direction detection becomes quite
a more complex project.  This is not necessarily bad, it just
means that Pango needs to call my new
fribidi_get_paragraph_direction call that I've added to FriBidi.

> If we accept this solution we should also send in a proposal
> to the Unicode consortium to make this a formal addition to
> the standard.

Sure, but we need to find real use cases for these.  If we just
send them our current situation the response will be that it
belongs to markup, not plain text, which is in fact right.

> The actual implementation would take Behdad ~5min, and me
> ~1h. ;-)

So, lets continue the discussion, we can allocate a couple slots
in Private Use Area and test with it.

> Regards,
> Dov

--behdad
  behdad.org