Re: Suboptimal display-reordering in minibuffer

Eli Zaretskii wrote:
> From: "Martin J. Dürst"<[email protected]>
> > Of course, if the characters in the overlay properties before-string and
> > after-string are not currently taken into account when running the bidi
> > algorithm, then that approach may not work very easily.
>
> You are right: they aren't taken into account.  I have yet to code
> support for reordering text in display strings.  To add this feature,
> I will need to solve quite a few problems.  Until I do, I won't know
> whether what you suggest is even doable with a reasonable effort.
>
> I also think that, even if doable, this is a somewhat hackish
> solution.  I think having a special text property that covers the text
> that needs to be reordered is a cleaner solution.
>
I'm second-guessing Martin here (correct me if I'm wrong!) but I suspect
this proposal is based on:

(A) HTML bidi experience, where CSS already supports before/after-string,
    and browsers respect bidi control codes appearing there (?)

(B) a presumption that it's the easiest for you because you planned to
    implement it anyway.

However, since it's not implemented yet (and it's no obvious how to),
I want to point out that having proper bidi in strings and comments
is a much more useful goal (to users) than supporting bidi reordering of
before/after-string properties of overlays, so if there is a better way
to get the former, you should feel free to skip the latter.

>> Date: Wed, 30 Jun 2010 02:04:30 +0300
>> From: Amit Aronovitch<[email protected]>
>> I believe that the required infrastructure has a lot in common with the
>> coloring (font-lock) system.
>>
If I may add some motivation:
In an ideal world, we would have a separate "bidi-lock" engine,
hand-tuned for each syntax in the world, parsing it and endowing
characters with whatever overlays/properties the display engine
needs to show it in perfect order.

But that's out of question because it'd require huge *manpower*
to maintain, and only a fraction of the world's programmers care
about bidi.  A much more realistic (even if less perfect) approach
is to harness existing polished font-lock engine, defining bidi
in terms of it output (e.g. "characters with font-lock-string-face
should be treated as embeddings and not mix with outside text").

[I'm just saying that'd be cool; see below on feasibility...]

There is one complication that's not obvious:

> It's not that simple.  The way bidi reordering is designed and
> implemented in Emacs, the reordering itself happens _before_ faces,
> overlays, and other display features are considered.  The bidi
> reordering engine is totally oblivious to text properties, overlays,
> images, etc.; it just controls which character will be considered next
> for delivering it to the display, and all the rest, i.e. calculation
> of the face of that character, its display metrics, etc. -- all this
> happens _after_ reordering, in code that calls the reordering engine.

So currently there is no way whatsoever by which lisp code (even the
hypothetical perfect bidi-lock) can affect reordering?  (Except for
actually inserting characters into the buffer, which we agree is no-no)

So to get smarter bidi, the bidi engine will have to inspect *some*
kind of metadata on the characters it handles.  Now let's separate
two mostly orthogonal concerns:

1. What bidi metadata exactly do we want?
   Emacs has many kinds of metadata (face info, invisibility,
   local vars/keymap, hooks...) but we'll need new one(s).
   - Virtual control characters?
   - An increment to embedding level?
   - Overridding character's unicode bidi cathegory?
   ...

   It's an interesting discussion, but I don't have much to say now.

2. How will it be attached to the text?
   It seems that Emacs has exactly two ways to attach metadata to text:
   - Text properties
   - Overlays (implemented by pairs of markers?)

   Almost all metadata can be attached by both, although a few (e.g.
   before/after-string) only work on overlays.

   Also, there there are several indirection mechanisms, notably faces:
   a lot of metadata (such as font size) is not directly attached to
   text/overlays but to a face object (which is a property of text/overlay).

Let's concentrate on question 2 here.

From the implementation POV:
How would you like bidi information to be attached so the bidi engine
could easily inspect it?  Is there a shortcut at all, or would *any*
support require changing the architecture to do full-blown resolution
of text properties before bidi?

in light of the above rationale of harnessing font-lock:
AFAIU, font-lock only sets faces as direct text properties (not overlays),
right?  So, it would be convenient to indirect bidi metadata through faces -
although in general it seems a weird place to put it.

--
Beni Cherniavsky-Paskin <[email protected]>