Re: Converting visual to logical string

On Wed, 10 Aug 2005, Kenneth M Horan wrote:

> So it sounds like treating log2vis like it was really vis2log should give
> me the best results?  I figured there has to be some good way to do this
> since that's pretty much what word processors do, right?  My files were
> typed with a mix of English and Arabic and it's really important that they
> aren't modified during the conversion, if that's even possible.  I guess
> I'll give it a shot and see what happens.  If it's not possible to go both
> ways... what's the point of this silly bidirectional algorithm then,
> anyway?

We need the bidi algorithm exactly because the vvisual string is
hardly useful for processing.  The point is that when you have
something like this: (CAPITAL letters are RTL text)

  he said SHALOM IS hello IN ENGLISH and left.

It's not obvious from an algorithmic point of view whether this
means:

  he said (SHALOM IS (hello) IN ENGLISH) and left.

or:

  he said (SHALOM IS) hello (IN ENGLISH) and left.

and you can compose two logical strings that look the same when
converted to visual.  Because bidirectional text is ambigious
inherently and it's the human mind that decides how to parse it.
But something's true is that without any bidi markup,
log2vis(log2vis(x)) is almost the same as x.  I did some quick
checks:

$ for x in ../test/test_*.input; do cs=${x#*_}; cs=${cs%_*};
echo $x; ./fribidi --charset $cs --nopad < $x | ./fribidi
--charset $cs --nopad | diff - $x | ( test $cs = CapRTL && cat ||
iconv -f $cs -t UTF-8 ) ; done

and it actually looks pretty good.  Try it yourself.  I get
perfect match for simple Persian text with or without diacritics.
Most of the failures is in the Hebrew tests heavily mixed with
English and punctuations. But the more interesting invariant is
whether log2vis(log2vis(log2vis(x))) == log2vis(x).  Doing that I
only get:

../../test/test_CapRTL_explicit.input
3c3
< missing A SAH FDP
---
> PDF missing A SAH
5c5
< a  - TSET.GOOD
---
> a TSET - DOOG.
7,9c7,9
<  987THEtxen  oNE:,123 ndA
<   and the last ONE IS
< best ENO andEHT
---
> 987THEtxen  oNE:,123  ndA
> SI and the last ONE
> best ENO and EHT
12,13c12,13
< !ENO ECIN 456-123 CIBARA NA
< RIAP  456 123-CIBARA NA
---
> !ENO ECIN 456-123  CIBARA NA
> RIAP   123-456 CIBARA NA
../../test/test_CapRTL_implicit.input
18c18
< 1+5 1/5 5-1 5*1 EVLOS
---
> 5+1 5/1 5-1 5*1 EVLOS


which is pretty cool!  So, yes, I think log2vis is currently the
best you can get.  Doing that correctly on lines and paragraphs
may be a bit more work though.  I'll definitely look at that in
more depth next time I'm hacking on FriBidi.


> Any chance there's a Windows executable?  I'm sure that's pushing it but
> it'd sure save me a lot of time.

I don't have any, but Fridrich Strba was compiling it a short
while ago.  Fridrich?

> Ken Horan
>

--behdad
http://behdad.org/