Re: FriBidi and emoji modifiers

--===============1176229502==
Content-Type: multipart/alternative; boundary=089e01227d944ab0b6051ff11d22

--089e01227d944ab0b6051ff11d22
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

It is correct according to the Bidi Algorithm as both the emoji and its
modifiers are neutral characters, and thus they inherit their direction
from the context, and in this case become RTL. I can think off of my head
think about a couple of options of how to resolve this:

1. Before doing directional rearrange, surround all runs of emoji like
characters with LRI/PDI (requires non-released fribidi).
2. Check the run level of the emoji after the bidi algo and increase the
run level to the nearest higher even run-level before calling fribidi
reorder lines.

I'm sure there are other options as well. Probably Behdad will come up with
a simple solution that I hadn't thought of. :-)

Regards,
Dov




On Thu, Sep 17, 2015 at 3:54 PM, Romain Ouabdelkader <
[email protected]> wrote:

> Hi,
>
> I'm having some trouble to get emojis to work with FriBidi.
> Basic emoji works fine, but when using emoji modifiers in a RTL language,
> the modifiers end up before the emoji.
>
> Here I have a string with an arabic text, the emoji U+1f476 (a baby) and =
a
> tone modifier U+1f3fb:
>
>
> const char utf8_input[] =3D u8"=D8=A7=D8=AE=D8=AA=D8=A8=D8=A7=D8=B1 \U000=
1f476\U0001f3fb";
> int utf8_len =3D sizeof(utf8_input) - 1;
>
> std::unique_ptr<FriBidiChar[]> unicode_str(new FriBidiChar[utf8_len]);
>
> FriBidiCharSet utf8_charset =3D fribidi_parse_charset("UTF-8");
> int len_unicode =3D fribidi_charset_to_unicode(utf8_charset, utf8_input,
>                                              utf8_len, unicode_str.get())=
;
>
> std::unique_ptr<FriBidiCharType[]> bidi_types(new
> FriBidiCharType[len_unicode]);
> std::unique_ptr<FriBidiLevel[]> levels(new FriBidiLevel[len_unicode]);
> FriBidiParType base =3D FRIBIDI_PAR_ON;
>
> fribidi_get_bidi_types(unicode_str.get(), len_unicode, bidi_types.get());
> fribidi_get_par_embedding_levels(bidi_types.get(), len_unicode, &base,
> levels.get());
>
> fribidi_reorder_line(0, bidi_types.get(), len_unicode, 0, base,
>                      levels.get(), unicode_str.get(), NULL);
>
> std::cout << std::hex;
> for (int i =3D 0; i < len_unicode; ++i)
>   {
>     std::cout << "\\u" << unicode_str[i];
>   }
> std::cout << std::endl;
>
>
> Output:
> \u1f3fb\u1f476\u20\u631\u627\u628\u62a\u62e\u627
>
> As you can see the tone modifier U+1f3fb is first and then the emoji
> U+1f476 is next.
> I've also tested this with log2vis().
>
> Is this a bug? If not, what is the correct way to handle emojis?
>
> Regards,
> Romain Ouabdelkader.
>
> _______________________________________________
> fribidi mailing list
> [email protected]
> http://lists.freedesktop.org/mailman/listinfo/fribidi
>
>

--089e01227d944ab0b6051ff11d22
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:arial,he=
lvetica,sans-serif">It is correct according to the Bidi Algorithm as both t=
he emoji and its modifiers are neutral characters, and thus they inherit th=
eir direction from the context, and in this case become RTL. I can think of=
f of my head think about a couple of options of how to resolve this:<br><br=
></div><div class=3D"gmail_default" style=3D"font-family:arial,helvetica,sa=
ns-serif">1. Before doing directional rearrange, surround all runs of emoji=
 like characters with LRI/PDI (requires non-released fribidi). <br></div><d=
iv class=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-serif"=
>2. Check the run level of the emoji after the bidi algo and increase the r=
un level to the nearest higher even run-level before calling fribidi reorde=
r lines.<br><br></div><div class=3D"gmail_default" style=3D"font-family:ari=
al,helvetica,sans-serif">I&#39;m sure there are other options as well. Prob=
ably Behdad will come up with a simple solution that I hadn&#39;t thought o=
f. :-)<br><br></div><div class=3D"gmail_default" style=3D"font-family:arial=
,helvetica,sans-serif">Regards,<br></div><div class=3D"gmail_default" style=
=3D"font-family:arial,helvetica,sans-serif">Dov<br><br></div><div class=3D"=
gmail_default" style=3D"font-family:arial,helvetica,sans-serif"><br><br></d=
iv></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Thu, =
Sep 17, 2015 at 3:54 PM, Romain Ouabdelkader <span dir=3D"ltr">&lt;<a href=
=3D"mailto:[email protected]" target=3D"_blank">romain.ouabdelk=
[email protected]</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" s=
tyle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div=
 dir=3D"ltr"><div style=3D"font-size:12.8px">Hi,</div><div style=3D"font-si=
ze:12.8px"><br></div><div style=3D"font-size:12.8px">I&#39;m having some tr=
ouble to get emojis to work with FriBidi.</div><div style=3D"font-size:12.8=
px">Basic emoji works fine, but when using emoji modifiers in a RTL languag=
e, the modifiers end up before the emoji.</div><div style=3D"font-size:12.8=
px"><br></div><div style=3D"font-size:12.8px">Here I have a string with an =
arabic text, the emoji U+1f476 (a baby) and a tone modifier U+1f3fb:</div><=
div style=3D"font-size:12.8px"><br></div><div style=3D"font-size:12.8px"><b=
r></div><div style=3D"font-size:12.8px">const char utf8_input[] =3D u8&quot=
;=D8=A7=D8=AE=D8=AA=D8=A8=D8=A7=D8=B1 \U0001f476\U0001f3fb&quot;;</div><div=
 style=3D"font-size:12.8px">int utf8_len =3D sizeof(utf8_input) - 1;</div><=
div style=3D"font-size:12.8px"><br></div><div style=3D"font-size:12.8px">st=
d::unique_ptr&lt;FriBidiChar[]&gt; unicode_str(new FriBidiChar[utf8_len]);<=
/div><div style=3D"font-size:12.8px"><br></div><div style=3D"font-size:12.8=
px">FriBidiCharSet utf8_charset =3D fribidi_parse_charset(&quot;UTF-8&quot;=
);</div><div style=3D"font-size:12.8px">int len_unicode =3D fribidi_charset=
_to_unicode(utf8_charset, utf8_input,</div><div style=3D"font-size:12.8px">=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0utf8_len, unicode_str.get());</div><div style=3D"font-size:12.=
8px"><br></div><div style=3D"font-size:12.8px">std::unique_ptr&lt;FriBidiCh=
arType[]&gt; bidi_types(new FriBidiCharType[len_unicode]);</div><div style=
=3D"font-size:12.8px">std::unique_ptr&lt;FriBidiLevel[]&gt; levels(new FriB=
idiLevel[len_unicode]);</div><div style=3D"font-size:12.8px">FriBidiParType=
 base =3D FRIBIDI_PAR_ON;</div><div style=3D"font-size:12.8px"><br></div><d=
iv style=3D"font-size:12.8px">fribidi_get_bidi_types(unicode_str.get(), len=
_unicode, bidi_types.get());</div><div style=3D"font-size:12.8px">fribidi_g=
et_par_embedding_levels(bidi_types.get(), len_unicode, &amp;base, levels.ge=
t());</div><div style=3D"font-size:12.8px"><br></div><div style=3D"font-siz=
e:12.8px">fribidi_reorder_line(0, bidi_types.get(), len_unicode, 0, base,</=
div><div style=3D"font-size:12.8px">=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0levels.get(), unicode_str.get(), NULL=
);</div><div style=3D"font-size:12.8px"><br></div><div style=3D"font-size:1=
2.8px">std::cout &lt;&lt; std::hex;</div><div style=3D"font-size:12.8px">fo=
r (int i =3D 0; i &lt; len_unicode; ++i)</div><div style=3D"font-size:12.8p=
x">=C2=A0 {</div><div style=3D"font-size:12.8px">=C2=A0 =C2=A0 std::cout &l=
t;&lt; &quot;\\u&quot; &lt;&lt; unicode_str[i];</div><div style=3D"font-siz=
e:12.8px">=C2=A0 }</div><div style=3D"font-size:12.8px">std::cout &lt;&lt; =
std::endl;</div><div style=3D"font-size:12.8px"><br></div><div style=3D"fon=
t-size:12.8px"><br></div><div style=3D"font-size:12.8px">Output:</div><div =
style=3D"font-size:12.8px">\u1f3fb\u1f476\u20\u631\u627\u628\u62a\u62e\u627=
</div><div style=3D"font-size:12.8px"><br></div><div style=3D"font-size:12.=
8px">As you can see the tone modifier U+1f3fb is first and then the emoji U=
+1f476 is next.</div><div style=3D"font-size:12.8px">I&#39;ve also tested t=
his with log2vis().</div><div style=3D"font-size:12.8px"><br></div><div sty=
le=3D"font-size:12.8px">Is this a bug? If not, what is the correct way to h=
andle emojis?</div><div style=3D"font-size:12.8px"><br></div><div style=3D"=
font-size:12.8px">Regards,</div><div style=3D"font-size:12.8px">Romain Ouab=
delkader.</div></div>
<br>_______________________________________________<br>
fribidi mailing list<br>
<a href=3D"mailto:[email protected]">[email protected].=
org</a><br>
<a href=3D"http://lists.freedesktop.org/mailman/listinfo/fribidi" rel=3D"no=
referrer" target=3D"_blank">http://lists.freedesktop.org/mailman/listinfo/f=
ribidi</a><br>
<br></blockquote></div><br></div>

--089e01227d944ab0b6051ff11d22--

--===============1176229502==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline

X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KZnJpYmlkaSBt
YWlsaW5nIGxpc3QKZnJpYmlkaUBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cDovL2xpc3RzLmZy
ZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ZyaWJpZGkK

--===============1176229502==--