Re: FriBidi and emoji modifiers

--===============0117493119==
Content-Type: multipart/alternative; boundary=001a11421c9aa1b74d052009e102

--001a11421c9aa1b74d052009e102
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thank you for your fast response,

The LRI/PDI seems like the best solution, I've already tried it without
success. I will try again with the patch applied (on the unicode63 branch).

Regards,
Romain Ouabdelkader.

2015-09-17 15:08 GMT+02:00 Dov Grobgeld <[email protected]>:

> It is correct according to the Bidi Algorithm as both the emoji and its
> modifiers are neutral characters, and thus they inherit their direction
> from the context, and in this case become RTL. I can think off of my head
> think about a couple of options of how to resolve this:
>
> 1. Before doing directional rearrange, surround all runs of emoji like
> characters with LRI/PDI (requires non-released fribidi).
> 2. Check the run level of the emoji after the bidi algo and increase the
> run level to the nearest higher even run-level before calling fribidi
> reorder lines.
>
> I'm sure there are other options as well. Probably Behdad will come up
> with a simple solution that I hadn't thought of. :-)
>
> Regards,
> Dov
>
>
>
>
> On Thu, Sep 17, 2015 at 3:54 PM, Romain Ouabdelkader <
> [email protected]> wrote:
>
>> Hi,
>>
>> I'm having some trouble to get emojis to work with FriBidi.
>> Basic emoji works fine, but when using emoji modifiers in a RTL language=
,
>> the modifiers end up before the emoji.
>>
>> Here I have a string with an arabic text, the emoji U+1f476 (a baby) and
>> a tone modifier U+1f3fb:
>>
>>
>> const char utf8_input[] =3D u8"=D8=A7=D8=AE=D8=AA=D8=A8=D8=A7=D8=B1 \U00=
01f476\U0001f3fb";
>> int utf8_len =3D sizeof(utf8_input) - 1;
>>
>> std::unique_ptr<FriBidiChar[]> unicode_str(new FriBidiChar[utf8_len]);
>>
>> FriBidiCharSet utf8_charset =3D fribidi_parse_charset("UTF-8");
>> int len_unicode =3D fribidi_charset_to_unicode(utf8_charset, utf8_input,
>>                                              utf8_len, unicode_str.get()=
);
>>
>> std::unique_ptr<FriBidiCharType[]> bidi_types(new
>> FriBidiCharType[len_unicode]);
>> std::unique_ptr<FriBidiLevel[]> levels(new FriBidiLevel[len_unicode]);
>> FriBidiParType base =3D FRIBIDI_PAR_ON;
>>
>> fribidi_get_bidi_types(unicode_str.get(), len_unicode, bidi_types.get())=
;
>> fribidi_get_par_embedding_levels(bidi_types.get(), len_unicode, &base,
>> levels.get());
>>
>> fribidi_reorder_line(0, bidi_types.get(), len_unicode, 0, base,
>>                      levels.get(), unicode_str.get(), NULL);
>>
>> std::cout << std::hex;
>> for (int i =3D 0; i < len_unicode; ++i)
>>   {
>>     std::cout << "\\u" << unicode_str[i];
>>   }
>> std::cout << std::endl;
>>
>>
>> Output:
>> \u1f3fb\u1f476\u20\u631\u627\u628\u62a\u62e\u627
>>
>> As you can see the tone modifier U+1f3fb is first and then the emoji
>> U+1f476 is next.
>> I've also tested this with log2vis().
>>
>> Is this a bug? If not, what is the correct way to handle emojis?
>>
>> Regards,
>> Romain Ouabdelkader.
>>
>> _______________________________________________
>> fribidi mailing list
>> [email protected]
>> http://lists.freedesktop.org/mailman/listinfo/fribidi
>>
>>
>

--001a11421c9aa1b74d052009e102
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thank you for your fast response,<div><br></div><div>The L=
RI/PDI seems like the best solution, I&#39;ve already tried=C2=A0<span styl=
e=3D"font-family:arial,helvetica,sans-serif;font-size:12.8px">it without su=
ccess.=C2=A0</span><span style=3D"font-family:arial,helvetica,sans-serif;fo=
nt-size:12.8px">I will try again with the patch applied (on the unicode63 b=
ranch).</span></div><div><span style=3D"font-family:arial,helvetica,sans-se=
rif;font-size:12.8px"><br></span></div><div><font face=3D"arial, helvetica,=
 sans-serif"><span style=3D"font-size:12.8px">Regards,</span></font></div><=
div><font face=3D"arial, helvetica, sans-serif"><span style=3D"font-size:12=
.8px">Romain Ouabdelkader.</span></font></div><div class=3D"gmail_extra"><b=
r><div class=3D"gmail_quote">2015-09-17 15:08 GMT+02:00 Dov Grobgeld <span =
dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"_blank"=
>[email protected]</a>&gt;</span>:<br><blockquote class=3D"gmail_quote=
" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><=
div dir=3D"ltr"><div style=3D"font-family:arial,helvetica,sans-serif">It is=
 correct according to the Bidi Algorithm as both the emoji and its modifier=
s are neutral characters, and thus they inherit their direction from the co=
ntext, and in this case become RTL. I can think off of my head think about =
a couple of options of how to resolve this:<br><br></div><div style=3D"font=
-family:arial,helvetica,sans-serif">1. Before doing directional rearrange, =
surround all runs of emoji like characters with LRI/PDI (requires non-relea=
sed fribidi). <br></div><div style=3D"font-family:arial,helvetica,sans-seri=
f">2. Check the run level of the emoji after the bidi algo and increase the=
 run level to the nearest higher even run-level before calling fribidi reor=
der lines.<br><br></div><div style=3D"font-family:arial,helvetica,sans-seri=
f">I&#39;m sure there are other options as well. Probably Behdad will come =
up with a simple solution that I hadn&#39;t thought of. :-)<br><br></div><d=
iv style=3D"font-family:arial,helvetica,sans-serif">Regards,<br></div><div =
style=3D"font-family:arial,helvetica,sans-serif">Dov<br><br></div><div styl=
e=3D"font-family:arial,helvetica,sans-serif"><br><br></div></div><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote"><div><div>On Thu, Sep 17, 2=
015 at 3:54 PM, Romain Ouabdelkader <span dir=3D"ltr">&lt;<a href=3D"mailto=
:[email protected]" target=3D"_blank">romain.ouabdelkader@gmail=
.com</a>&gt;</span> wrote:<br></div></div><blockquote class=3D"gmail_quote"=
 style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><d=
iv><div><div dir=3D"ltr"><div style=3D"font-size:12.8px">Hi,</div><div styl=
e=3D"font-size:12.8px"><br></div><div style=3D"font-size:12.8px">I&#39;m ha=
ving some trouble to get emojis to work with FriBidi.</div><div style=3D"fo=
nt-size:12.8px">Basic emoji works fine, but when using emoji modifiers in a=
 RTL language, the modifiers end up before the emoji.</div><div style=3D"fo=
nt-size:12.8px"><br></div><div style=3D"font-size:12.8px">Here I have a str=
ing with an arabic text, the emoji U+1f476 (a baby) and a tone modifier U+1=
f3fb:</div><div style=3D"font-size:12.8px"><br></div><div style=3D"font-siz=
e:12.8px"><br></div><div style=3D"font-size:12.8px">const char utf8_input[]=
 =3D u8&quot;=D8=A7=D8=AE=D8=AA=D8=A8=D8=A7=D8=B1 \U0001f476\U0001f3fb&quot=
;;</div><div style=3D"font-size:12.8px">int utf8_len =3D sizeof(utf8_input)=
 - 1;</div><div style=3D"font-size:12.8px"><br></div><div style=3D"font-siz=
e:12.8px">std::unique_ptr&lt;FriBidiChar[]&gt; unicode_str(new FriBidiChar[=
utf8_len]);</div><div style=3D"font-size:12.8px"><br></div><div style=3D"fo=
nt-size:12.8px">FriBidiCharSet utf8_charset =3D fribidi_parse_charset(&quot=
;UTF-8&quot;);</div><div style=3D"font-size:12.8px">int len_unicode =3D fri=
bidi_charset_to_unicode(utf8_charset, utf8_input,</div><div style=3D"font-s=
ize:12.8px">=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0utf8_len, unicode_str.get());</div><div style=3D"fo=
nt-size:12.8px"><br></div><div style=3D"font-size:12.8px">std::unique_ptr&l=
t;FriBidiCharType[]&gt; bidi_types(new FriBidiCharType[len_unicode]);</div>=
<div style=3D"font-size:12.8px">std::unique_ptr&lt;FriBidiLevel[]&gt; level=
s(new FriBidiLevel[len_unicode]);</div><div style=3D"font-size:12.8px">FriB=
idiParType base =3D FRIBIDI_PAR_ON;</div><div style=3D"font-size:12.8px"><b=
r></div><div style=3D"font-size:12.8px">fribidi_get_bidi_types(unicode_str.=
get(), len_unicode, bidi_types.get());</div><div style=3D"font-size:12.8px"=
>fribidi_get_par_embedding_levels(bidi_types.get(), len_unicode, &amp;base,=
 levels.get());</div><div style=3D"font-size:12.8px"><br></div><div style=
=3D"font-size:12.8px">fribidi_reorder_line(0, bidi_types.get(), len_unicode=
, 0, base,</div><div style=3D"font-size:12.8px">=C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0levels.get(), unicode_str.=
get(), NULL);</div><div style=3D"font-size:12.8px"><br></div><div style=3D"=
font-size:12.8px">std::cout &lt;&lt; std::hex;</div><div style=3D"font-size=
:12.8px">for (int i =3D 0; i &lt; len_unicode; ++i)</div><div style=3D"font=
-size:12.8px">=C2=A0 {</div><div style=3D"font-size:12.8px">=C2=A0 =C2=A0 s=
td::cout &lt;&lt; &quot;\\u&quot; &lt;&lt; unicode_str[i];</div><div style=
=3D"font-size:12.8px">=C2=A0 }</div><div style=3D"font-size:12.8px">std::co=
ut &lt;&lt; std::endl;</div><div style=3D"font-size:12.8px"><br></div><div =
style=3D"font-size:12.8px"><br></div><div style=3D"font-size:12.8px">Output=
:</div><div style=3D"font-size:12.8px">\u1f3fb\u1f476\u20\u631\u627\u628\u6=
2a\u62e\u627</div><div style=3D"font-size:12.8px"><br></div><div style=3D"f=
ont-size:12.8px">As you can see the tone modifier U+1f3fb is first and then=
 the emoji U+1f476 is next.</div><div style=3D"font-size:12.8px">I&#39;ve a=
lso tested this with log2vis().</div><div style=3D"font-size:12.8px"><br></=
div><div style=3D"font-size:12.8px">Is this a bug? If not, what is the corr=
ect way to handle emojis?</div><div style=3D"font-size:12.8px"><br></div><d=
iv style=3D"font-size:12.8px">Regards,</div><div style=3D"font-size:12.8px"=
>Romain Ouabdelkader.</div></div>
<br></div></div>_______________________________________________<br>
fribidi mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">fribidi@=
lists.freedesktop.org</a><br>
<a href=3D"http://lists.freedesktop.org/mailman/listinfo/fribidi" rel=3D"no=
referrer" target=3D"_blank">http://lists.freedesktop.org/mailman/listinfo/f=
ribidi</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div></div>

--001a11421c9aa1b74d052009e102--

--===============0117493119==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline

X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KZnJpYmlkaSBt
YWlsaW5nIGxpc3QKZnJpYmlkaUBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cDovL2xpc3RzLmZy
ZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ZyaWJpZGkK

--===============0117493119==--