FriBidi and emoji modifiers

--===============0415965187==
Content-Type: multipart/alternative; boundary=001a11492bc6ea9978051ff0ea8b

--001a11492bc6ea9978051ff0ea8b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi,

I'm having some trouble to get emojis to work with FriBidi.
Basic emoji works fine, but when using emoji modifiers in a RTL language,
the modifiers end up before the emoji.

Here I have a string with an arabic text, the emoji U+1f476 (a baby) and a
tone modifier U+1f3fb:


const char utf8_input[] =3D u8"=D8=A7=D8=AE=D8=AA=D8=A8=D8=A7=D8=B1 \U0001f=
476\U0001f3fb";
int utf8_len =3D sizeof(utf8_input) - 1;

std::unique_ptr<FriBidiChar[]> unicode_str(new FriBidiChar[utf8_len]);

FriBidiCharSet utf8_charset =3D fribidi_parse_charset("UTF-8");
int len_unicode =3D fribidi_charset_to_unicode(utf8_charset, utf8_input,
                                             utf8_len, unicode_str.get());

std::unique_ptr<FriBidiCharType[]> bidi_types(new
FriBidiCharType[len_unicode]);
std::unique_ptr<FriBidiLevel[]> levels(new FriBidiLevel[len_unicode]);
FriBidiParType base =3D FRIBIDI_PAR_ON;

fribidi_get_bidi_types(unicode_str.get(), len_unicode, bidi_types.get());
fribidi_get_par_embedding_levels(bidi_types.get(), len_unicode, &base,
levels.get());

fribidi_reorder_line(0, bidi_types.get(), len_unicode, 0, base,
                     levels.get(), unicode_str.get(), NULL);

std::cout << std::hex;
for (int i =3D 0; i < len_unicode; ++i)
  {
    std::cout << "\\u" << unicode_str[i];
  }
std::cout << std::endl;


Output:
\u1f3fb\u1f476\u20\u631\u627\u628\u62a\u62e\u627

As you can see the tone modifier U+1f3fb is first and then the emoji
U+1f476 is next.
I've also tested this with log2vis().

Is this a bug? If not, what is the correct way to handle emojis?

Regards,
Romain Ouabdelkader.

--001a11492bc6ea9978051ff0ea8b
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div style=3D"font-size:12.8px">Hi,</div><div style=3D"fon=
t-size:12.8px"><br></div><div style=3D"font-size:12.8px">I&#39;m having som=
e trouble to get emojis to work with FriBidi.</div><div style=3D"font-size:=
12.8px">Basic emoji works fine, but when using emoji modifiers in a RTL lan=
guage, the modifiers end up before the emoji.</div><div style=3D"font-size:=
12.8px"><br></div><div style=3D"font-size:12.8px">Here I have a string with=
 an arabic text, the emoji U+1f476 (a baby) and a tone modifier U+1f3fb:</d=
iv><div style=3D"font-size:12.8px"><br></div><div style=3D"font-size:12.8px=
"><br></div><div style=3D"font-size:12.8px">const char utf8_input[] =3D u8&=
quot;=D8=A7=D8=AE=D8=AA=D8=A8=D8=A7=D8=B1 \U0001f476\U0001f3fb&quot;;</div>=
<div style=3D"font-size:12.8px">int utf8_len =3D sizeof(utf8_input) - 1;</d=
iv><div style=3D"font-size:12.8px"><br></div><div style=3D"font-size:12.8px=
">std::unique_ptr&lt;FriBidiChar[]&gt; unicode_str(new FriBidiChar[utf8_len=
]);</div><div style=3D"font-size:12.8px"><br></div><div style=3D"font-size:=
12.8px">FriBidiCharSet utf8_charset =3D fribidi_parse_charset(&quot;UTF-8&q=
uot;);</div><div style=3D"font-size:12.8px">int len_unicode =3D fribidi_cha=
rset_to_unicode(utf8_charset, utf8_input,</div><div style=3D"font-size:12.8=
px">=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0utf8_len, unicode_str.get());</div><div style=3D"font-size=
:12.8px"><br></div><div style=3D"font-size:12.8px">std::unique_ptr&lt;FriBi=
diCharType[]&gt; bidi_types(new FriBidiCharType[len_unicode]);</div><div st=
yle=3D"font-size:12.8px">std::unique_ptr&lt;FriBidiLevel[]&gt; levels(new F=
riBidiLevel[len_unicode]);</div><div style=3D"font-size:12.8px">FriBidiParT=
ype base =3D FRIBIDI_PAR_ON;</div><div style=3D"font-size:12.8px"><br></div=
><div style=3D"font-size:12.8px">fribidi_get_bidi_types(unicode_str.get(), =
len_unicode, bidi_types.get());</div><div style=3D"font-size:12.8px">fribid=
i_get_par_embedding_levels(bidi_types.get(), len_unicode, &amp;base, levels=
.get());</div><div style=3D"font-size:12.8px"><br></div><div style=3D"font-=
size:12.8px">fribidi_reorder_line(0, bidi_types.get(), len_unicode, 0, base=
,</div><div style=3D"font-size:12.8px">=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0levels.get(), unicode_str.get(), N=
ULL);</div><div style=3D"font-size:12.8px"><br></div><div style=3D"font-siz=
e:12.8px">std::cout &lt;&lt; std::hex;</div><div style=3D"font-size:12.8px"=
>for (int i =3D 0; i &lt; len_unicode; ++i)</div><div style=3D"font-size:12=
.8px">=C2=A0 {</div><div style=3D"font-size:12.8px">=C2=A0 =C2=A0 std::cout=
 &lt;&lt; &quot;\\u&quot; &lt;&lt; unicode_str[i];</div><div style=3D"font-=
size:12.8px">=C2=A0 }</div><div style=3D"font-size:12.8px">std::cout &lt;&l=
t; std::endl;</div><div style=3D"font-size:12.8px"><br></div><div style=3D"=
font-size:12.8px"><br></div><div style=3D"font-size:12.8px">Output:</div><d=
iv style=3D"font-size:12.8px">\u1f3fb\u1f476\u20\u631\u627\u628\u62a\u62e\u=
627</div><div style=3D"font-size:12.8px"><br></div><div style=3D"font-size:=
12.8px">As you can see the tone modifier U+1f3fb is first and then the emoj=
i U+1f476 is next.</div><div style=3D"font-size:12.8px">I&#39;ve also teste=
d this with log2vis().</div><div style=3D"font-size:12.8px"><br></div><div =
style=3D"font-size:12.8px">Is this a bug? If not, what is the correct way t=
o handle emojis?</div><div style=3D"font-size:12.8px"><br></div><div style=
=3D"font-size:12.8px">Regards,</div><div style=3D"font-size:12.8px">Romain =
Ouabdelkader.</div></div>

--001a11492bc6ea9978051ff0ea8b--

--===============0415965187==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline

X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KZnJpYmlkaSBt
YWlsaW5nIGxpc3QKZnJpYmlkaUBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cDovL2xpc3RzLmZy
ZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ZyaWJpZGkK

--===============0415965187==--