Re: [2.1] Overzealous escaping of high Unicode code points
gelo1234 <[email protected]> Thu, 8 Jun 2017 20:11:03 +0200
| Newsgroups | gmane.text.xml.cocoon.user |
|---|---|
| Message-ID | <CAPJaKUp+nL6jo8XuRvg885Rq5j=n5bufpFdWfjP6w3QUFLj5nQ@mail.gmail.com> |
--001a114be9f89ad978055176c6fa Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable It depends on what type of Serializer you use and what kind of Serlializer config you put into your sitemap? By default XMLSerializer/HTMLSerializer uses UTF-8 encoding. So instead of 1 UTF-16 char you got 2 chars UTF-8 encoded. Of cource there might be also issue with emoji charset, but I would first try to change encoding in Serliazer config (to be UTF-16). Greetings, -Greg 2017-06-07 10:43 GMT+02:00 Flynn, Peter <[email protected]>: > I had a related problem with 3=E2=80=934 CJK characters being converted t= o their > &#hex; format. Very weird, but it turned out to be the old and buggy copy > of jtidy, and I can't figure out how to replace it. > > I haven't had the problem you describe, though, and I have a user who has > implemented emoji in Cocoon, see http://research.ucc.ie/emojis/ > > P > > -- > Peter Flynn | Academic and Collaborative Technologies | IT Services | > University College Cork | Ireland | [email protected] | > http://research.ucc.ie/profiles/H505/pflynn | Sent from Hiri > <https://www.hiri.com/> > > > On 2017-06-06 17:08:51+01:00 Christopher Schultz wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > All, > > I've been testing my application for use with high Unicode code points > such as emoji like =F0=9F=98=8D which is this one:http://www.fileformat.i= nfo/info/unicode/char/1F60D/index.htm > > My application and database can handle this code point, but Cocoon > butchers it in a way that I have seen before -- the way that > commons-lang's StringEscapeUtils.escapeXml/escapeHtml seems to do. > > Instead of letting the character through as-is, it tries to convert it > into these two numbered entities: > > =EF=BF=BD=EF=BF=BD > > Oddly enough, those are the two double-byte UTF-16 characters you'd > get, but they shouldn't be split-up like that, I don't think. > > I haven't found a version of commons-lang 2.x that doesn't break these > kinds of characters. commons-lang3 does the right thing, but they are > incompatible libraries. > > Does anyone know the code well enough to know how difficult it would > be to change the way Cocoon 2.1 escapes its output? For example, by > using commons-lang3? > > I haven't tried Cocoon 2.2, yet, and I can't tell what dependencies it > has. I also can't exactly tell what to do now that I've downloaded the > binary package. Can this just be used as a drop-in replacement for > Cocoon 2.1.x? Cocoon 2.1.x could build a WAR file that I then > customized for my own application, adding various libraries and > configuration files to it. I think I'll follow-up with a separate post > about this. > > - -chris > > -----BEGIN PGP SIGNATURE----- > Comment: GPGTools - http://gpgtools.org > > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJZNtOBAAoJEBzwKT+lPKRYEuIP/3gSJZDNEbzsHkI5zYjMZbFf > vKvRRnBSl+6IdrcUasftf+AkXIIYwj6xnUQ7winsLW/n8TdDG6jPqsg4Khsozc6z > aa23qDly62gmCsqpLohXxt/ZNKdPY4sOTghaaEUFTtTgpeD3M/INF90myT8SwO4K > WUtqVparSqp/Zf9JMm3OCIguMKbsRNYWVIQuiJxDQJkWYwrw0iVk2v8mc6iz/mDF > w6np4EvFr9fqdDufKpPw8anEkrp5JEuTx47vMOtz4sixVr2C6ehgP4zs3kVzdVid > QPeUsrosV1tsRC9bMVLGmjo7UhNseeXCp/AceIT6AQE8Q1clgy9GcoNMf60dgGku > et0xoGptYgbCfmJL+PuA9y7fJYjgTTQheqzuC721n2/sx+kyBSBWSMIhqia2sd4y > spcT4kw+uChsWjwoeGOHOm4IimrVgXkfJeHVSXV4m66sHS9t+bDiiErwS1SikvSV > qF64/L0u8hYFLD1ehURoHBi4foE1Td3eRGOGHgodcYL9C8U+Yv+fWaiYQ5O4CCnW > pToFvVoQOdZY+VVC8hz1ggbRMSxjT2GQLLJ2mjbGzGUJjlwyQaoZnADSSu0efj88 > O2AlWB2Bf/Ag6E4C9jEjj+cauBfR+1NIK7F1Jo6C02yY1SUOSoOAFDZ7EkO4qYAO > YhvgSQXNmKps6rusNjNZ > =3Dq8Eh > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --001a114be9f89ad978055176c6fa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small"><br= ></div><div class=3D"gmail_default" style=3D"font-size:small">It depends on= what type of Serializer you use and what kind of Serlializer config you pu= t into your sitemap?<br><br></div><div class=3D"gmail_default" style=3D"fon= t-size:small">By default XMLSerializer/HTMLSerializer uses UTF-8 encoding. = So instead of 1 UTF-16 char you got 2 chars UTF-8 encoded.<br></div><div cl= ass=3D"gmail_default" style=3D"font-size:small">Of cource there might be al= so issue with emoji charset, but I would first try to change encoding in Se= rliazer config (to be UTF-16).<br></div><div class=3D"gmail_default" style= =3D"font-size:small"><br></div><div class=3D"gmail_default" style=3D"font-s= ize:small">Greetings,<br></div><div class=3D"gmail_default" style=3D"font-s= ize:small">-Greg<br></div></div><div class=3D"gmail_extra"><br><div class= =3D"gmail_quote">2017-06-07 10:43 GMT+02:00 Flynn, Peter <span dir=3D"ltr">= <<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>>= ;</span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;b= order-left:1px #ccc solid;padding-left:1ex"> <div style=3D"font-family:'Segoe UI',Frutiger,'Frutiger Linotyp= e','Dejavu Sans','Helvetica Neue',Arial,sans-serif;font= -size:14px"> <div class=3D"m_-921611568221235900hiri-body-wrapper"> <div>I had a related problem with 3=E2=80=934 CJK characters being converte= d to their &#hex; format. Very weird, but it turned out to be the old a= nd buggy copy of jtidy, and I can't figure out how to replace it.=C2=A0= <br> <br> I haven't had the problem you describe, though, and I have a user who h= as implemented emoji in Cocoon, see <a href=3D"http://research.ucc.ie/emojis/" target=3D"_blank">http://researc= h.ucc.ie/emojis/</a><br> <br> P<br> =C2=A0</div> </div> <div class=3D"m_-921611568221235900hiri-signature-wrapper">--<br> Peter Flynn | Academic and Collaborative Technologies | IT Services | Unive= rsity College Cork | Ireland | <a href=3D"mailto:[email protected]" target=3D"_= blank">[email protected]</a> | <a href=3D"http://research.ucc.ie/profiles/H505/pflynn" target=3D"_blank"> http://research.ucc.ie/<wbr>profiles/H505/pflynn</a>=C2=A0|=C2=A0Sent from = <a href=3D"https://www.hiri.com/" target=3D"_blank"> Hiri</a></div> =C2=A0 <div class=3D"m_-921611568221235900hiri-extra"><span class=3D""> <p>On 2017-06-06 17:08:51+01:00 Christopher Schultz wrote:</p> </span><blockquote style=3D"padding-left:10px;border-left:1px solid #ccc;ma= rgin:0"> <div> <pre class=3D"m_-921611568221235900hiri-plaintext-quote"><span class=3D"">-= ----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 All, I've been testing my application for use with high Unicode code points such as emoji like =F0=9F=98=8D which is this one: <a href=3D"http://www.fileformat.info/info/unicode/char/1F60D/index.htm" ta= rget=3D"_blank">http://www.fileformat.info/<wbr>info/unicode/char/1F60D/ind= ex.<wbr>htm</a> My application and database can handle this code point, but Cocoon butchers it in a way that I have seen before -- the way that commons-lang's StringEscapeUtils.escapeXml/<wbr></span>escapeHtml seems= to do. Instead of letting the character through as-is, it tries to convert it into these two numbered entities: =EF=BF=BD=EF=BF=BD Oddly enough, those are the two double-byte UTF-16 characters you'd get, but they shouldn't be split-up like that, I don't think. I haven't found a version of commons-lang 2.x that doesn't break th= ese kinds of characters. commons-lang3 does the right thing, but they are incompatible libraries. Does anyone know the code well enough to know how difficult it would be to change the way Cocoon 2.1 escapes its output? For example, by using commons-lang3? I haven't tried Cocoon 2.2, yet, and I can't tell what dependencies= it has. I also can't exactly tell what to do now that I've downloaded = the binary package. Can this just be used as a drop-in replacement for Cocoon 2.1.x? Cocoon 2.1.x could build a WAR file that I then customized for my own application, adding various libraries and configuration files to it. I think I'll follow-up with a separate post about this. - -chris -----BEGIN PGP SIGNATURE----- Comment: GPGTools - <a href=3D"http://gpgtools.org" target=3D"_blank">http:= //gpgtools.org</a><div><div class=3D"h5"> Comment: Using GnuPG with Thunderbird - <a href=3D"http://www.enigmail.net/= " target=3D"_blank">http://www.enigmail.net/</a> iQIcBAEBCAAGBQJZNtOBAAoJEBzwKT<wbr>+lPKRYEuIP/<wbr>3gSJZDNEbzsHkI5zYjMZbFf vKvRRnBSl+6IdrcUasftf+<wbr>AkXIIYwj6xnUQ7winsLW/<wbr>n8TdDG6jPqsg4Khsozc6z aa23qDly62gmCsqpLohXxt/<wbr>ZNKdPY4sOTghaaEUFTtTgpeD3M/<wbr>INF90myT8SwO4K WUtqVparSqp/<wbr>Zf9JMm3OCIguMKbsRNYWVIQuiJxDQJ<wbr>kWYwrw0iVk2v8mc6iz/mDF w6np4EvFr9fqdDufKpPw8anEkrp5JE<wbr>uTx47vMOtz4sixVr2C6ehgP4zs3kVz<wbr>dVid QPeUsrosV1tsRC9bMVLGmjo7UhNsee<wbr>XCp/<wbr>AceIT6AQE8Q1clgy9GcoNMf60dgGku et0xoGptYgbCfmJL+<wbr>PuA9y7fJYjgTTQheqzuC721n2/sx+<wbr>kyBSBWSMIhqia2sd4y spcT4kw+<wbr>uChsWjwoeGOHOm4IimrVgXkfJeHVSX<wbr>V4m66sHS9t+bDiiErwS1SikvSV qF64/<wbr>L0u8hYFLD1ehURoHBi4foE1Td3eRGO<wbr>GHgodcYL9C8U+Yv+fWaiYQ5O4CCnW pToFvVoQOdZY+<wbr>VVC8hz1ggbRMSxjT2GQLLJ2mjbGzGU<wbr>JjlwyQaoZnADSSu0efj88 O2AlWB2Bf/Ag6E4C9jEjj+cauBfR+<wbr>1NIK7F1Jo6C02yY1SUOSoOAFDZ7EkO<wbr>4qYAO YhvgSQXNmKps6rusNjNZ =3Dq8Eh -----END PGP SIGNATURE----- ------------------------------<wbr>------------------------------<wbr>-----= ---- To unsubscribe, e-mail: <a href=3D"mailto:[email protected]= rg" target=3D"_blank">users-unsubscribe@cocoon.<wbr>apache.org</a> For additional commands, e-mail: <a href=3D"mailto:[email protected]= .org" target=3D"_blank">[email protected]</a> </div></div></pre> </div> </blockquote> </div> </div> </blockquote></div><br></div> --001a114be9f89ad978055176c6fa--