Re: [2.1] Overzealous escaping of high Unicode code points
gelo1234 <[email protected]> Thu, 8 Jun 2017 20:17:11 +0200
| Newsgroups | gmane.text.xml.cocoon.user |
|---|---|
| Message-ID | <CAPJaKUqCybLQjs_dtrnrKprE_-9F9cWjhr0Q_G38_g4VD=ozjg@mail.gmail.com> |
--001a11432ade8c612a055176dcd0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Chris,
Even with C3 (cocoon 3.0 beta) unless you specify optional encoding in your
Serializer config, you fallback to default UTF-8:
org.apache.cocoon.optional.servlet.components.sax.serializers.util
public class ConfigurationUtils {
private ConfigurationUtils() {
}
public static String getEncoding(Map<String, ? extends Object>
configuration) {
String encoding =3D (String) configuration.get("encoding");
if (encoding =3D=3D null || "".equals(encoding)) {
encoding =3D "UTF-8";
}
return encoding;
}
...
Greetings,
Greg
2017-06-08 20:11 GMT+02:00 gelo1234 <[email protected]>:
>
> It depends on what type of Serializer you use and what kind of Serlialize=
r
> config you put into your sitemap?
>
> By default XMLSerializer/HTMLSerializer uses UTF-8 encoding. So instead o=
f
> 1 UTF-16 char you got 2 chars UTF-8 encoded.
> Of cource there might be also issue with emoji charset, but I would first
> try to change encoding in Serliazer config (to be UTF-16).
>
> Greetings,
> -Greg
>
> 2017-06-07 10:43 GMT+02:00 Flynn, Peter <[email protected]>:
>
>> I had a related problem with 3=E2=80=934 CJK characters being converted =
to their
>> &#hex; format. Very weird, but it turned out to be the old and buggy cop=
y
>> of jtidy, and I can't figure out how to replace it.
>>
>> I haven't had the problem you describe, though, and I have a user who ha=
s
>> implemented emoji in Cocoon, see http://research.ucc.ie/emojis/
>>
>> P
>>
>> --
>> Peter Flynn | Academic and Collaborative Technologies | IT Services |
>> University College Cork | Ireland | [email protected] |
>> http://research.ucc.ie/profiles/H505/pflynn | Sent from Hiri
>> <https://www.hiri.com/>
>>
>>
>> On 2017-06-06 17:08:51+01:00 Christopher Schultz wrote:
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> All,
>>
>> I've been testing my application for use with high Unicode code points
>> such as emoji like =F0=9F=98=8D which is this one:http://www.fileformat.=
info/info/unicode/char/1F60D/index.htm
>>
>> My application and database can handle this code point, but Cocoon
>> butchers it in a way that I have seen before -- the way that
>> commons-lang's StringEscapeUtils.escapeXml/escapeHtml seems to do.
>>
>> Instead of letting the character through as-is, it tries to convert it
>> into these two numbered entities:
>>
>> =EF=BF=BD=EF=BF=BD
>>
>> Oddly enough, those are the two double-byte UTF-16 characters you'd
>> get, but they shouldn't be split-up like that, I don't think.
>>
>> I haven't found a version of commons-lang 2.x that doesn't break these
>> kinds of characters. commons-lang3 does the right thing, but they are
>> incompatible libraries.
>>
>> Does anyone know the code well enough to know how difficult it would
>> be to change the way Cocoon 2.1 escapes its output? For example, by
>> using commons-lang3?
>>
>> I haven't tried Cocoon 2.2, yet, and I can't tell what dependencies it
>> has. I also can't exactly tell what to do now that I've downloaded the
>> binary package. Can this just be used as a drop-in replacement for
>> Cocoon 2.1.x? Cocoon 2.1.x could build a WAR file that I then
>> customized for my own application, adding various libraries and
>> configuration files to it. I think I'll follow-up with a separate post
>> about this.
>>
>> - -chris
>>
>> -----BEGIN PGP SIGNATURE-----
>> Comment: GPGTools - http://gpgtools.org
>>
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>> iQIcBAEBCAAGBQJZNtOBAAoJEBzwKT+lPKRYEuIP/3gSJZDNEbzsHkI5zYjMZbFf
>> vKvRRnBSl+6IdrcUasftf+AkXIIYwj6xnUQ7winsLW/n8TdDG6jPqsg4Khsozc6z
>> aa23qDly62gmCsqpLohXxt/ZNKdPY4sOTghaaEUFTtTgpeD3M/INF90myT8SwO4K
>> WUtqVparSqp/Zf9JMm3OCIguMKbsRNYWVIQuiJxDQJkWYwrw0iVk2v8mc6iz/mDF
>> w6np4EvFr9fqdDufKpPw8anEkrp5JEuTx47vMOtz4sixVr2C6ehgP4zs3kVzdVid
>> QPeUsrosV1tsRC9bMVLGmjo7UhNseeXCp/AceIT6AQE8Q1clgy9GcoNMf60dgGku
>> et0xoGptYgbCfmJL+PuA9y7fJYjgTTQheqzuC721n2/sx+kyBSBWSMIhqia2sd4y
>> spcT4kw+uChsWjwoeGOHOm4IimrVgXkfJeHVSXV4m66sHS9t+bDiiErwS1SikvSV
>> qF64/L0u8hYFLD1ehURoHBi4foE1Td3eRGOGHgodcYL9C8U+Yv+fWaiYQ5O4CCnW
>> pToFvVoQOdZY+VVC8hz1ggbRMSxjT2GQLLJ2mjbGzGUJjlwyQaoZnADSSu0efj88
>> O2AlWB2Bf/Ag6E4C9jEjj+cauBfR+1NIK7F1Jo6C02yY1SUOSoOAFDZ7EkO4qYAO
>> YhvgSQXNmKps6rusNjNZ
>> =3Dq8Eh
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>>
>>
>
--001a11432ade8c612a055176dcd0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small"><br=
></div><div class=3D"gmail_default" style=3D"font-size:small">Chris, <br><b=
r>Even with C3 (cocoon 3.0 beta) unless you specify optional encoding in yo=
ur Serializer config, you fallback to default UTF-8:<br><br>org.apache.coco=
on.optional.servlet.components.sax.serializers.util<br><br>public class Con=
figurationUtils {<br><br>=C2=A0=C2=A0=C2=A0 private ConfigurationUtils() {<=
br>=C2=A0=C2=A0=C2=A0 }<br><br>=C2=A0=C2=A0=C2=A0 public static String getE=
ncoding(Map<String, ? extends Object> configuration) {<br>=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 String encoding =3D (String) configuratio=
n.get("encoding");<br><br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 if (encoding =3D=3D null || "".equals(encoding)) {<br>=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 encoding =3D &=
quot;UTF-8";<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }<br><br>=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return encoding;<br>=C2=A0=C2=A0=
=C2=A0 }<br>...<br><br></div><div class=3D"gmail_default" style=3D"font-siz=
e:small">Greetings,<br></div><div class=3D"gmail_default" style=3D"font-siz=
e:small">Greg<br><br></div><div class=3D"gmail_extra"><br><div class=3D"gma=
il_quote">2017-06-08 20:11 GMT+02:00 gelo1234 <span dir=3D"ltr"><<a href=
=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>>=
</span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo=
rder-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div style=3D"f=
ont-size:small"><br></div><div style=3D"font-size:small">It depends on what=
type of Serializer you use and what kind of Serlializer config you put int=
o your sitemap?<br><br></div><div style=3D"font-size:small">By default XMLS=
erializer/HTMLSerializer uses UTF-8 encoding. So instead of 1 UTF-16 char y=
ou got 2 chars UTF-8 encoded.<br></div><div style=3D"font-size:small">Of co=
urce there might be also issue with emoji charset, but I would first try to=
change encoding in Serliazer config (to be UTF-16).<br></div><div style=3D=
"font-size:small"><br></div><div style=3D"font-size:small">Greetings,<br></=
div><div style=3D"font-size:small">-Greg<br></div></div><div class=3D"HOEnZ=
b"><div class=3D"h5"><div class=3D"gmail_extra"><br><div class=3D"gmail_quo=
te">2017-06-07 10:43 GMT+02:00 Flynn, Peter <span dir=3D"ltr"><<a href=
=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>></span>:<b=
r><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:=
1px #ccc solid;padding-left:1ex">
<div style=3D"font-family:'Segoe UI',Frutiger,'Frutiger Linotyp=
e','Dejavu Sans','Helvetica Neue',Arial,sans-serif;font=
-size:14px">
<div class=3D"m_4165950864530771173m_-921611568221235900hiri-body-wrapper">
<div>I had a related problem with 3=E2=80=934 CJK characters being converte=
d to their &#hex; format. Very weird, but it turned out to be the old a=
nd buggy copy of jtidy, and I can't figure out how to replace it.=C2=A0=
<br>
<br>
I haven't had the problem you describe, though, and I have a user who h=
as implemented emoji in Cocoon, see
<a href=3D"http://research.ucc.ie/emojis/" target=3D"_blank">http://researc=
h.ucc.ie/emojis/</a><br>
<br>
P<br>
=C2=A0</div>
</div>
<div class=3D"m_4165950864530771173m_-921611568221235900hiri-signature-wrap=
per">--<br>
Peter Flynn | Academic and Collaborative Technologies | IT Services | Unive=
rsity College Cork | Ireland | <a href=3D"mailto:[email protected]" target=3D"_=
blank">[email protected]</a> |
<a href=3D"http://research.ucc.ie/profiles/H505/pflynn" target=3D"_blank">
http://research.ucc.ie/profile<wbr>s/H505/pflynn</a>=C2=A0|=C2=A0Sent from =
<a href=3D"https://www.hiri.com/" target=3D"_blank">
Hiri</a></div>
=C2=A0
<div class=3D"m_4165950864530771173m_-921611568221235900hiri-extra"><span>
<p>On 2017-06-06 17:08:51+01:00 Christopher Schultz wrote:</p>
</span><blockquote style=3D"padding-left:10px;border-left:1px solid #ccc;ma=
rgin:0">
<div>
<pre class=3D"m_4165950864530771173m_-921611568221235900hiri-plaintext-quot=
e"><span>-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
All,
I've been testing my application for use with high Unicode code points
such as emoji like =F0=9F=98=8D which is this one:
<a href=3D"http://www.fileformat.info/info/unicode/char/1F60D/index.htm" ta=
rget=3D"_blank">http://www.fileformat.info/inf<wbr>o/unicode/char/1F60D/ind=
ex.htm</a>
My application and database can handle this code point, but Cocoon
butchers it in a way that I have seen before -- the way that
commons-lang's StringEscapeUtils.escapeXml/</span>es<wbr>capeHtml seems=
to do.
Instead of letting the character through as-is, it tries to convert it
into these two numbered entities:
=EF=BF=BD=EF=BF=BD
Oddly enough, those are the two double-byte UTF-16 characters you'd
get, but they shouldn't be split-up like that, I don't think.
I haven't found a version of commons-lang 2.x that doesn't break th=
ese
kinds of characters. commons-lang3 does the right thing, but they are
incompatible libraries.
Does anyone know the code well enough to know how difficult it would
be to change the way Cocoon 2.1 escapes its output? For example, by
using commons-lang3?
I haven't tried Cocoon 2.2, yet, and I can't tell what dependencies=
it
has. I also can't exactly tell what to do now that I've downloaded =
the
binary package. Can this just be used as a drop-in replacement for
Cocoon 2.1.x? Cocoon 2.1.x could build a WAR file that I then
customized for my own application, adding various libraries and
configuration files to it. I think I'll follow-up with a separate post
about this.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - <a href=3D"http://gpgtools.org" target=3D"_blank">http:=
//gpgtools.org</a><div><div class=3D"m_4165950864530771173h5">
Comment: Using GnuPG with Thunderbird - <a href=3D"http://www.enigmail.net/=
" target=3D"_blank">http://www.enigmail.net/</a>
iQIcBAEBCAAGBQJZNtOBAAoJEBzwKT<wbr>+lPKRYEuIP/3gSJZDNEbzsHkI5zYjM<wbr>ZbFf
vKvRRnBSl+6IdrcUasftf+AkXIIYwj<wbr>6xnUQ7winsLW/n8TdDG6jPqsg4Khso<wbr>zc6z
aa23qDly62gmCsqpLohXxt/ZNKdPY4<wbr>sOTghaaEUFTtTgpeD3M/INF90myT8S<wbr>wO4K
WUtqVparSqp/Zf9JMm3OCIguMKbsRN<wbr>YWVIQuiJxDQJkWYwrw0iVk2v8mc6iz<wbr>/mDF
w6np4EvFr9fqdDufKpPw8anEkrp5JE<wbr>uTx47vMOtz4sixVr2C6ehgP4zs3kVz<wbr>dVid
QPeUsrosV1tsRC9bMVLGmjo7UhNsee<wbr>XCp/AceIT6AQE8Q1clgy9GcoNMf60d<wbr>gGku
et0xoGptYgbCfmJL+PuA9y7fJYjgTT<wbr>QheqzuC721n2/sx+kyBSBWSMIhqia2<wbr>sd4y
spcT4kw+uChsWjwoeGOHOm4IimrVgX<wbr>kfJeHVSXV4m66sHS9t+<wbr>bDiiErwS1SikvSV
qF64/L0u8hYFLD1ehURoHBi4foE1Td<wbr>3eRGOGHgodcYL9C8U+Yv+<wbr>fWaiYQ5O4CCnW
pToFvVoQOdZY+VVC8hz1ggbRMSxjT2<wbr>GQLLJ2mjbGzGUJjlwyQaoZnADSSu0e<wbr>fj88
O2AlWB2Bf/Ag6E4C9jEjj+cauBfR+1<wbr>NIK7F1Jo6C02yY1SUOSoOAFDZ7EkO4<wbr>qYAO
YhvgSQXNmKps6rusNjNZ
=3Dq8Eh
-----END PGP SIGNATURE-----
------------------------------<wbr>------------------------------<wbr>-----=
----
To unsubscribe, e-mail: <a href=3D"mailto:[email protected]=
rg" target=3D"_blank">[email protected]<wbr>e.org</a>
For additional commands, e-mail: <a href=3D"mailto:[email protected]=
.org" target=3D"_blank">[email protected]</a>
</div></div></pre>
</div>
</blockquote>
</div>
</div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>
--001a11432ade8c612a055176dcd0--