Re: [2.1] Overzealous escaping of high Unicode code points
gelo1234 <[email protected]> Wed, 21 Jun 2017 09:26:55 +0200
| Newsgroups | gmane.text.xml.cocoon.user |
|---|---|
| Message-ID | <CAPJaKUoYhdCGwBtmg4_iU+bvS7CY5YudaP7E8OP7j=oZkoqxhw@mail.gmail.com> |
--089e0822e08cf28bf30552734a0b Content-Type: text/plain; charset="UTF-8" Hi Chris, I suppose you cannot use 2 different encodings in 1 Serializer, so if you changed your Serializer config to be UTF16, you also have to use _external_ UTF16 encoded CSS styles. Of couse you can define many different Serializer configs per each pipeline. By default common-lang/cocoon uses 2-byte char sequence as encoding base. If you had UTF-8 and 32 bits, you would have 4 chars (each 8 bits), encoded as 1 PAIR 2-bytes sequence. if you switched to UTF-16, you would have 2 chars (each 16 bits), encoded as 1 SINGLE 4-bytes sequence. Greetings, Greg 2017-06-20 22:14 GMT+02:00 Christopher Schultz <[email protected] >: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Greg, > > On 6/20/17 4:11 PM, Christopher Schultz wrote: > > Greg, > > > > On 6/8/17 2:17 PM, gelo1234 wrote: > >> Chris, > > > >> Even with C3 (cocoon 3.0 beta) unless you specify optional > >> encoding in your Serializer config, you fallback to default > >> UTF-8: > > > >> org.apache.cocoon.optional.servlet.components.sax.serializers.util > > > >> public class ConfigurationUtils { > > > >> private ConfigurationUtils() { } > > > >> public static String getEncoding(Map<String, ? extends Object> > >> configuration) { String encoding = (String) > >> configuration.get("encoding"); > > > >> if (encoding == null || "".equals(encoding)) { encoding = > >> "UTF-8"; } > > > >> return encoding; } ... > > > > I would have expected the Unicode codepoint to be converted into a > > single 4-byte UTF-8 byte without any &-encoding at all. It looks > > like what I got was a pair of 2-byte characters with &-encoding. > > > > I'll try UTF-16 but my expectation is that it's going to get > > worse, not better. > > Interestingly enough, my emojis are now showing (which I don't totally > understand why!) but it looks like my CSS aren't being loaded. That's > a separate problem I'll have to figure out for myself. > > In my own application, switching from commons-lang to commans-lang3 > HTML/XML escaping allowed me to use these 4-byte emojis and UTF-8 > together. I'm surprised that Cocoon can't do the same thing. (I think > it comes down to exactly how the character-escaper makes its decisions). > > Thanks, > - -chris > -----BEGIN PGP SIGNATURE----- > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAllJgiwACgkQHPApP6U8 > pFgJkRAAqiXn7DWNDN41m1V98aI5xWjTuoka0tKcadN1IUGemTZwipaXHtYQcois > 6yuI3st31ZuanghIpRPcBu9pZzuHtOSBVSHZSIhDGqPwYgczScQ2LgnfMi6zwAdd > j2LFlSWtKGjgCczV5Ok56PyMq1BEAOVw96vmF5xfXmpLAyNA/PvLKsncoW4pN+ES > 1MQMm1aPwbmEpWz7ykReUzfauwBtL4rEX1wO3pl88m9Wq3x174AKHWs/a+4Z1Hdq > 0CnxfrdTK50p7Ng+ECfnPwx8y1Em64lA7KKMuz2jTd0PnxlpZTAgO6lq8S7BdSeY > H1lwBJojVT/+m2w8b9OC/XoyiAyiC/zIswQ3TSMA3ZC2SnCxxAXMTsmT49Ql+lyq > 01JRCIVMitKeoKI4I4066oaBW91FpSSpZXX14XCHrMBtKnIJI+NxBnI++eQq8wdi > ZdX3GzLF2zaPHvZMSz4DRskR1xKGLsAxZAukINW3AGrEAZ/GwbPd76ml3YJam5Yy > R31u0kcRJl4z79pd1n46yxB66V10Rn5IkSMQ8R7uK/ht9wLi5T8bkeAoLjZFFoyq > awmfQTbJzquXAtwjX99WKWEzviN2ph+P0h2rBInHnos5ud8IlLjcS7FmdxQ4DNOw > Nirmj7cikxcr2Fn22pGQh6o3/Eph0lMf1d1HjUZ1C7SchEgsqrk= > =0nTd > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --089e0822e08cf28bf30552734a0b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">Hi = Chris,<br></div><div class=3D"gmail_default" style=3D"font-size:small"><br>= </div><div class=3D"gmail_default" style=3D"font-size:small">I suppose you = cannot use 2 different encodings in 1 Serializer, so if you changed<br></di= v><div class=3D"gmail_default" style=3D"font-size:small">your Serializer co= nfig to be UTF16, you also have to use _external_ UTF16 encoded<br></div><d= iv class=3D"gmail_default" style=3D"font-size:small">CSS styles. Of couse y= ou can define many different Serializer configs per each pipeline.<br></div= ><div class=3D"gmail_default" style=3D"font-size:small"><br></div><div clas= s=3D"gmail_default" style=3D"font-size:small">By default common-lang/cocoon= uses 2-byte char sequence as encoding base.<br></div><div class=3D"gmail_d= efault" style=3D"font-size:small">If you had UTF-8 and 32 bits, you would h= ave 4 chars (each 8 bits), encoded<br></div><div class=3D"gmail_default" st= yle=3D"font-size:small">as 1 PAIR 2-bytes sequence.<br></div><div class=3D"= gmail_default" style=3D"font-size:small">if you switched to UTF-16, you wou= ld have 2 chars (each 16 bits), encoded<br></div><div class=3D"gmail_defaul= t" style=3D"font-size:small">as 1 SINGLE 4-bytes sequence.<br><br></div><di= v class=3D"gmail_default" style=3D"font-size:small">Greetings,<br></div><di= v class=3D"gmail_default" style=3D"font-size:small">Greg<br></div><div clas= s=3D"gmail_default" style=3D"font-size:small">=C2=A0</div></div><div class= =3D"gmail_extra"><br><div class=3D"gmail_quote">2017-06-20 22:14 GMT+02:00 = Christopher Schultz <span dir=3D"ltr"><<a href=3D"mailto:chris@christoph= erschultz.net" target=3D"_blank">[email protected]</a>></span= >:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-l= eft:1px #ccc solid;padding-left:1ex"><span class=3D"">-----BEGIN PGP SIGNED= MESSAGE-----<br> Hash: SHA256<br> <br> Greg,<br> <br> </span><span class=3D"">On 6/20/17 4:11 PM, Christopher Schultz wrote:<br> > Greg,<br> ><br> > On 6/8/17 2:17 PM, gelo1234 wrote:<br> >> Chris,<br> ><br> >> Even with C3 (cocoon 3.0 beta) unless you specify optional<br> >> encoding in your Serializer config, you fallback to default<br> >> UTF-8:<br> ><br> >> org.apache.cocoon.optional.<wbr>servlet.components.sax.<wbr>serial= izers.util<br> ><br> >>=C2=A0 public class ConfigurationUtils {<br> ><br> >> private ConfigurationUtils() { }<br> ><br> >> public static String getEncoding(Map<String, ? extends Object&g= t;<br> >> configuration) { String encoding =3D (String)<br> >> configuration.get("encoding");<br> ><br> >> if (encoding =3D=3D null || "".equals(encoding)) { encod= ing =3D<br> >> "UTF-8"; }<br> ><br> >> return encoding; } ...<br> ><br> > I would have expected the Unicode codepoint to be converted into a<br> > single 4-byte UTF-8 byte without any &-encoding at all. It looks<b= r> > like what I got was a pair of 2-byte characters with &-encoding.<b= r> ><br> > I'll try UTF-16 but my expectation is that it's going to get<b= r> > worse, not better.<br> <br> </span>Interestingly enough, my emojis are now showing (which I don't t= otally<br> understand why!) but it looks like my CSS aren't being loaded. That'= ;s<br> a separate problem I'll have to figure out for myself.<br> <br> In my own application, switching from commons-lang to commans-lang3<br> HTML/XML escaping allowed me to use these 4-byte emojis and UTF-8<br> together. I'm surprised that Cocoon can't do the same thing. (I thi= nk<br> it comes down to exactly how the character-escaper makes its decisions).<br= > <span class=3D""><br> Thanks,<br> - -chris<br> -----BEGIN PGP SIGNATURE-----<br> Comment: GPGTools - <a href=3D"http://gpgtools.org" rel=3D"noreferrer" targ= et=3D"_blank">http://gpgtools.org</a><br> Comment: Using GnuPG with Thunderbird - <a href=3D"http://www.enigmail.net/= " rel=3D"noreferrer" target=3D"_blank">http://www.enigmail.net/</a><br> <br> </span>iQIzBAEBCAAdFiEEMmKgYcQvxMe7tc<wbr>JcHPApP6U8pFgFAllJgiwACgkQHPAp<wb= r>P6U8<br> pFgJkRAAqiXn7DWNDN41m1V98aI5xW<wbr>jTuoka0tKcadN1IUGemTZwipaXHtYQ<wbr>cois<= br> 6yuI3st31ZuanghIpRPcBu9pZzuHtO<wbr>SBVSHZSIhDGqPwYgczScQ2LgnfMi6z<wbr>wAdd<= br> j2LFlSWtKGjgCczV5Ok56PyMq1BEAO<wbr>Vw96vmF5xfXmpLAyNA/<wbr>PvLKsncoW4pN+ES<= br> 1MQMm1aPwbmEpWz7ykReUzfauwBtL4<wbr>rEX1wO3pl88m9Wq3x174AKHWs/a+<wbr>4Z1Hdq<= br> 0CnxfrdTK50p7Ng+<wbr>ECfnPwx8y1Em64lA7KKMuz2jTd0Pnx<wbr>lpZTAgO6lq8S7BdSeY<= br> H1lwBJojVT/+m2w8b9OC/XoyiAyiC/<wbr>zIswQ3TSMA3ZC2SnCxxAXMTsmT49Ql<wbr>+lyq<= br> 01JRCIVMitKeoKI4I4066oaBW91FpS<wbr>SpZXX14XCHrMBtKnIJI+NxBnI++<wbr>eQq8wdi<= br> ZdX3GzLF2zaPHvZMSz4DRskR1xKGLs<wbr>AxZAukINW3AGrEAZ/<wbr>GwbPd76ml3YJam5Yy<= br> R31u0kcRJl4z79pd1n46yxB66V10Rn<wbr>5IkSMQ8R7uK/<wbr>ht9wLi5T8bkeAoLjZFFoyq<= br> awmfQTbJzquXAtwjX99WKWEzviN2ph<wbr>+<wbr>P0h2rBInHnos5ud8IlLjcS7FmdxQ4D<wbr= >NOw<br> Nirmj7cikxcr2Fn22pGQh6o3/<wbr>Eph0lMf1d1HjUZ1C7SchEgsqrk=3D<br> =3D0nTd<br> <div class=3D"HOEnZb"><div class=3D"h5">-----END PGP SIGNATURE-----<br> <br> ------------------------------<wbr>------------------------------<wbr>-----= ----<br> To unsubscribe, e-mail: <a href=3D"mailto:[email protected]= rg">users-unsubscribe@cocoon.<wbr>apache.org</a><br> For additional commands, e-mail: <a href=3D"mailto:[email protected]= .org">[email protected]</a><br> <br> </div></div></blockquote></div><br></div> --089e0822e08cf28bf30552734a0b--