Re: Getting UTF-16 encoding on dynamic content regardless of output content type
gelo1234 <[email protected]> Tue, 29 Mar 2022 19:37:48 +0200
| Newsgroups | gmane.text.xml.cocoon.user |
|---|---|
| Message-ID | <CAPJaKUqOKh2BD8vf8VZYdb=DE8+h-5NB2s+=wiZ2XuVLuNrE7A@mail.gmail.com> |
--000000000000d45e9305db5ee309 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Chris, I think you will not get any icon-type character on output without using proper font rendering - like Emoji support? Emoji might not be supported by default in Cocoon. So this might be the reason why you get HTML entities instead of Emoji-icons. Also notice: https://www.mail-archive.com/[email protected]/msg61629.html Greetings, Greg wt., 29 mar 2022 o 18:36 Christopher Schultz <[email protected]> napisa=C5=82(a): > C=C3=A9dric, > > On 3/29/22 12:06, C=C3=A9dric Damioli wrote: > > Could you provide more details ? > > How is your XML processed before outputting the wrong UTF-8 sequence ? > > It's somewhat straightforward: > > <map:match pattern=3D"/foo"> > <map:generate src=3D"https://source/" /> > > <map:transform src=3D"stuff-to-cincludes.xsl" /> > > <map:transform src=3D"other-stuff-to-cincludes.xsl" /> > > <map:transform type=3D"cinclude" /> > > <map:transform src=3D"my-big-transformer-to-xhtml.xsl" /> > > <map:transform type=3D"cinclude" /><!-- Yes, another one --> > > <map:transform type=3D"i18n" /> > > <map:transform src=3D"strip-namespaces.xsl" /><!-- This is mine, not > Cocoons -> > > <map:serialize type=3D"xhtml" /> > </map:match> > > The xhtml serializer is the default, with encoding set to UTF-8. The > HTTP response has "Content-Type: text/html" and the document itself > contains: > > <?xml version=3D"1.0" encoding=3D"UTF-8"?> > > and > > <meta content=3D"text/html; charset=3Dutf-8" http-equiv=3D"content-type" = /> > > So I think everything is configured correctly; it's just that those > characters are getting mangled by something. I can try to cut-out some > of those steps and see where it's happening. > > I seem to remember being able to give each pipeline step a "marker" or > something where you can say "stop after step 3" or whatever instead of > having to chop-out configuration. Can you remind me or what that is again= ? > > Thanks, > -chris > > > Le 29/03/2022 =C3=A0 17:48, Christopher Schultz a =C3=A9crit : > >> All, > >> > >> I'm still struggling with this. I have upgraded to 2.1.13 which > >> includes the fix for https://issues.apache.org/jira/browse/COCOON-2352 > >> but I'm still getting that American flag converted into those 4 HTML > >> entities: > >> > >> ���� > >> > >> I would expect there to be a single (multibyte) character in the > >> output with no HTML entities. > >> > >> I've double-checked, and the source XML contains the flag as a single > >> multi-byte character, served as UTF-8. > >> > >> Any ideas for how to get this working? I'm sure I could put together a > >> trivial test-case. > >> > >> Thanks, > >> -chris > >> > >> On 10/30/18 12:18, Christopher Schultz wrote: > >>> All, > >>> > >>> Some additional information at the end. > >>> > >>> On 10/30/18 11:58, Christopher Schultz wrote: > >>>> All, > >>> > >>>> I'm attempting to do everything with UTF-8 in Cocoon 2.1.11. I have > >>>> a servlet generating XML in UTF-8 encoding and I have a pipeline > >>>> with a few transforms in it, ultimately serializing to XHTML. > >>> > >>>> If I have a Unicode character in the XML which is outside of the > >>>> BMP, such as this one: =F0=9F=87=BA=F0=9F=87=B8 (that's an American= flag, in case your > >>>> mail reader doesn't render it correctly), then I end up getting a > >>>> series of bytes coming from Cocoon after the transform that look > >>>> like UTF-16. > >>> > >>>> Here's what's in the XML: > >>> > >>>> <first-name>Test=F0=9F=87=BA=F0=9F=87=B8</first-name> > >>> > >>>> Just like that. The bytes in the message for the flag character > >>>> are: > >>> > >>>> f0 9f 87 ba f0 9f 87 b8 > >>> > >>>> When rendering that into XHTML, I'm getting this in the output: > >>> > >>>> Test���� > >>> > >>>> The American flag in Unicode reference can be found here: > >>>> > https://apps.timwhitlock.info/unicode/inspect?s=3D%F0%9F%87%BA%F0%9F%87% > >>> B8 > >>> > >>>> You can see it broken down a bit better here for "Regional U": > >>>> http://www.fileformat.info/info/unicode/char/1f1fa/index.htm > >>> > >>>> and "Regional S": > >>>> http://www.fileformat.info/info/unicode/char/1f1f8/index.htm > >>> > >>>> What's happening is that some component in Cocoon has decided to > >>>> generate HTML entities instead of just emitting the character. > >>>> That's okay IMO. But what it does doesn't make sense for a UTF-8 > >>>> output encodin g. > >>> > >>>> The first two entities "��" are the decimal numbers > >>>> that represent the UTF-16 character for that "Regional Indicator > >>>> Symbol Letter U" and they are correct... for UTF-16. If I change > >>>> the output encoding from UTF-8 to UTF0-16, then the browser will > >>>> render these correctly. Using UTF-8, they show as four of those > >>>> ugly [?] characters on the screen. > >>> > >>>> I had originally just decided to throw up my hands and use UTF-16 > >>>> encoding even though it's dumb. But it seems that MSIE cannot be > >>>> convinced to use UTF-16 no matter what, and I must continue to > >>>> support MSIE. :( > >>> > >>>> So it's back to UTF-8 for me. > >>> > >>>> How can I get Cocoon to output that character (or "those > >>>> characters") correctly? > >>> > >>>> It needs to be one of the following: > >>> > >>>> 🇺🇸 (HTML decimal entities) > >>>> 🇺🇸 (HTML hex entities) f0 9f 87 ba > >>>> f0 9f 87 b8 (raw UTF-8 bytes) > >>> > >>>> Does anyone know how/where this conversion is being performed ion > >>>> Cocoon? Probably in a XHTML serializer (I'm using > >>>> org.apache.cocoon.serialization.XMLSerializer). I'm using > >>>> mime-type "text/html" and <encoding>UTF-8</encoding> in my sitemap > >>>> for that serializer (the one named "xhtml"). I believe I've mads > >>>> very few changes from the default, if any. > >>> > >>>> I haven't yet figured out how to get from what Java sees (\uE50C > >>>> for the "S" for example) to 🇸, but knowing where the code > >>>> is that is making that decision would be very helpful. > >>> > >>>> Any ideas? > >>> > >>>> -chris > >>> > >>> I created a text file (UTF-8) containing only the flag and read it in > >>> using Java and printed all of the code points. There should be 2 > >>> "characters" in the file. It's 4 bytes per UTF-8 character so I > >>> assumed I'd end up with 2 'char' primitives in the file, but I ended > >>> up with more. > >>> > >>> Here's the loop and the output: > >>> > >>> try(java.io.FileReader in =3D new > java.io.FileReader("file.txt")) > >>> { > >>> char[] chars =3D new char[10]; > >>> > >>> int count =3D in.read(chars); > >>> > >>> for(int i=3D0; i<count; ++i) > >>> System.out.println("Code point at " + i + " is " + > >>> Integer.toHexString(Character.codePointAt(chars, i))); > >>> > >>> } catch (Exception e) { > >>> e.printStackTrace(); > >>> } > >>> > >>> =3D=3D output =3D=3D > >>> > >>> Code point at 0 is 1f1fa > >>> Code point at 1 is ddfa > >>> Code point at 2 is 1f1f8 > >>> Code point at 3 is ddf8 > >>> Code point at 4 is a > >>> > >>> So Java thinks there are 4 things there, not 2. That could be a part > >>> of the confusion. The code points shown for indexes 0 and 2 are the > >>> "correct" ones. Those at indexes 1 and 3 should actually be *skipped*= . > >>> > >>> So, to render this string as an HTML numeric entity, we'd do somethin= g > >>> like this: > >>> > >>> String str =3D // this is the input > >>> > >>> for(int i=3D0; i<str.length(); ++i) { > >>> int cp =3D Character.codePointAt(chars, i); > >>> > >>> out.print("&#x"); > >>> out.print(Integer.toHexString(cp)); > >>> out.println(';'); > >>> > >>> // Skip any trailing "characters" that are actually a part of this > >>> one > >>> if(1 < Character.charCount(cp)) > >>> i +=3D Character.charCount(cp) - 1; > >>> } > >>> > >>> Using the above code is completely encoding-agnostic, because it's > >>> describing the Unicode code point and not some set of bytes in a > >>> particular flavor of UTF-x. > >>> > >>> -chris > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --000000000000d45e9305db5ee309 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">Hel= lo Chris,<br></div><div class=3D"gmail_default" style=3D"font-size:small"><= br></div><div class=3D"gmail_default" style=3D"font-size:small">I think you= will not get any icon-type character on output without using proper font r= endering - like Emoji support? Emoji might not be supported by default in C= ocoon.</div><div class=3D"gmail_default" style=3D"font-size:small">So this = might be the reason why you get HTML entities instead of Emoji-icons. <br><= /div><div class=3D"gmail_default" style=3D"font-size:small">Also notice: <a= href=3D"https://www.mail-archive.com/[email protected]/msg61629.html">= https://www.mail-archive.com/[email protected]/msg61629.html</a></div><= div class=3D"gmail_default" style=3D"font-size:small"><br></div><div class= =3D"gmail_default" style=3D"font-size:small">Greetings,</div><div class=3D"= gmail_default" style=3D"font-size:small">Greg<br></div><div class=3D"gmail_= default" style=3D"font-size:small"><br></div><div class=3D"gmail_default" s= tyle=3D"font-size:small"><br></div></div><br><div class=3D"gmail_quote"><di= v dir=3D"ltr" class=3D"gmail_attr">wt., 29 mar 2022 o 18:36=C2=A0Christophe= r Schultz <<a href=3D"mailto:[email protected]">chris@christo= pherschultz.net</a>> napisa=C5=82(a):<br></div><blockquote class=3D"gmai= l_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,20= 4,204);padding-left:1ex">C=C3=A9dric,<br> <br> On 3/29/22 12:06, C=C3=A9dric Damioli wrote:<br> > Could you provide more details ?<br> > How is your XML processed before outputting the wrong UTF-8 sequence ?= <br> <br> It's somewhat straightforward:<br> <br> <map:match pattern=3D"/foo"><br> =C2=A0 =C2=A0<map:generate src=3D"<a href=3D"https://source/" rel= =3D"noreferrer" target=3D"_blank">https://source/</a>" /><br> <br> =C2=A0 =C2=A0<map:transform src=3D"stuff-to-cincludes.xsl" /&g= t;<br> <br> =C2=A0 =C2=A0<map:transform src=3D"other-stuff-to-cincludes.xsl&quo= t; /><br> <br> =C2=A0 =C2=A0<map:transform type=3D"cinclude" /><br> <br> =C2=A0 =C2=A0<map:transform src=3D"my-big-transformer-to-xhtml.xsl&= quot; /><br> <br> =C2=A0 =C2=A0<map:transform type=3D"cinclude" /><!-- Yes= , another one --><br> <br> =C2=A0 =C2=A0<map:transform type=3D"i18n" /><br> <br> =C2=A0 =C2=A0<map:transform src=3D"strip-namespaces.xsl" />= <!-- This is mine, not <br> Cocoons -><br> <br> =C2=A0 =C2=A0<map:serialize type=3D"xhtml" /><br> </map:match><br> <br> The xhtml serializer is the default, with encoding set to UTF-8. The <br> HTTP response has "Content-Type: text/html" and the document itse= lf <br> contains:<br> <br> <?xml version=3D"1.0" encoding=3D"UTF-8"?><br> <br> and<br> <br> <meta content=3D"text/html; charset=3Dutf-8" http-equiv=3D&quo= t;content-type" /><br> <br> So I think everything is configured correctly; it's just that those <br= > characters are getting mangled by something. I can try to cut-out some <br> of those steps and see where it's happening.<br> <br> I seem to remember being able to give each pipeline step a "marker&quo= t; or <br> something where you can say "stop after step 3" or whatever inste= ad of <br> having to chop-out configuration. Can you remind me or what that is again?<= br> <br> Thanks,<br> -chris<br> <br> > Le 29/03/2022 =C3=A0 17:48, Christopher Schultz a =C3=A9crit=C2=A0:<br= > >> All,<br> >><br> >> I'm still struggling with this. I have upgraded to 2.1.13 whic= h <br> >> includes the fix for <a href=3D"https://issues.apache.org/jira/bro= wse/COCOON-2352" rel=3D"noreferrer" target=3D"_blank">https://issues.apache= .org/jira/browse/COCOON-2352</a> <br> >> but I'm still getting that American flag converted into those = 4 HTML <br> >> entities:<br> >><br> >> &#55356;&#56826;&#55356;&#56824;<br> >><br> >> I would expect there to be a single (multibyte) character in the <= br> >> output with no HTML entities.<br> >><br> >> I've double-checked, and the source XML contains the flag as a= single <br> >> multi-byte character, served as UTF-8.<br> >><br> >> Any ideas for how to get this working? I'm sure I could put to= gether a <br> >> trivial test-case.<br> >><br> >> Thanks,<br> >> -chris<br> >><br> >> On 10/30/18 12:18, Christopher Schultz wrote:<br> >>> All,<br> >>><br> >>> Some additional information at the end.<br> >>><br> >>> On 10/30/18 11:58, Christopher Schultz wrote:<br> >>>> All,<br> >>><br> >>>> I'm attempting to do everything with UTF-8 in Cocoon 2= .1.11. I have<br> >>>> a servlet generating XML in UTF-8 encoding and I have a pi= peline<br> >>>> with a few transforms in it, ultimately serializing to XHT= ML.<br> >>><br> >>>> If I have a Unicode character in the XML which is outside = of the<br> >>>> BMP, such as this one: =F0=9F=87=BA=F0=9F=87=B8=C2=A0 (tha= t's an American flag, in case your<br> >>>> mail reader doesn't render it correctly), then I end u= p getting a<br> >>>> series of bytes coming from Cocoon after the transform tha= t look<br> >>>> like UTF-16.<br> >>><br> >>>> Here's what's in the XML:<br> >>><br> >>>> <first-name>Test=F0=9F=87=BA=F0=9F=87=B8</first-n= ame><br> >>><br> >>>> Just like that. The bytes in the message for the flag char= acter<br> >>>> are:<br> >>><br> >>>> f0=C2=A0 9f=C2=A0 87=C2=A0 ba=C2=A0 f0=C2=A0 9f=C2=A0 87= =C2=A0 b8<br> >>><br> >>>> When rendering that into XHTML, I'm getting this in th= e output:<br> >>><br> >>>> Test&#55356;&#56826;&#55356;&#56824;<br> >>><br> >>>> The American flag in Unicode reference can be found here:<= br> >>>> <a href=3D"https://apps.timwhitlock.info/unicode/inspect?s= =3D%F0%9F%87%BA%F0%9F%87%" rel=3D"noreferrer" target=3D"_blank">https://app= s.timwhitlock.info/unicode/inspect?s=3D%F0%9F%87%BA%F0%9F%87%</a><br> >>> B8<br> >>><br> >>>> =C2=A0 You can see it broken down a bit better here for &q= uot;Regional U":<br> >>>> <a href=3D"http://www.fileformat.info/info/unicode/char/1f= 1fa/index.htm" rel=3D"noreferrer" target=3D"_blank">http://www.fileformat.i= nfo/info/unicode/char/1f1fa/index.htm</a><br> >>><br> >>>> and "Regional S":<br> >>>> <a href=3D"http://www.fileformat.info/info/unicode/char/1f= 1f8/index.htm" rel=3D"noreferrer" target=3D"_blank">http://www.fileformat.i= nfo/info/unicode/char/1f1f8/index.htm</a><br> >>><br> >>>> What's happening is that some component in Cocoon has = decided to<br> >>>> generate HTML entities instead of just emitting the charac= ter.<br> >>>> That's okay IMO. But what it does doesn't make sen= se for a UTF-8<br> >>>> output encodin g.<br> >>><br> >>>> The first two entities "&#55356;&#56826;"= ; are the decimal numbers<br> >>>> that represent the UTF-16 character for that "Regiona= l Indicator<br> >>>> Symbol Letter U" and they are correct... for UTF-16. = If I change<br> >>>> the output encoding from UTF-8 to UTF0-16, then the browse= r will<br> >>>> render these correctly. Using UTF-8, they show as four of = those<br> >>>> ugly [?] characters on the screen.<br> >>><br> >>>> I had originally just decided to throw up my hands and use= UTF-16<br> >>>> encoding even though it's dumb. But it seems that MSIE= cannot be<br> >>>> convinced to use UTF-16 no matter what, and I must continu= e to<br> >>>> support MSIE. :(<br> >>><br> >>>> So it's back to UTF-8 for me.<br> >>><br> >>>> How can I get Cocoon to output that character (or "th= ose<br> >>>> characters") correctly?<br> >>><br> >>>> It needs to be one of the following:<br> >>><br> >>>> &#127482;&#127480; (HTML decimal entities)<br> >>>> &#x1f1fa;&#x1f1f8;=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (HTML hex entities) f0 9f=C2=A0 = 87=C2=A0 ba<br> >>>> f0=C2=A0 9f=C2=A0 87=C2=A0 b8 (raw UTF-8 bytes)<br> >>><br> >>>> Does anyone know how/where this conversion is being perfor= med ion<br> >>>> Cocoon? Probably in a XHTML serializer (I'm using<br> >>>> org.apache.cocoon.serialization.XMLSerializer). I'm us= ing<br> >>>> mime-type "text/html" and <encoding>UTF-8&= lt;/encoding> in my sitemap<br> >>>> for that serializer (the one named "xhtml"). I b= elieve I've mads<br> >>>> very few changes from the default, if any.<br> >>><br> >>>> I haven't yet figured out how to get from what Java se= es (\uE50C<br> >>>> for the "S" for example) to &#x1f1f8;, but k= nowing where the code<br> >>>> is that is making that decision would be very helpful.<br> >>><br> >>>> Any ideas?<br> >>><br> >>>> -chris<br> >>><br> >>> I created a text file (UTF-8) containing only the flag and rea= d it in<br> >>> using Java and printed all of the code points. There should be= 2<br> >>> "characters" in the file. It's 4 bytes per UTF-8= character so I<br> >>> assumed I'd end up with 2 'char' primitives in the= file, but I ended<br> >>> up with more.<br> >>><br> >>> Here's the loop and the output:<br> >>><br> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 try(java.io.F= ileReader in =3D new java.io.FileReader("file.txt"))<br> >>> {<br> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 char[] chars =3D new char[10];<br> >>><br> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 int count =3D in.read(chars);<br> >>><br> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 for(int i=3D0; i<count; ++i)<br> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 System.out.println("Code point at= " + i + " is " +<br> >>> Integer.toHexString(Character.codePointAt(chars, i)));<br> >>><br> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 } catch (Exce= ption e) {<br> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 e.printStackTrace();<br> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }<br> >>><br> >>> =3D=3D output =3D=3D<br> >>><br> >>> Code point at 0 is 1f1fa<br> >>> Code point at 1 is ddfa<br> >>> Code point at 2 is 1f1f8<br> >>> Code point at 3 is ddf8<br> >>> Code point at 4 is a<br> >>><br> >>> So Java thinks there are 4 things there, not 2. That could be = a part<br> >>> of the confusion. The code points shown for indexes 0 and 2 ar= e the<br> >>> "correct" ones. Those at indexes 1 and 3 should actu= ally be *skipped*.<br> >>><br> >>> So, to render this string as an HTML numeric entity, we'd = do something<br> >>> like this:<br> >>><br> >>> String str =3D // this is the input<br> >>><br> >>> for(int i=3D0; i<str.length(); ++i) {<br> >>> =C2=A0=C2=A0 int cp =3D Character.codePointAt(chars, i);<br> >>><br> >>> =C2=A0=C2=A0 out.print("&#x");<br> >>> =C2=A0=C2=A0 out.print(Integer.toHexString(cp));<br> >>> =C2=A0=C2=A0 out.println(';');<br> >>><br> >>> =C2=A0=C2=A0 // Skip any trailing "characters" that = are actually a part of this <br> >>> one<br> >>> =C2=A0=C2=A0 if(1 < Character.charCount(cp))<br> >>> =C2=A0=C2=A0=C2=A0=C2=A0 i +=3D Character.charCount(cp) - 1;<b= r> >>> }<br> >>><br> >>> Using the above code is completely encoding-agnostic, because = it's<br> >>> describing the Unicode code point and not some set of bytes in= a<br> >>> particular flavor of UTF-x.<br> >>><br> >>> -chris<br> >><br> >> ------------------------------------------------------------------= ---<br> >> To unsubscribe, e-mail: <a href=3D"mailto:users-unsubscribe@cocoon= .apache.org" target=3D"_blank">[email protected]</a><br> >> For additional commands, e-mail: <a href=3D"mailto:users-help@coco= on.apache.org" target=3D"_blank">[email protected]</a><br> >><br> > <br> <br> ---------------------------------------------------------------------<br> To unsubscribe, e-mail: <a href=3D"mailto:[email protected]= rg" target=3D"_blank">[email protected]</a><br> For additional commands, e-mail: <a href=3D"mailto:[email protected]= .org" target=3D"_blank">[email protected]</a><br> <br> </blockquote></div> --000000000000d45e9305db5ee309--