[jira] [Comment Edited] (XALANJ-2419) Astral characters written as a pair of NCRs with the surrogate scalar values when using UTF-8

    [ https://issues.apache.org/jira/browse/XALANJ-2419?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D180=
89348#comment-18089348 ]=20

Ilya Basin edited comment on XALANJ-2419 at 6/16/26 10:25 AM:
--------------------------------------------------------------

All was fine until our *{{<xsl:copy>}}* in a transform tried to split an as=
tral character at a chunk boundary which produced two invalid xml entities =
again. So we were forced to apply more patches from upstream.

Also we tried to clone and run [xalan-test xalan-j_2_7_3-rc10|https://githu=
b.com/apache/xalan-test/tree/xalan-j_2_7_3-rc10] but *{{ant check}}* was fa=
iling with missing target "{*}api{*}" and after I reverted the commit that =
renamed that target it was failing to create the *{{results-api/Pass-*.xml}=
}* files so we gave up.

=C2=A0

=C2=A0
{code:java}
commit dfb727767ccbebdb989049de89904521ee981610
Author: kubycsolutions <[email protected]>
Date: =C2=A0 Wed Feb 21 19:55:04 2024 -0500
=C2=A0 =C2=A0 Document the characters()other()characters() issue if first c=
har buffer ended in a high surrogate.
commit ec7f0e25d85192443a9fef2534e7625176fbfa4c
Author: kubycsolutions <[email protected]>
Date: =C2=A0 Wed Feb 21 14:51:48 2024 -0500
=C2=A0 =C2=A0 This one's working for the test added in 2725. May not be cle=
anest solution, and I'm not sure whether any of the other surrogate handlin=
g needs similar fixes -- I don't know whether they ever run into the buffer=
 break problem.
commit 856e896e42bc409e730ed5de0c1e5cd416b8bbc7
Author: kubycsolutions <[email protected]>
Date: =C2=A0 Mon Feb 19 17:03:53 2024 -0500
=C2=A0 =C2=A0 refactoring
commit 162e1f0b4c71669e3c8da8c6d1b7b4ddcdda5789
Author: kubycsolutions <[email protected]>
Date: =C2=A0 Fri Feb 2 14:02:15 2024 -0500
=C2=A0 =C2=A0 just documentation/parameter names
commit beb73389025828731a776d3e10de6cecd6bab1fd
Author: kubycsolutions <[email protected]>
Date: =C2=A0 Sun Oct 22 18:58:13 2023 -0400
=C2=A0 =C2=A0 Deletions, additions, and modifications to complete Maven cut=
-over.
{code}
=C2=A0

_


was (Author: basinilya):
All was fine until our *{{<xsl:copy>}}* in a transform tried to split an as=
tral character at a chunk boundary which produced two invalid xml entities =
again. So we were forced to apply more patches from upstream.

Also we tried to clone and run [xalan-test xalan-j_2_7_3-rc10|https://githu=
b.com/apache/xalan-test/tree/xalan-j_2_7_3-rc10] but *{{ant check}}* was fa=
iling with missing target "{*}api{*}"=C2=A0and after I reverted the commit =
that renamed that target it was failing to create the *results-api/Pass-*.x=
ml* files so we gave up.

=C2=A0

=C2=A0
{code:java}
commit dfb727767ccbebdb989049de89904521ee981610
Author: kubycsolutions <[email protected]>
Date: =C2=A0 Wed Feb 21 19:55:04 2024 -0500
=C2=A0 =C2=A0 Document the characters()other()characters() issue if first c=
har buffer ended in a high surrogate.
commit ec7f0e25d85192443a9fef2534e7625176fbfa4c
Author: kubycsolutions <[email protected]>
Date: =C2=A0 Wed Feb 21 14:51:48 2024 -0500
=C2=A0 =C2=A0 This one's working for the test added in 2725. May not be cle=
anest solution, and I'm not sure whether any of the other surrogate handlin=
g needs similar fixes -- I don't know whether they ever run into the buffer=
 break problem.
commit 856e896e42bc409e730ed5de0c1e5cd416b8bbc7
Author: kubycsolutions <[email protected]>
Date: =C2=A0 Mon Feb 19 17:03:53 2024 -0500
=C2=A0 =C2=A0 refactoring
commit 162e1f0b4c71669e3c8da8c6d1b7b4ddcdda5789
Author: kubycsolutions <[email protected]>
Date: =C2=A0 Fri Feb 2 14:02:15 2024 -0500
=C2=A0 =C2=A0 just documentation/parameter names
commit beb73389025828731a776d3e10de6cecd6bab1fd
Author: kubycsolutions <[email protected]>
Date: =C2=A0 Sun Oct 22 18:58:13 2023 -0400
=C2=A0 =C2=A0 Deletions, additions, and modifications to complete Maven cut=
-over.
{code}
=C2=A0

_

> Astral characters written as a pair of NCRs with the surrogate scalar val=
ues when using UTF-8
> -------------------------------------------------------------------------=
--------------------
>
>                 Key: XALANJ-2419
>                 URL: https://issues.apache.org/jira/browse/XALANJ-2419
>             Project: XalanJ2
>          Issue Type: Bug
>          Components: Serialization
>    Affects Versions: 2.7.1
>            Reporter: Henri Sivonen
>            Assignee: Joe Kesselman
>            Priority: Major
>             Fix For: The Latest Development Code
>
>         Attachments: XALANJ-2419-fix-v3.txt, XALANJ-2419-tests-v3.txt
>
>
> org.apache.xml.serializer.ToStream contains the following code:
>                     else if (m_encodingInfo.isInEncoding(ch)) {
>                         // If the character is in the encoding, and
>                         // not in the normal ASCII range, we also
>                         // just leave it get added on to the clean charac=
ters
>                        =20
>                     }
>                     else {
>                         // This is a fallback plan, we should never get h=
ere
>                         // but if the character wasn't previously handled
>                         // (i.e. isn't in the encoding, etc.) then what
>                         // should we do?  We choose to write out an entit=
y
>                         writeOutCleanChars(chars, i, lastDirtyCharProcess=
ed);
>                         writer.write("&#");
>                         writer.write(Integer.toString(ch));
>                         writer.write(';');
>                         lastDirtyCharProcessed =3D i;
>                     }
> This leads to the wrong (latter) if branch running for surrogates, becaus=
e isInEncoding() for UTF-8 returns false for surrogates. It is always wrong=
 (regardless of encoding) to escape a surrogate as an NCR.
> The practical effect of this bug is that any document with astral charact=
ers in it ends up in an ill-formed serialization and does not parse back us=
ing an XML parser.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)