Re: libsword issue in handling XML entities such as &
DM Smith <[email protected]> Sun, 3 Aug 2025 20:12:38 -0400
| Newsgroups | gmane.comp.literature.sword.devel |
|---|---|
| Message-ID | <[email protected]> |
--===============7628692950046350330== Content-Type: multipart/alternative; boundary="Apple-Mail=_19E169DA-8FDC-4A71-8B3C-1A0084275150" --Apple-Mail=_19E169DA-8FDC-4A71-8B3C-1A0084275150 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Aug 2, 2025, at 9:40=E2=80=AFAM, David Haslam = <dfhdfh-g/[email protected]> wrote: >=20 > Hi DM, >=20 > Does this part of your reply contain a typo? > Also, osis2mod will properly convert it to &AMP; And it will = output a diagnostic for the conversion. No typo. & may be valid for HTML, but it is not valid for XML. In = XML, <, >, &, " and ' are the only entities that are = defined. " and ' are only needed within attributes when the = mark matches the quotation used for the attribute. & is needed to = have these 4 named attributes but allow for & in the text. CrossWire=E2=80=99s module team always uses an XML validator, xmllint, = to validate that the XML is well formed and conformant to the schema. It = would report & as an error. When osis2mod encounters a named character entity, it doesn=E2=80=99t = assume that xmllint or other was used. Rather it escapes all other named = entities that are valid in HTML. So & will become &AMP; with a = warning. The other choice would be to quit at that point because the = user didn=E2=80=99t use an XML validator first. In Him, DM >=20 > Best regards, >=20 > David >=20 > Sent with Proton Mail <https://pr.tn/ref/SWXT9A5YZ67G> secure email. >=20 > On Thursday, July 31st, 2025 at 2:35 PM, DM Smith = <[email protected]> wrote: >> See below. >>=20 >>> On Jul 31, 2025, at 8:53=E2=80=AFAM, David Haslam = <dfhdfh-g/[email protected] <mailto:dfhdfh-g/[email protected]>> wrote: >>>=20 >>> Hi DM, >>>=20 >>> Further to the feedback relating to the recent updates of the = FreGeneve1669 module... >>>=20 >>> Has anyone yet created an issue in JIRA for the failure of libsword = to output the & character wherever a module contains the XML entity = & ? >>=20 >> I didn=E2=80=99t see any when I surveyed the Jira issues. I noticed = that &c. is quite common in the KJV margin notes, and front matter. >>=20 >>>=20 >>> Which other XML entities have the same issue? >>> Likely candidates include < and > >>> There may be others such as ' and " >>=20 >> Yes these are other possible problems. >>=20 >>>=20 >>> Does the issue also pertain to their numerical equivalents, whether = in decimal or hexadecimal form? e.g. &x#0027; >>=20 >> We have a Jira issue to convert these to their unicode equivalent. = I=E2=80=99m almost done with that code, having started yesterday. >>=20 >>> Any remedy should also take account that some such entities are also = valid when the name is uppercase, e.g. & >>=20 >> The remedy is that xmllint will complain as it is not valid in xml. = This should be a friend of all OSIS module makers. >>=20 >> Also, osis2mod will properly convert it to &AMP; And it will = output a diagnostic for the conversion. >>=20 >>>=20 >>> Further reading: = https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_refere= nces >>>=20 >>> Best regards, >>>=20 >>> David >>>=20 >>> Sent with Proton Mail <https://pr.tn/ref/SWXT9A5YZ67G> secure email. >>> _______________________________________________ >>> sword-devel mailing list: [email protected] = <mailto:[email protected]> >>> http://crosswire.org/mailman/listinfo/sword-devel >>> Instructions to unsubscribe/change your settings at above page >=20 > _______________________________________________ > sword-devel mailing list: [email protected] > http://crosswire.org/mailman/listinfo/sword-devel > Instructions to unsubscribe/change your settings at above page --Apple-Mail=_19E169DA-8FDC-4A71-8B3C-1A0084275150 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;"><br = id=3D"lineBreakAtBeginningOfMessage"><div><br><blockquote = type=3D"cite"><div>On Aug 2, 2025, at 9:40=E2=80=AFAM, David Haslam = <dfhdfh-g/[email protected]> wrote:</div><br = class=3D"Apple-interchange-newline"><div><div style=3D"font-family: = Arial, sans-serif; font-size: 14px;">Hi DM,<br><br>Does this part of = your reply contain a typo?<br><blockquote style=3D"border-left: 3px = solid rgb(200, 200, 200); border-top-color: rgb(200, 200, 200); = border-right-color: rgb(200, 200, 200); border-bottom-color: rgb(200, = 200, 200); padding-left: 10px; color: rgb(102, 102, 102);"><div>Also, = osis2mod will properly convert it to &amp;AMP; And it will output a = diagnostic for the = conversion.<br></div></blockquote></div></div></blockquote>No typo. = &AMP; may be valid for HTML, but it is not valid for XML. In XML, = &lt;, &gt;, &amp;, &quot; and &apos; are the only = entities that are defined. &quot; and &apos; are only needed = within attributes when the mark matches the quotation used for the = attribute. &amp; is needed to have these 4 named attributes but = allow for & in the text.</div><div><br></div><div>CrossWire=E2=80=99s = module team always uses an XML validator, xmllint, to validate that the = XML is well formed and conformant to the schema. It would report = &AMP; as an error.</div><div><br></div><div>When osis2mod encounters = a named character entity, it doesn=E2=80=99t assume that xmllint or = other was used. Rather it escapes all other named entities that are = valid in HTML. So &AMP; will become &amp;AMP; with a warning. = The other choice would be to quit at that point because the user = didn=E2=80=99t use an XML validator first.</div><div><br></div><div>In = Him,</div><div><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>DM</div><div><br></div><div><blockquote type=3D"cite"><div><div = style=3D"font-family: Arial, sans-serif; font-size: 14px;"><br></div> <div style=3D"font-family: Arial, sans-serif; font-size: 14px;" = class=3D"protonmail_signature_block"> <div class=3D"protonmail_signature_block-user"> Best regards,<br><br>David </div> <div style=3D"font-family: Arial, sans-serif; font-size: = 14px;"><br></div> <div class=3D"protonmail_signature_block-proton"> Sent with <a href=3D"https://pr.tn/ref/SWXT9A5YZ67G" = target=3D"_blank">Proton Mail</a> secure email. </div> </div> <div style=3D"font-family: Arial, sans-serif; font-size: = 14px;"><br></div><div class=3D"protonmail_quote"> On Thursday, July 31st, 2025 at 2:35 PM, DM Smith = <[email protected]> wrote:<br> <blockquote class=3D"protonmail_quote" type=3D"cite"> See below.<br = id=3D"lineBreakAtBeginningOfMessage"><div><br><blockquote = type=3D"cite"><div>On Jul 31, 2025, at 8:53=E2=80=AFAM, David Haslam = <<a href=3D"mailto:dfhdfh-g/[email protected]" rel=3D"noreferrer nofollow = noopener">dfhdfh-g/[email protected]</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div><div style=3D"font-family: = Arial, sans-serif; font-size: 14px;">Hi DM,<br><br>Further to the = feedback relating to the recent updates of = the <b>FreGeneve1669</b> module...<br><br>Has anyone yet = created an issue in JIRA for the failure of <b>libsword</b> to = output the <b>&</b> character wherever a module contains = the XML entity <span style=3D"font-family: Menlo, Consolas, = "Courier New", = monospace;">&amp;</span> ?<br></div></div></blockquote><div><br><= /div>I didn=E2=80=99t see any when I surveyed the Jira issues. I noticed = that &c. is quite common in the KJV margin notes, and front = matter.</div><div><br><blockquote type=3D"cite"><div style=3D"font-family:= Arial, sans-serif; font-size: 14px;"><br>Which other XML entities have = the same issue?<br></div><div style=3D"font-family: Arial, sans-serif; = font-size: 14px;"><ul = data-editing-info=3D"{"orderedStyleType":1,"unorderedStyleT= ype":1}" style=3D"margin-top: 0px; margin-bottom: 0px;"><li = style=3D"list-style-type: disc;">Likely candidates include <span = style=3D"font-family: Menlo, Consolas, "Courier New", = monospace;">&lt;</span> and <span style=3D"font-family: = Menlo, Consolas, "Courier New", = monospace;">&gt;</span><br></li><li style=3D"list-style-type: = disc;">There may be others such as <span style=3D"font-family: = Menlo, Consolas, "Courier New", = monospace;">&apos;</span> and <span style=3D"font-family: = Menlo, Consolas, "Courier New", = monospace;">&quot;</span><br></li></ul></div></blockquote><div><br></d= iv>Yes these are other possible problems.</div><div><br><blockquote = type=3D"cite"><div style=3D"font-family: Arial, sans-serif; font-size: = 14px;"><br><div class=3D"protonmail_signature_block"><div = class=3D"protonmail_signature_block-user"><span style=3D"line-height: = normal;">Does the issue also pertain to = their <b>numerical</b> equivalents, </span><span = style=3D"line-height: normal;">whether = in <b>decimal</b> or <b>hexadecimal</b> form? = e.g. </span><span style=3D"line-height: = normal;">&x#0027;</span><span style=3D"font-family: Menlo, Consolas, = "Courier New", = monospace;"><br></span></div></div></div></blockquote><div><br></div>We = have a Jira issue to convert these to their unicode equivalent. I=E2=80=99= m almost done with that code, having started = yesterday.</div><div><br><blockquote type=3D"cite"><div = style=3D"font-family: Arial, sans-serif; font-size: 14px;"><div = class=3D"protonmail_signature_block"><div = class=3D"protonmail_signature_block-user"><span style=3D"line-height: = normal;">Any remedy should also take account that some such entities are = also valid when the name is <b>uppercase</b>, = e.g. </span><span style=3D"font-family: Menlo, Consolas, = "Courier New", monospace; line-height: = normal;">&AMP;</span><span style=3D"font-family: Menlo, Consolas, = "Courier New", = monospace;"><br></span></div></div></div></blockquote><div><br></div>The = remedy is that xmllint will complain as it is not valid in xml. This = should be a friend of all OSIS module = makers.</div><div><br></div><div>Also, osis2mod will properly convert it = to &amp;AMP; And it will output a diagnostic for the = conversion.</div><div><br><blockquote type=3D"cite"><div = style=3D"font-family: Arial, sans-serif; font-size: 14px;"><div = class=3D"protonmail_signature_block"><div = class=3D"protonmail_signature_block-user"><span style=3D"font-family: = Menlo, Consolas, "Courier New", = monospace;"><br></span></div><div = class=3D"protonmail_signature_block-user"><span style=3D"line-height: = normal;">Further reading: </span><a target=3D"_blank" = rel=3D"noreferrer nofollow noopener" = href=3D"https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entit= y_references"><span style=3D"line-height: = normal;">https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_enti= ty_references</span></a></div><div = class=3D"protonmail_signature_block-user"><br></div><div = class=3D"protonmail_signature_block-user">Best = regards,<br><br>David</div><div><br></div><div = class=3D"protonmail_signature_block-proton">Sent with <a = href=3D"https://pr.tn/ref/SWXT9A5YZ67G" target=3D"_blank" = rel=3D"noreferrer nofollow noopener">Proton Mail</a> secure = email.</div></div></div>_______________________________________________<br= >sword-devel mailing list: <a = href=3D"mailto:[email protected]" rel=3D"noreferrer nofollow = noopener">[email protected]</a><br><a = href=3D"http://crosswire.org/mailman/listinfo/sword-devel" = target=3D"_blank" rel=3D"noreferrer nofollow = noopener">http://crosswire.org/mailman/listinfo/sword-devel</a><br>Instruc= tions to unsubscribe/change your settings at above = page</blockquote></div> </blockquote><br> </div>_______________________________________________<br>sword-devel = mailing list: = [email protected]<br>http://crosswire.org/mailman/listinfo/sword-d= evel<br>Instructions to unsubscribe/change your settings at above = page<br></div></blockquote></div><br></body></html>= --Apple-Mail=_19E169DA-8FDC-4A71-8B3C-1A0084275150-- --===============7628692950046350330== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ sword-devel mailing list: [email protected] http://crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page --===============7628692950046350330==--