Re: Seeking a smarter tokenize for augmented text

"Martin Honnen [email protected]" <[email protected]> Fri, 7 May 2021 10:12:00 -0000
Newsgroups gmane.text.xml.xsl.general.mulberrytech
Message-ID <[email protected]>
This is a multi-part message in MIME format.
--------------6B97B6DC1B2A742D50DCF5D8
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: quoted-printable


Am 06.05.2021 um 11:09 schrieb Trevor Nicholls [email protected]:
>
> I would like to extend this so that it can handle an element
> consisting of text lines with some embedded markup (there won't be
> much, but there will be some). For example, taking:
>
> <textlines>this is line <seq>1</seq>
>
> this is <var>line</var> 2
>
> this <emph>is</emph> line 3</textlines>
>
> and producing
>
> <textlines>
>
> <line>this is line <seq>1</seq></line>
>
> <line>this is <var>line</var> 2</line>
>
> <line>this <emph>is</emph> line 3</line>
>
> </textlines>
>

That part could be done with


<xsl:stylesheet xmlns:xsl=3D"http://www.w3.org/1999/XSL/Transform"
 =A0=A0=A0 xmlns:xs=3D"http://www.w3.org/2001/XMLSchema"
 =A0=A0=A0 exclude-result-prefixes=3D"#all"
 =A0=A0=A0 version=3D"3.0">

 =A0 <xsl:mode on-no-match=3D"shallow-copy"/>

 =A0 <xsl:mode name=3D"insert-markers" on-no-match=3D"shallow-copy"/>

 =A0 <xsl:template match=3D"textlines">
 =A0=A0=A0=A0=A0 <xsl:copy>
 =A0=A0=A0=A0=A0=A0=A0=A0=A0 <xsl:variable name=3D"transformed-textlines">
 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 <xsl:apply-templates mode=3D"inser=
t-markers"/>
 =A0=A0=A0=A0=A0=A0=A0=A0=A0 </xsl:variable>
 =A0=A0=A0=A0=A0=A0=A0=A0=A0 <xsl:for-each-group select=3D"$transformed-tex=
tlines/node()"
group-ending-with=3D"lb">
 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 <line>
 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 <xsl:apply-templates s=
elect=3D"current-group()"/>
 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 </line>
 =A0=A0=A0=A0=A0=A0=A0=A0=A0 </xsl:for-each-group>
 =A0=A0=A0=A0=A0 </xsl:copy>
 =A0 </xsl:template>

 =A0 <xsl:template mode=3D"insert-markers" match=3D"text()">
 =A0=A0=A0=A0=A0 <xsl:analyze-string select=3D"." regex=3D"\n+">
 =A0=A0=A0=A0=A0=A0=A0=A0=A0 <xsl:matching-substring>
 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 <lb/>
 =A0=A0=A0=A0=A0=A0=A0=A0=A0 </xsl:matching-substring>
 =A0=A0=A0=A0=A0=A0=A0=A0=A0 <xsl:non-matching-substring>
 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 <xsl:value-of select=3D"."/>
 =A0=A0=A0=A0=A0=A0=A0=A0=A0 </xsl:non-matching-substring>
 =A0=A0=A0=A0=A0 </xsl:analyze-string>
 =A0 </xsl:template>

 =A0 <xsl:template match=3D"lb"/>

</xsl:stylesheet>


https://xsltfiddle.liberty-development.net/6qaHaQK
--~----------------------------------------------------------------
XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list
EasyUnsubscribe: http://lists.mulberrytech.com/unsub/xsl-list/3329386
or by email: [email protected]
--~--

--------------6B97B6DC1B2A742D50DCF5D8
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

<html>
  <head>
    <meta http-equiv=3D"Content-Type" content=3D"text/html;
      charset=3Dwindows-1252">
  </head>
  <body>
    <p><br>
    </p>
    <div class=3D"moz-cite-prefix">Am 06.05.2021 um 11:09 schrieb Trevor
      Nicholls <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:trevor@=
castingthevoid.com">[email protected]</a>:<br>
    </div>
    <blockquote type=3D"cite"
      cite=3D"mid:[email protected]">
      <p class=3D"MsoNormal">I would like to extend this so that it can
        handle an element consisting of text lines with some embedded
        markup (there won't be much, but there will be some). For
        example, taking:<o:p></o:p></p>
      <p class=3D"MsoNormal"><o:p>=A0</o:p></p>
      <p class=3D"MsoNormal">&lt;textlines&gt;this is line
        &lt;seq&gt;1&lt;/seq&gt;<o:p></o:p></p>
      <p class=3D"MsoNormal">this is &lt;var&gt;line&lt;/var&gt; 2<o:p></o:=
p></p>
      <p class=3D"MsoNormal">this &lt;emph&gt;is&lt;/emph&gt; line
        3&lt;/textlines&gt;<o:p></o:p></p>
      <p class=3D"MsoNormal"><o:p>=A0</o:p></p>
      <p class=3D"MsoNormal">and producing<o:p></o:p></p>
      <p class=3D"MsoNormal"><o:p>=A0</o:p></p>
      <p class=3D"MsoNormal">&lt;textlines&gt;<o:p></o:p></p>
      <p class=3D"MsoNormal">&lt;line&gt;this is line
        &lt;seq&gt;1&lt;/seq&gt;&lt;/line&gt;<o:p></o:p></p>
      <p class=3D"MsoNormal">&lt;line&gt;this is
        &lt;var&gt;line&lt;/var&gt; 2&lt;/line&gt;<o:p></o:p></p>
      <p class=3D"MsoNormal">&lt;line&gt;this &lt;emph&gt;is&lt;/emph&gt;
        line 3&lt;/line&gt;<o:p></o:p></p>
      <p class=3D"MsoNormal">&lt;/textlines&gt;</p>
    </blockquote>
    <p><br>
    </p>
    <p>That part could be done with</p>
    <p><br>
    </p>
    <p>&lt;xsl:stylesheet
      xmlns:xsl=3D<a class=3D"moz-txt-link-rfc2396E" href=3D"http://www.w3.=
org/1999/XSL/Transform">"http://www.w3.org/1999/XSL/Transform"</a><br>
      =A0=A0=A0 xmlns:xs=3D<a class=3D"moz-txt-link-rfc2396E" href=3D"http:=
//www.w3.org/2001/XMLSchema">"http://www.w3.org/2001/XMLSchema"</a><br>
      =A0=A0=A0 exclude-result-prefixes=3D"#all"<br>
      =A0=A0=A0 version=3D"3.0"&gt;<br>
      <br>
      =A0 &lt;xsl:mode on-no-match=3D"shallow-copy"/&gt;<br>
      =A0 <br>
      =A0 &lt;xsl:mode name=3D"insert-markers"
      on-no-match=3D"shallow-copy"/&gt;<br>
      <br>
      =A0 &lt;xsl:template match=3D"textlines"&gt;<br>
      =A0=A0=A0=A0=A0 &lt;xsl:copy&gt;<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;xsl:variable name=3D"transformed-text=
lines"&gt;<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;xsl:apply-templates mode=
=3D"insert-markers"/&gt;<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;/xsl:variable&gt;<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;xsl:for-each-group
      select=3D"$transformed-textlines/node()" group-ending-with=3D"lb"&gt;=
<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;line&gt;<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;xsl:apply-tem=
plates
      select=3D"current-group()"/&gt;<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;/line&gt;<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;/xsl:for-each-group&gt;<br>
      =A0=A0=A0=A0=A0 &lt;/xsl:copy&gt;<br>
      =A0 &lt;/xsl:template&gt;<br>
      =A0 <br>
      =A0 &lt;xsl:template mode=3D"insert-markers" match=3D"text()"&gt;<br>
      =A0=A0=A0=A0=A0 &lt;xsl:analyze-string select=3D"." regex=3D"\n+"&gt;=
<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;xsl:matching-substring&gt;<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;lb/&gt;<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;/xsl:matching-substring&gt;<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;xsl:non-matching-substring&gt;<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;xsl:value-of select=3D"."=
/&gt;<br>
      =A0=A0=A0=A0=A0=A0=A0=A0=A0 &lt;/xsl:non-matching-substring&gt;<br>
      =A0=A0=A0=A0=A0 &lt;/xsl:analyze-string&gt;<br>
      =A0 &lt;/xsl:template&gt;<br>
      =A0 <br>
      =A0 &lt;xsl:template match=3D"lb"/&gt;<br>
      =A0 <br>
      &lt;/xsl:stylesheet&gt;</p>
    <p><br>
    </p>
    <p><a class=3D"moz-txt-link-freetext" href=3D"https://xsltfiddle.libert=
y-development.net/6qaHaQK">https://xsltfiddle.liberty-development.net/6qaHa=
QK</a><br>
    </p>
  </body>
</html>
<div><!-- begin bl.html.trailer -->
<div style=3D"border-top:1px solid black; background-color: #dddddd;
color: #888888; font-size: smaller; padding: 5px; text-align: center;
font-family: arial,verdana,arial,sans-serif; margin-top:1em; clear:
both; margin: auto">
<a href=3D"http://www.mulberrytech.com/xsl/xsl-list">
XSL-List info and archive</a>
<div style=3D"text-align:center;">
<a style=3D"color: blue;"
  href=3D"http://lists.mulberrytech.com/unsub/xsl-list/3329386"
>EasyUnsubscribe</a>
(<a style=3D"color: blue;"
href=3D"mailto:[email protected]?subject=3Dremove"
>by email</a>)
</div>
</div>
<!-- end bl.html.trailer --></div>

--------------6B97B6DC1B2A742D50DCF5D8--