Re: Seeking a smarter tokenize for augmented text

"Michael Kay [email protected]" <[email protected]> Thu, 6 May 2021 09:29:04 -0000
Newsgroups gmane.text.xml.xsl.general.mulberrytech
Message-ID <[email protected]>
--Apple-Mail=_8C9FD249-B05E-484D-B843-34FF2D7D29AA
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

There seem to be two broad strategies for this kind of problem:

(a) process the text nodes to insert "markup" structure, e.g. replace newli=
nes by <nl/>, and then process as nodes, e.g. using for-each group-starting=
=3D"nl".

(b) process the embedded elements converting them to text, e.g. convert <va=
r>line</var> to %line%, then process as text, then reconstitute the markup.

For this example, (a) seems easier.

Michael Kay
Saxonica

> On 6 May 2021, at 10:09, Trevor Nicholls [email protected] <xsl-l=
[email protected]> wrote:
>=20
> Hi
>=20=20
> Turning xml like this:
>=20=20
> <textlines>this is line 1
> this is line 2
> this is line 3</textlines>
>=20=20
> into something like this:
>=20=20
> <textlines>
> <line>this is line 1</line>
> <line>this is line 2</line>
> <line>this is line 3</line>
> </textlines>
>=20=20
> is simple and I have functions that do it. But they rely on <textlines> b=
eing an xs:string, which is processed by a trivial function I wrote:
>=20=20
> <xsl:function name=3D"my:intoLines" as=3D"xs:string*">
>   <xsl:param name=3D"arg" as=3D"xs:string?" />
>   <xsl:sequence select=3D"tokenize( $arg, '(\r\n?|\n\r?)' )" />
> </xsl:function>
>=20=20
> I would like to extend this so that it can handle an element consisting o=
f text lines with some embedded markup (there won't be much, but there will=
 be some). For example, taking:
>=20=20
> <textlines>this is line <seq>1</seq>
> this is <var>line</var> 2
> this <emph>is</emph> line 3</textlines>
>=20=20
> and producing
>=20=20
> <textlines>
> <line>this is line <seq>1</seq></line>
> <line>this is <var>line</var> 2</line>
> <line>this <emph>is</emph> line 3</line>
> </textlines>
>=20=20
> Sequence and tokenize isn't adequate but is there a way to do this that w=
on't overtax my brain too much?
>=20=20
> cheers
> T
> XSL-List info and archive <http://www.mulberrytech.com/xsl/xsl-list>
> EasyUnsubscribe <http://lists.mulberrytech.com/unsub/xsl-list/293509> (by=
 email <>)
--~----------------------------------------------------------------
XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list
EasyUnsubscribe: http://lists.mulberrytech.com/unsub/xsl-list/3329386
or by email: [email protected]
--~--

--Apple-Mail=_8C9FD249-B05E-484D-B843-34FF2D7D29AA
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; charset=
=3Dus-ascii"></head><body style=3D"word-wrap: break-word; -webkit-nbsp-mode=
: space; line-break: after-white-space;" class=3D"">There seem to be two br=
oad strategies for this kind of problem:<div class=3D""><br class=3D""></di=
v><div class=3D"">(a) process the text nodes to insert "markup" structure, =
e.g. replace newlines by &lt;nl/&gt;, and then process as nodes, e.g. using=
 for-each group-starting=3D"nl".</div><div class=3D""><br class=3D""></div>=
<div class=3D"">(b) process the embedded elements converting them to text, =
e.g. convert &lt;var&gt;line&lt;/var&gt; to %line%, then process as text, t=
hen reconstitute the markup.</div><div class=3D""><br class=3D""></div><div=
 class=3D"">For this example, (a) seems easier.</div><div class=3D""><br cl=
ass=3D""></div><div class=3D"">Michael Kay</div><div class=3D"">Saxonica<br=
 class=3D""><div><br class=3D""><blockquote type=3D"cite" class=3D""><div c=
lass=3D"">On 6 May 2021, at 10:09, Trevor Nicholls <a href=3D"mailto:trevor=
@castingthevoid.com" class=3D"">[email protected]</a> &lt;<a href=
=3D"mailto:[email protected]" class=3D"">xsl-list-ser=
[email protected]</a>&gt; wrote:</div><br class=3D"Apple-intercha=
nge-newline"><div class=3D""><meta charset=3D"UTF-8" class=3D""><div class=
=3D"WordSection1" style=3D"page: WordSection1; caret-color: rgb(0, 0, 0); f=
ont-family: Helvetica; font-size: 13px; font-style: normal; font-variant-ca=
ps: normal; font-weight: normal; letter-spacing: normal; text-align: start;=
 text-indent: 0px; text-transform: none; white-space: normal; word-spacing:=
 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style=3D=
"margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"=
">Hi<o:p class=3D""></o:p></div><div style=3D"margin: 0cm; font-size: 11pt;=
 font-family: Calibri, sans-serif;" class=3D""><o:p class=3D"">&nbsp;</o:p>=
</div><div style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, san=
s-serif;" class=3D"">Turning xml like this:<o:p class=3D""></o:p></div><div=
 style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" =
class=3D""><o:p class=3D"">&nbsp;</o:p></div><div style=3D"margin: 0cm; fon=
t-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">&lt;textlines&g=
t;this is line 1<o:p class=3D""></o:p></div><div style=3D"margin: 0cm; font=
-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">this is line 2<o=
:p class=3D""></o:p></div><div style=3D"margin: 0cm; font-size: 11pt; font-=
family: Calibri, sans-serif;" class=3D"">this is line 3&lt;/textlines&gt;<o=
:p class=3D""></o:p></div><div style=3D"margin: 0cm; font-size: 11pt; font-=
family: Calibri, sans-serif;" class=3D""><o:p class=3D"">&nbsp;</o:p></div>=
<div style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, sans-seri=
f;" class=3D"">into something like this:<o:p class=3D""></o:p></div><div st=
yle=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" cla=
ss=3D""><o:p class=3D"">&nbsp;</o:p></div><div style=3D"margin: 0cm; font-s=
ize: 11pt; font-family: Calibri, sans-serif;" class=3D"">&lt;textlines&gt;<=
o:p class=3D""></o:p></div><div style=3D"margin: 0cm; font-size: 11pt; font=
-family: Calibri, sans-serif;" class=3D"">&lt;line&gt;this is line 1&lt;/li=
ne&gt;<o:p class=3D""></o:p></div><div style=3D"margin: 0cm; font-size: 11p=
t; font-family: Calibri, sans-serif;" class=3D"">&lt;line&gt;this is line 2=
&lt;/line&gt;<o:p class=3D""></o:p></div><div style=3D"margin: 0cm; font-si=
ze: 11pt; font-family: Calibri, sans-serif;" class=3D"">&lt;line&gt;this is=
 line 3&lt;/line&gt;<o:p class=3D""></o:p></div><div style=3D"margin: 0cm; =
font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">&lt;/textlin=
es&gt;<o:p class=3D""></o:p></div><div style=3D"margin: 0cm; font-size: 11p=
t; font-family: Calibri, sans-serif;" class=3D""><o:p class=3D"">&nbsp;</o:=
p></div><div style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, s=
ans-serif;" class=3D"">is simple and I have functions that do it. But they =
rely on &lt;textlines&gt; being an xs:string, which is processed by a trivi=
al function I wrote:<o:p class=3D""></o:p></div><div style=3D"margin: 0cm; =
font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><o:p class=
=3D"">&nbsp;</o:p></div><div style=3D"margin: 0cm; font-size: 11pt; font-fa=
mily: Calibri, sans-serif;" class=3D"">&lt;xsl:function name=3D"my:intoLine=
s" as=3D"xs:string*"&gt;<o:p class=3D""></o:p></div><div style=3D"margin: 0=
cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">&nbsp; &=
lt;xsl:param name=3D"arg" as=3D"xs:string?" /&gt;<o:p class=3D""></o:p></di=
v><div style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, sans-se=
rif;" class=3D"">&nbsp; &lt;xsl:sequence select=3D"tokenize( $arg, '(\r\n?|=
\n\r?)' )" /&gt;<o:p class=3D""></o:p></div><div style=3D"margin: 0cm; font=
-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">&lt;/xsl:functio=
n&gt;<o:p class=3D""></o:p></div><div style=3D"margin: 0cm; font-size: 11pt=
; font-family: Calibri, sans-serif;" class=3D""><o:p class=3D"">&nbsp;</o:p=
></div><div style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, sa=
ns-serif;" class=3D"">I would like to extend this so that it can handle an =
element consisting of text lines with some embedded markup (there won't be =
much, but there will be some). For example, taking:<o:p class=3D""></o:p></=
div><div style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, sans-=
serif;" class=3D""><o:p class=3D"">&nbsp;</o:p></div><div style=3D"margin: =
0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">&lt;tex=
tlines&gt;this is line &lt;seq&gt;1&lt;/seq&gt;<o:p class=3D""></o:p></div>=
<div style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, sans-seri=
f;" class=3D"">this is &lt;var&gt;line&lt;/var&gt; 2<o:p class=3D""></o:p><=
/div><div style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, sans=
-serif;" class=3D"">this &lt;emph&gt;is&lt;/emph&gt; line 3&lt;/textlines&g=
t;<o:p class=3D""></o:p></div><div style=3D"margin: 0cm; font-size: 11pt; f=
ont-family: Calibri, sans-serif;" class=3D""><o:p class=3D"">&nbsp;</o:p></=
div><div style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, sans-=
serif;" class=3D"">and producing<o:p class=3D""></o:p></div><div style=3D"m=
argin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">=
<o:p class=3D"">&nbsp;</o:p></div><div style=3D"margin: 0cm; font-size: 11p=
t; font-family: Calibri, sans-serif;" class=3D"">&lt;textlines&gt;<o:p clas=
s=3D""></o:p></div><div style=3D"margin: 0cm; font-size: 11pt; font-family:=
 Calibri, sans-serif;" class=3D"">&lt;line&gt;this is line &lt;seq&gt;1&lt;=
/seq&gt;&lt;/line&gt;<o:p class=3D""></o:p></div><div style=3D"margin: 0cm;=
 font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">&lt;line&gt=
;this is &lt;var&gt;line&lt;/var&gt; 2&lt;/line&gt;<o:p class=3D""></o:p></=
div><div style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, sans-=
serif;" class=3D"">&lt;line&gt;this &lt;emph&gt;is&lt;/emph&gt; line 3&lt;/=
line&gt;<o:p class=3D""></o:p></div><div style=3D"margin: 0cm; font-size: 1=
1pt; font-family: Calibri, sans-serif;" class=3D"">&lt;/textlines&gt;<o:p c=
lass=3D""></o:p></div><div style=3D"margin: 0cm; font-size: 11pt; font-fami=
ly: Calibri, sans-serif;" class=3D""><o:p class=3D"">&nbsp;</o:p></div><div=
 style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" =
class=3D"">Sequence and tokenize isn't adequate but is there a way to do th=
is that won't overtax my brain too much?<o:p class=3D""></o:p></div><div st=
yle=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" cla=
ss=3D""><o:p class=3D"">&nbsp;</o:p></div><div style=3D"margin: 0cm; font-s=
ize: 11pt; font-family: Calibri, sans-serif;" class=3D"">cheers<o:p class=
=3D""></o:p></div><div style=3D"margin: 0cm; font-size: 11pt; font-family: =
Calibri, sans-serif;" class=3D"">T<o:p class=3D""></o:p></div></div><div st=
yle=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; =
font-style: normal; font-variant-caps: normal; font-weight: normal; letter-=
spacing: normal; text-align: start; text-indent: 0px; text-transform: none;=
 white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; te=
xt-decoration: none;" class=3D""><div style=3D"border-top-width: 1px; borde=
r-top-style: solid; border-top-color: black; background-color: rgb(221, 221=
, 221); color: rgb(136, 136, 136); font-size: smaller; padding: 5px; text-a=
lign: center; font-family: arial, verdana, arial, sans-serif; clear: both; =
margin: auto;" class=3D""><a href=3D"http://www.mulberrytech.com/xsl/xsl-li=
st" class=3D"">XSL-List info and archive</a><div style=3D"text-align: cente=
r;" class=3D""><a href=3D"http://lists.mulberrytech.com/unsub/xsl-list/2935=
09" style=3D"color: blue;" class=3D"">EasyUnsubscribe</a><span class=3D"App=
le-converted-space">&nbsp;</span>(<a href=3D"" style=3D"color: blue;" class=
=3D"">by email</a>)</div></div></div></div></blockquote></div><br class=3D"=
"></div></body></html>
<div><!-- begin bl.html.trailer -->
<div style=3D"border-top:1px solid black; background-color: #dddddd;
color: #888888; font-size: smaller; padding: 5px; text-align: center;
font-family: arial,verdana,arial,sans-serif; margin-top:1em; clear:
both; margin: auto">
<a href=3D"http://www.mulberrytech.com/xsl/xsl-list">
XSL-List info and archive</a>
<div style=3D"text-align:center;">
<a style=3D"color: blue;"
  href=3D"http://lists.mulberrytech.com/unsub/xsl-list/3329386"
>EasyUnsubscribe</a>
(<a style=3D"color: blue;"
href=3D"mailto:[email protected]?subject=3Dremove"
>by email</a>)
</div>
</div>
<!-- end bl.html.trailer --></div>

--Apple-Mail=_8C9FD249-B05E-484D-B843-34FF2D7D29AA--