Re: Seeking a smarter tokenize for augmented text

"Geert Bormans [email protected]" <[email protected]> Thu, 6 May 2021 09:25:22 -0000
Newsgroups gmane.text.xml.xsl.general.mulberrytech
Message-ID <[email protected]>
--=_9163a362-1dd4-4940-bfa4-5f37a1880922
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

I would have a two-step 

transform the new-line into an empty element <newline/> in a first pass (identity transform with an analyze-string in the text nodes) 
and do a for-each-group on all nodes, starting on the newline element in the second pass 

you could do the in a single step, for each group on all non text nodes + all text nodes, but split (when they have a newline in them) 

Met vriendelijke groeten, 
Best regards, 

Geert Bormans 

----- Op 6 mei 2021 om 11:16 schreef Trevor Nicholls [email protected] <[email protected]>: 



Hi 



Turning xml like this: 



<textlines>this is line 1 

this is line 2 

this is line 3</textlines> 



into something like this: 



<textlines> 

<line>this is line 1</line> 

<line>this is line 2</line> 

<line>this is line 3</line> 

</textlines> 



is simple and I have functions that do it. But they rely on <textlines> being an xs:string, which is processed by a trivial function I wrote: 



<xsl:function name="my:intoLines" as="xs:string*"> 

<xsl:param name="arg" as="xs:string?" /> 

<xsl:sequence select="tokenize( $arg, '(\r\n?|\n\r?)' )" /> 

</xsl:function> 



I would like to extend this so that it can handle an element consisting of text lines with some embedded markup (there won't be much, but there will be some). For example, taking: 



<textlines>this is line <seq>1</seq> 

this is <var>line</var> 2 

this <emph>is</emph> line 3</textlines> 



and producing 



<textlines> 

<line>this is line <seq>1</seq></line> 

<line>this is <var>line</var> 2</line> 

<line>this <emph>is</emph> line 3</line> 

</textlines> 



Sequence and tokenize isn't adequate but is there a way to do this that won't overtax my brain too much? 



cheers 

T 
[ http://www.mulberrytech.com/xsl/xsl-list | 
XSL-List info and archive ] 
[ http://lists.mulberrytech.com/unsub/xsl-list/554170 | EasyUnsubscribe ] ( [  | by email ] ) 
--~----------------------------------------------------------------
XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list
EasyUnsubscribe: http://lists.mulberrytech.com/unsub/xsl-list/3329386
or by email: [email protected]
--~--

--=_9163a362-1dd4-4940-bfa4-5f37a1880922
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><body><div style=3D"font-family: Arial; font-size: 10pt; color: #0000=
00"><div>I would have a two-step</div><div><br data-mce-bogus=3D"1"></div><=
div>transform the new-line into an empty element &lt;newline/&gt; in a firs=
t pass (identity transform with an analyze-string in the text nodes)</div><=
div>and do a for-each-group on all nodes, starting on the newline element i=
n the second pass</div><div><br data-mce-bogus=3D"1"></div><div>you could d=
o the in a single step, for each group on all non text nodes + all text nod=
es, but split (when they have a newline in them)</div><div><br></div><div d=
ata-marker=3D"__SIG_PRE__"><div>Met vriendelijke groeten,<br>Best regards,<=
br><br>Geert Bormans<br></div></div><div><br><span id=3D"zwchr" data-marker=
=3D"__DIVIDER__">----- Op 6 mei 2021 om 11:16 schreef Trevor Nicholls trevo=
[email protected] &lt;[email protected]&gt;:<br></=
span><br><style>/*<![CDATA[*/p.MsoNormal, li.MsoNormal, div.MsoNormal {
	margin: 0.0cm;
	font-size: 11.0pt;
	font-family: Calibri , sans-serif;
}
span.EmailStyle17 {
	font-family: Calibri , sans-serif;
	color: windowtext;
}
*.MsoChpDefault {
	font-family: Calibri , sans-serif;
}
div.WordSection1 {
	page: WordSection1;
}
/*]]>*/</style></div><div data-marker=3D"__QUOTED_TEXT__"><div class=3D"Wor=
dSection1"><p class=3D"MsoNormal">Hi</p><p class=3D"MsoNormal">&nbsp;</p><p=
 class=3D"MsoNormal">Turning xml like this:</p><p class=3D"MsoNormal">&nbsp=
;</p><p class=3D"MsoNormal">&lt;textlines&gt;this is line 1</p><p class=3D"=
MsoNormal">this is line 2</p><p class=3D"MsoNormal">this is line 3&lt;/text=
lines&gt;</p><p class=3D"MsoNormal">&nbsp;</p><p class=3D"MsoNormal">into s=
omething like this:</p><p class=3D"MsoNormal">&nbsp;</p><p class=3D"MsoNorm=
al">&lt;textlines&gt;</p><p class=3D"MsoNormal">&lt;line&gt;this is line 1&=
lt;/line&gt;</p><p class=3D"MsoNormal">&lt;line&gt;this is line 2&lt;/line&=
gt;</p><p class=3D"MsoNormal">&lt;line&gt;this is line 3&lt;/line&gt;</p><p=
 class=3D"MsoNormal">&lt;/textlines&gt;</p><p class=3D"MsoNormal">&nbsp;</p=
><p class=3D"MsoNormal">is simple and I have functions that do it. But they=
 rely on &lt;textlines&gt; being an xs:string, which is processed by a triv=
ial function I wrote:</p><p class=3D"MsoNormal">&nbsp;</p><p class=3D"MsoNo=
rmal">&lt;xsl:function name=3D"my:intoLines" as=3D"xs:string*"&gt;</p><p cl=
ass=3D"MsoNormal">&nbsp; &lt;xsl:param name=3D"arg" as=3D"xs:string?" /&gt;=
</p><p class=3D"MsoNormal">&nbsp; &lt;xsl:sequence select=3D"tokenize( $arg=
, '(\r\n?|\n\r?)' )" /&gt;</p><p class=3D"MsoNormal">&lt;/xsl:function&gt;<=
/p><p class=3D"MsoNormal">&nbsp;</p><p class=3D"MsoNormal">I would like to =
extend this so that it can handle an element consisting of text lines with =
some embedded markup (there won't be much, but there will be some). For exa=
mple, taking:</p><p class=3D"MsoNormal">&nbsp;</p><p class=3D"MsoNormal">&l=
t;textlines&gt;this is line &lt;seq&gt;1&lt;/seq&gt;</p><p class=3D"MsoNorm=
al">this is &lt;var&gt;line&lt;/var&gt; 2</p><p class=3D"MsoNormal">this &l=
t;emph&gt;is&lt;/emph&gt; line 3&lt;/textlines&gt;</p><p class=3D"MsoNormal=
">&nbsp;</p><p class=3D"MsoNormal">and producing</p><p class=3D"MsoNormal">=
&nbsp;</p><p class=3D"MsoNormal">&lt;textlines&gt;</p><p class=3D"MsoNormal=
">&lt;line&gt;this is line &lt;seq&gt;1&lt;/seq&gt;&lt;/line&gt;</p><p clas=
s=3D"MsoNormal">&lt;line&gt;this is &lt;var&gt;line&lt;/var&gt; 2&lt;/line&=
gt;</p><p class=3D"MsoNormal">&lt;line&gt;this &lt;emph&gt;is&lt;/emph&gt; =
line 3&lt;/line&gt;</p><p class=3D"MsoNormal">&lt;/textlines&gt;</p><p clas=
s=3D"MsoNormal">&nbsp;</p><p class=3D"MsoNormal">Sequence and tokenize isn'=
t adequate but is there a way to do this that won't overtax my brain too mu=
ch?</p><p class=3D"MsoNormal">&nbsp;</p><p class=3D"MsoNormal">cheers</p><p=
 class=3D"MsoNormal">T</p></div>
<div>
<div style=3D"border-top:1px solid black;background-color:#dddddd;color:#88=
8888;font-size:smaller;padding:5px;text-align:center;font-family:'arial' , =
'verdana' , 'arial' , sans-serif;margin-top:1em;margin:auto">
<a href=3D"http://www.mulberrytech.com/xsl/xsl-list" target=3D"_blank" rel=
=3D"nofollow noopener noreferrer">
XSL-List info and archive</a>
<div style=3D"text-align:center">
<a style=3D"color:blue" href=3D"http://lists.mulberrytech.com/unsub/xsl-lis=
t/554170" target=3D"_blank" rel=3D"nofollow noopener noreferrer">EasyUnsubs=
cribe</a>
(<a style=3D"color:blue" href=3D"" target=3D"_blank" rel=3D"nofollow noopen=
er noreferrer">by email</a>)
</div>
</div>
</div><br></div></div></body></html>
<div><!-- begin bl.html.trailer -->
<div style=3D"border-top:1px solid black; background-color: #dddddd;
color: #888888; font-size: smaller; padding: 5px; text-align: center;
font-family: arial,verdana,arial,sans-serif; margin-top:1em; clear:
both; margin: auto">
<a href=3D"http://www.mulberrytech.com/xsl/xsl-list">
XSL-List info and archive</a>
<div style=3D"text-align:center;">
<a style=3D"color: blue;"
  href=3D"http://lists.mulberrytech.com/unsub/xsl-list/3329386"
>EasyUnsubscribe</a>
(<a style=3D"color: blue;"
href=3D"mailto:[email protected]?subject=3Dremove"
>by email</a>)
</div>
</div>
<!-- end bl.html.trailer --></div>

--=_9163a362-1dd4-4940-bfa4-5f37a1880922--