Re: Seeking a smarter tokenize for augmented text

I have made some progress on this, not to a working point yet but I'm more =
confident than I was, so thanks to all for the suggestions which have been =
helpful. I also found some hints in a stackoverflow answer of Martin Honnen=
's which reinforced the advice to work on this by adding a line marker elem=
ent and using grouping.=20

The original statement of the requirement was a bit vague, and the content =
model currently in use is a bit too flexible. So I think I can stipulate th=
at inline elements will not run across line breaks (and if they do I should=
 be able to run a pre-fix which splits them), nor will the content include =
any nested inline elements.

At the moment I'm assuming that in the step where I insert line marker elem=
ents, I also have to use modal templates to insert inline element markers, =
then run another pass to restore the inline elements. Something like this, =
correct?

  <xsl:variable name=3D"brokenlines">
    <xsl:element name=3D"textlines">
      <xsl:element name=3D"linemarker"/>
      <xsl:analyze-string select=3D"." regex=3D"(\r\n?|\n\r?)">
        <xsl:matching-substring>
          <xsl:element name=3D"linemarker"/>
        </xsl:matching-substring>
        <xsl:non-matching-substring>
          <xsl:apply-templates mode=3D"break"/>
        </xsl:non-matching-substring>
      </xsl:analyze-string>
    <xsl:element>
  </xsl:variable>
  <xsl:variable name=3D"textlines">
    <xsl:call-template name=3D"rebuild">
      <xsl:with-param name=3D"lines" select=3D"$brokenlines"/>
    </xsl:call-template>
  <xsl:variable>
  <-- $textlines/textlines is now the original textlines with line children=
 -->
  ...

  <xsl:template match=3D"textlines/*/text()" mode=3D"break">
    <xsl:value-of select=3D"concat('[[{', name(..), '}', ., ']]')" />
  </xsl:template>

  <xsl:template name=3D"rebuild">
    <xsl:param name=3D"lines" as=3D"document-node()" />
    <xsl:element name=3D"textlines">
      <xsl:for-each select=3D"$lines/textlines">
        <xsl:for-each-group select=3D"node()" group-starting-with=3D"linema=
rker">
          <xsl:element name=3D"line">
            <xsl:apply-templates select=3D"current-group()[not(self::linema=
rker)]" mode=3D"rebuild" />
          </xsl:element>
        </xsl:for-each-group>
      </xsl:element>
    </xsl:template>

  <xsl:template match=3D"text()" mode=3D"rebuild">
    <xsl:analyze-string select=3D"." regex=3D"something matching [[{name}co=
ntent]]">
      <xsl:matching-substring>
        <xsl:element name=3D"the name in the regex">
          the content in the regex
        </xsl:element>
      </xsl:matching-substring>
      <xsl:non-matching-substring>
        <xsl-value-of select=3D"." />
      </xsl:non-matching-substring>
    </xsl:analyze-string>
  </xsl:template>

Am I going along the right lines? I'd prefer to be set straight sooner rath=
er than later!

Cheers
T

-----Original Message-----
From: Michael M=C3=BCller-Hillebrand [email protected] <xsl-list-service@lists.=
mulberrytech.com>=20
Sent: Friday, 7 May 2021 20:26
To: [email protected]
Subject: Re: [xsl] Seeking a smarter tokenize for augmented text

Hi,

=46rom Gerrit I would have expected something like

<xsl:for-each-group select=3D"node()" group-starting-with=3D"text()[matches=
(., '^&#xA;')]">
  ...=20=20=20=20
</xsl:for-each-group>

but it is possibly too much hassle to handle corner cases.=20

Because I assume it is very likely that inline elements may run across line=
 breaks, it is time for Gerrit=E2=80=99s "Upward Projection", see https://a=
rchive.xmlprague.cz/2019/files/xmlprague-2019-proceedings.pdf#page=3D347

- Michael MH


> Am 06.05.2021 um 11:14 schrieb Imsieke, Gerrit, le-tex gerrit.imsieke@le-=
tex.de <[email protected]>:
>=20
> Hi Trevor,
>=20
> If the markup in the lines doesn't stretch across line breaks, you can in=
sert empty elements for each line break using xsl:analyze-string in a first=
 pass.
>=20
> In the second pass, you then group the nodes below textlines, starting or=
 ending with such an empty linebreak element that you introduced in the fir=
st pass.
>=20
> Gerrit
>=20
> On 06.05.2021 11:09, Trevor Nicholls [email protected] wrote:
>> Hi
>> Turning xml like this:
>> <textlines>this is line 1
>> this is line 2
>> this is line 3</textlines>
>> into something like this:
>> <textlines>
>> <line>this is line 1</line>
>> <line>this is line 2</line>
>> <line>this is line 3</line>
>> </textlines>
>> is simple and I have functions that do it. But they rely on <textlines> =
being an xs:string, which is processed by a trivial function I wrote:
>> <xsl:function name=3D"my:intoLines" as=3D"xs:string*">
>>   <xsl:param name=3D"arg" as=3D"xs:string?" />
>>   <xsl:sequence select=3D"tokenize( $arg, '(\r\n?|\n\r?)' )" />=20
>> </xsl:function> I would like to extend this so that it can handle an=20
>> element consisting of text lines with some embedded markup (there won't =
be much, but there will be some). For example, taking:
>> <textlines>this is line <seq>1</seq>
>> this is <var>line</var> 2
>> this <emph>is</emph> line 3</textlines> and producing <textlines>=20
>> <line>this is line <seq>1</seq></line> <line>this is <var>line</var>=20
>> 2</line> <line>this <emph>is</emph> line 3</line> </textlines>=20
>> Sequence and tokenize isn't adequate but is there a way to do this that =
won't overtax my brain too much?
>> cheers
>> T
>=20
>=20
--~----------------------------------------------------------------
XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list
EasyUnsubscribe: http://lists.mulberrytech.com/unsub/xsl-list/3329386
or by email: [email protected]
--~--