Re: Speed of processing man pages

"Michael(tm) Smith" <[email protected]> Wed, 19 Apr 2006 13:06:29 +0900

Newsgroups	gmane.text.docbook.docbook2x.general
Message-ID	<[email protected]>

Steve Cheng <[email protected]> writes:

> To convert 187 pages:
> 
> xsltproc takes approx 41 seconds
> db2x_manxml takes approx 26.5 seconds without charset conversion
> (Haven't tested with charset conversion yet, however it does seem to
> be much faster
> than the first two stages.)
> 
> This is with obvious optimizations such as only loading the XSLT stylesheet once
> for the 187 documents, and also loading db2x_manxml once only.
> 
> So about .36 seconds to process one page.
> Or one hour to process 10000 pages. :-(
> 
> It would also be helpful to have some comparison with other
> DocBook-to-man solutions.

FWIW, here's output from time(1) for a run through the DocBook
project manpages stylesheets of all 187 refentry instances.

  real    1m17.942s

That's on a ThinkPad T42p with a 2GHz CPU, and using the attached
makefile.

Note that one of the things the makefile does is to first create a
number of "wrapper" files that combine the refentry instances into
groups of 20. Then it runs the transformation on those wrapper
files instead of the individual source files. The output is
exactly the same, but it's significantly faster.[1]

For the sake of comparison, here's time output for a run through
the HTML chunk stylesheet on the same set of files:

  real    0m59.359s

So the manpages stylesheet is measurably slower than the chunk
stylesheet, but not way far behind.

You can change the number of refentry instances per group by
setting CHUNKSIZE:

  make CHUNKSIZE=10

Here's output from time for a manpages run if I set that (10
refentry instances per file instead of 20):

  real    1m35.519s

And for chunked HTML:

  real    1m19.731s

To show how much of a difference it makes if the refentry
instances are not groupr, here's time output for a manpages run
with just one file per refentry:

  real    7m14.579s

And for (non-chunked) HTML with one file per refentry:

  real    6m32.523s

So it's quite slow doing it that way, but not too surprising,
since it's reloading the stylesheet and invoking xsltproc 187
separate times.

  --Mike

[1] Running the transformation against files that contain sets of
refentry instances (instead of one-file-per-refentry) also more
closely models the way that most users have their source files
organized. Most users don't maintain a separate source file for
each refentry -- especially if they are also generating print/PDF
and HTML output.  In my experiences, users typically maintain sets
of refentry instances instead -- or at least use XInclude or some
other mechanism to to group them before generating output.

Makefile (text/plain, 5.5 KB)

FILES=$(shell find . -maxdepth 1 -name "*.xml" | cut -c3-)

MAN_XSL = http://docbook.sourceforge.net/snapshots/xsl/tools/make/Makefile.DocBook

WGET=wget
WGET_FLAGS=

XSLTPROC=xsltproc
XSLTPROC_FLAGS=

SED=sed
SED_FLAGS=-i

CHUNKSIZE ?= 20

COMBINE_XSL = <?xml version="1.0"?> \
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" \
                xmlns:exsl="http://exslt.org/common" \
                xmlns:xi="http://www.w3.org/2001/XInclude" \
                exclude-result-prefixes="exsl xi" \
                extension-element-prefixes="exsl" \
                version="1.0"> \
  <xsl:param name="files"/> \
  <xsl:param name="chunk.size"/> \
  <xsl:template match="/"> \
    <xsl:call-template name="make.file"/> \
  </xsl:template> \
 \
  <xsl:template name="make.file"> \
    <xsl:param name="count" select="1"/> \
    <xsl:param name="current.files" select="concat(normalize-space($$files), ^^ ^^)"/> \
    <xsl:param name="more.files" \
               select="concat(normalize-space(substring-after($$current.files, ^^ ^^)),^^ ^^)"/> \
    <xsl:param name="file.number" select="1"/> \
    <xsl:param name="filename" select="concat(^^./build/^^,$$file.number,^^.xml^^)"/> \
 \
    <xsl:choose> \
      <xsl:when test="$$more.files = ^^ ^^"/> \
      <xsl:when test="$$count mod $$chunk.size = 0"> \
        <xsl:variable name="fileset" select="concat($$current.files, ^^ ^^, \
          substring-before($$more.files, ^^ ^^))"/> \
        <exsl:document href="{$$filename}" \
                       method="xml" \
                       encoding="UTF-8" \
                       indent="yes" \
                       omit-xml-declaration="yes" \
                       media-type="" \
                       standalone="no"> \
          <reference> \
            <title>foo</title> \
            <xsl:call-template name="make.xinclude"> \
              <xsl:with-param name="file"> \
                <xsl:choose> \
                  <xsl:when test="contains($$fileset, ^^ ^^)"> \
                    <xsl:value-of \
                        select="normalize-space(substring-before($$fileset, ^^ ^^))"/> \
                  </xsl:when> \
                  <xsl:otherwise> \
                    <xsl:value-of select="$$fileset"/> \
                  </xsl:otherwise> \
                </xsl:choose> \
              </xsl:with-param> \
              <xsl:with-param \
                  name="remaining.files" \
                  select="concat(normalize-space(substring-after($$fileset, ^^ ^^)),^^ ^^)"/> \
            </xsl:call-template> \
          </reference> \
        </exsl:document> \
        <xsl:call-template name="make.file"> \
          <xsl:with-param name="count" select="1"/> \
          <xsl:with-param name="current.files"  \
                          select="$$more.files"/> \
          <xsl:with-param name="file.number" select="number($$file.number) + 1"/> \
          <xsl:with-param name="filename" select="concat(^^./build/^^,$$file.number,^^.xml^^)"/> \
        </xsl:call-template> \
      </xsl:when> \
      <xsl:otherwise> \
        <xsl:call-template name="make.file"> \
          <xsl:with-param name="count" select="$$count + 1"/> \
          <xsl:with-param name="current.files"> \
            <xsl:choose> \
              <xsl:when test="$$count = 1 and $$file.number = 1"> \
                <xsl:value-of  \
                    select="concat(substring-before($$current.files, ^^ ^^), \
                            ^^ ^^, \
                            substring-before($$more.files, ^^ ^^))"/> \
              </xsl:when> \
              <xsl:when test="$$count = 1"> \
                <xsl:value-of  \
                    select="substring-before($$more.files, ^^ ^^)"/> \
              </xsl:when> \
              <xsl:otherwise> \
                <xsl:value-of  \
                    select="concat($$current.files, ^^ ^^, \
                            substring-before($$more.files, ^^ ^^))"/> \
              </xsl:otherwise> \
            </xsl:choose> \
          </xsl:with-param> \
          <xsl:with-param name="more.files" \
                          select="substring-after($$more.files, ^^ ^^)"/> \
          <xsl:with-param name="file.number" select="$$file.number"/> \
        </xsl:call-template> \
      </xsl:otherwise> \
    </xsl:choose> \
  </xsl:template> \
 \
  <xsl:template name="make.xinclude"> \
    <xsl:param name="file"/> \
    <xsl:param name="remaining.files"/> \
    <xsl:param name="count" select="1"/> \
    <xsl:if test="not($$file = ^^^^)"> \
      <xi:include href="../{$$file}"/> \
      <xsl:call-template name="make.xinclude"> \
        <xsl:with-param \
            name="file" \
            select="substring-before($$remaining.files, ^^ ^^)"/> \
        <xsl:with-param \
            name="remaining.files" \
            select="substring-after($$remaining.files, ^^ ^^)"/> \
        <xsl:with-param name="count" select="$$count + 1"/> \
      </xsl:call-template> \
    </xsl:if> \
  </xsl:template> \
 \
</xsl:stylesheet>

all: man

man: build/man

build/Makefile:
	if [ ! -d build ]; then mkdir build; fi
	$(WGET) $(WGET_FLAGS) -O $@ $(MAN_XSL)

combine.xsl: Makefile
	@echo '$(COMBINE_XSL)' > $@
	$(SED) $(SED_FLAGS) "s/\^\^/'/g" $@

build/1.xml: combine.xsl
	$(XSLTPROC) $(XSLTPROC_FLAGS) \
	--stringparam files "$(FILES)" \
	--stringparam chunk.size $(CHUNKSIZE) \
	$< $<

build/man: build/Makefile build/1.xml
	time $(MAKE) -C build man \
		MAN_PARAMS="--stringparam man.output.quietly 1 \
		   --stringparam refentry.meta.get.quietly 1 \
		   --stringparam man.charmap.enabled 0"

debug:

clean:
	$(RM) -r build