Re: Speed of processing man pages

Steve Cheng <[email protected]> writes:

> The goal is to process from 1000 to 10000 typical refentry
> documents as quickly as possible. (In my case, these documents
> come from using Eric S.  Raymond's doclifter on all the man
> pages installed on my system.)
> 
> XSLT processor is libxslt
> Man-XML translator is the Perl-based one
> 
> To convert 187 pages:
> 
> xsltproc takes approx 41 seconds
> db2x_manxml takes approx 26.5 seconds without charset conversion

One minute to process 187 man pages does not seem at all
excessive to me. Compare it (just for kicks) to, say, compiling
and linking a C application that has as many lines of code as
there are lines in the source for thos 187 pages. I reckon it
might take a lot longer than one minute to build that much C code.

> (Haven't tested with charset conversion yet, however it does
> seem to be much faster than the first two stages.)

By "charset conversion" do you really mean charset conversion, or
so you mean the Unicode-to-roff character translations that
DocBook2X does?

If you mean the Unicode-to-roff character translations, I would
reckon that adding that will slow things down significantly.

Back in June of last year, inspired by the existing character-
translatin support in DocBook2X, I added support in the DocBook
project manpages stylesheet for user-configurable Unicode-to-roff
character translations. It relies on use of a map file that is
very much like the one you came up with for DocBook2X. The
current version of it is here:

  http://cvs.sourceforge.net/viewcvs.py/*checkout*/docbook/xsl/manpages/charmap.groff.xsl

The full map includes translations for about 800 characters. By
default, the manpages stylesheet only uses a subset of the full
map. But even using just a subset, I've found (through checking
with the --profile in xsltproc) that total time for all calls to
the XSLT template that performs the character translation far
exceeds the total time needed for any other single template.
Doing string substitutions in XSLT 1.0 is just not very efficient.

> This is with obvious optimizations such as only loading the XSLT
> stylesheet once for the 187 documents, and also loading
> db2x_manxml once only.

Another optimization that makes a significant difference, at least
in the case of processing with the DocBook project manpages
stylesheet, is combining all the source files together into a
single XML document prior to running the XSLT process on that
source. You could have DocBook2X do that quickly and efficiently
using a simple shell script or of course using Perl.

> So about .36 seconds to process one page.

I would take pride in that figure if I were you.

> Or one hour to process 10000 pages. :-(

That's at all excessive as far as I am concerned. Again, for the
sake of comparison, consider how long it might take to compile a
C application or set of applications from 10,000 separate source
files. I think it might take a lot longer than one hour.

> It would also be helpful to have some comparison with other
> DocBook-to-man solutions.

It is probably significantly faster than what the DocBook project
manpages stylesheet might be able to do with the same document
collection. Perhaps as much as 10 times or more faster.

But it is not really useful to consider performance of man-page
conversion in isolation. I think if you also need to compare it
to performance of other conversion for other output formats,
primarily compare it to performance for HTML and FO transformation.

I have found that converting a source file to man output using
the DocBook project manpages stylesheet requires roughly as much
processing time as converting that same source to "chunked" HTML
output using the chunk stylesheet in the DocBook XSL distro.

So the manpages stylesheet, slow though it may be, really isn't
very slow at all relative to processing for other output formats.

As far as other DocBook-to-man solutions, I think it might be
unlikely that you will be able to achieve processing times as
small as those of Fred Dalrymple's instant(1)-based
docbook-to-man converter.

But I personally wouldn't bother taking any time to compare
anything to that. It's true that some people still use it, but
output from it is just worthless unless you are willing put all
kinds of workarounds in your source to deal with the many
deficiencies that docbook-to-man has. (For one thing, it requires
you to keep the source for all your refentry instances in separate
files, because it's not capable of bursting or chunking out
separate man pages from a source file that contains multiple
refentry instances).

To be fair to it, I would say that the deficiencies are due to
the fact that it's basically just ancient (it was written almost
10 years ago and really has not been updated significantly
since). Now that there solutions available that produce vastly
superior output, Dalrymple's docbook-to-man is only useful for
people who want to generate man-output really, really quickly and
are willing to make the really, really big tradeoff of hobbling
their source markup so as to be able to produce worthwhile output
from docbook-to-man.

> Anyone like to do these? No coding required!

I would like to test with your document collection and see what I
get. Could you tar up the whole set of source files you are
currently testing with, and put them online somewhere so that I
can download them?

If you can do that, I can give you some data on how long it takes
to process that same document collection using the the DocBook
manpages stylesheet.

  --Mike