Re: Speed of processing man pages

Before I begin, I want to say much of my concern is of a more general
/ philosophical nature, rather than just generating 10000 man pages.
It is frustrating to be constantly waiting for computers with 1.7GHz
CPUs.  I would much prefer not to exacerbate the problem with my own
software. If I fix a bug in docbook2X and re-run a regression test to
check, I want to wait as short a time as possible for the results.

Also, there are situations where waiting is just not possible.
For example, IIRC Solaris man pages are generated from SGML DocBook sources
on-the-fly.  The .36s processing time for one man page is already over the limit
considering that people expect near-instantaneous response from the man command,
and other GUI programs (e.g. <0.2s latency on getting a response after
clicking a button).

Another kind of XML processing I do is literate programming, with
edit-compile-test
cycles. I can't afford .36s * 10 added to my building time.

> One minute to process 187 man pages does not seem at all
> excessive to me. Compare it (just for kicks) to, say, compiling
> and linking a C application that has as many lines of code as
> there are lines in the source for thos 187 pages. I reckon it
> might take a lot longer than one minute to build that much C code.

But the C compiler is typically loaded once for each file (so the
comparison isn't fair),
and C compilers analyze the source heavily; they do a lot more than
DocBook to man page converters.  I mean, man pages are just 5K text
files.
Besides, I hear the tcc compiler (alternative to gcc) is very fast.

> > (Haven't tested with charset conversion yet, however it does
> > seem to be much faster than the first two stages.)
>
> By "charset conversion" do you really mean charset conversion, or
> so you mean the Unicode-to-roff character translations that
> DocBook2X does?

Both.

> If you mean the Unicode-to-roff character translations, I would
> reckon that adding that will slow things down significantly.

Well, utf8trans and iconv are blazingly fast. 0.330s on 187 man pages.
Yes, that's with the 800 or so character translations from you :)

> The full map includes translations for about 800 characters. By
> default, the manpages stylesheet only uses a subset of the full
> map. But even using just a subset, I've found (through checking

Btw, why is it necessary to use a small subset?
If you've got xsl:key, the performance shouldn't depend on the size
of the character map.

> Another optimization that makes a significant difference, at least
> in the case of processing with the DocBook project manpages
> stylesheet, is combining all the source files together into a
> single XML document prior to running the XSLT process on that
> source. You could have DocBook2X do that quickly and efficiently
> using a simple shell script or of course using Perl.
>
Interesting. Though I'm not in favor of this approach because it seems
wasteful to load a 5MB XML document
(who knows how big it would expand to in memory)
just to process 10K chunks at a time.

It's also sad that a C-based thing (libxslt) can be slower than a Perl script,
operating on XML documents of about the same size (refentry documents
and Man-XML documents).

> But it is not really useful to consider performance of man-page
> conversion in isolation. I think if you also need to compare it
> to performance of other conversion for other output formats,
> primarily compare it to performance for HTML and FO transformation.

Well, the slowness of HTML and FO is even more crippling,
because HTML and FO are used even more often than man pages.
(one reason why I stopped using DocBook for my latest literate
programming project)

> So the manpages stylesheet, slow though it may be, really isn't
> very slow at all relative to processing for other output formats.

Well, I want to raise the standard :)

>
> As far as other DocBook-to-man solutions, I think it might be
> unlikely that you will be able to achieve processing times as
> small as those of Fred Dalrymple's instant(1)-based
> docbook-to-man converter.

STX (Streaming Transformations for XML, http://stx.sourceforge.net/)
looks promising.
Yes, it's not tree-based, but I wonder if the built-in support
buffering would be enough/
One day I might just get so curious to write a C-based implementation
and see how fast it could get.

>
> I would like to test with your document collection and see what I
> get. Could you tar up the whole set of source files you are
> currently testing with, and put them online somewhere so that I
> can download them?

Sure.
http://docbook2x.sourceforge.net/man7.tar.gz

Thanks for your help and criticism!
--
Steve Cheng
&#x912D;&#x541B;&#x535A;        http://gold-saucer.afraid.org


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642