Re: Speed of processing man pages

Steve Cheng <[email protected]> writes:

> Also, there are situations where waiting is just not possible.
> For example, IIRC Solaris man pages are generated from SGML
> DocBook sources on-the-fly.

They were still several years ago, but they are not now. The
Solaris doc team is now authoring all their man pages in nroff,
and has been doing so for a while now.

Anyway, setting up a system for on-the-fly generation of outbook
from DocBook sources was not a particularly good idea. The process
of taking a document of some arbitrary length that is completely
free from any presentational markup and turning it into a
rendering format on the fly is not something that anyone should
reasonably expect to be able to do.

> The .36s processing time for one man page is already over the
> limit considering that people expect near-instantaneous response
> from the man command, and other GUI programs (e.g. <0.2s latency
> on getting a response after clicking a button).

Right. And that's the *average* time you have managed to achieve
right? That means you have DocBook sources in your collection
that probably take significantly longer than that to process.

If your goal is to come up with a system that does DocBook-to-man
transformation on the fly and provides a near-instantaneous
response from the man command, I don't think you are setting a
reasonable goal for yourself.

> Another kind of XML processing I do is literate programming,
> with edit-compile-test cycles. I can't afford .36s * 10 added to
> my building time.

Is .36s * 10 really such an unreasonable amount of time to add to
your build in order to generate docs for your application?

And is it really that slow relative to other alternatives? Do you
know, for example, How fast the doc part of your build would take
if you embedded comments in some other way, and used Doc++ or
DocGen or whatever to build the docs?

Anyway, you do of course need to rebuild the docs each time you
make a code change (that is, a change to your C or whatever code
as opposed to a change to the embedded doc). You can set up your
build process with a "distrib" or whatever target such that you
build only when you are packaging the distribution version of
your application, or only build when you want to test doc
changes. A build setup the rebuilds the doc every time a code
change is made is not a good build setup -- regardless of the
system you are using for building your docs.

> But the C compiler is typically loaded once for each file (so
> the comparison isn't fair), and C compilers analyze the source
> heavily; they do a lot more than DocBook to man page converters.

Which is not to say that good DocBook-to-man converters aren't
doing a lot of stuff themselves. I think I would personally say
that any converter is doing quite a lot when it takes a large,
complex DocBook document -- which is completely free from any
presentational markup -- and transforms it into a rendering format
(be is XSL-FO, HTML, groff, or whatever) that is primarily
presentational.

> I mean, man pages are just 5K text files.

If that's your assumption about what a man page is, I think you
might need to adjust your assumption.

The ethereal-filter(4) man page, for example, is 4.2 megabytes.

If you consider that obscure, here a examples for some more
widely used man pages:

perlfunc man page: 333K
gcc-4.0.1 man page: 515K

We could go on and on. The size of the man file simply depends
on what application you're documenting, and how extensively is it
documented in source the man page is generated from.

How reasonable would it be to expect to be able to convert the
source for the 333K perfunc man page on the fly, and get a
near-instantaneous response when doing it?

> > The full map includes translations for about 800 characters. By
> > default, the manpages stylesheet only uses a subset of the full
> > map. But even using just a subset, I've found (through checking
> 
> Btw, why is it necessary to use a small subset?
> If you've got xsl:key, the performance shouldn't depend on the size
> of the character map.

I'm not using xsl:key. I'm using what is basically an XSLT 2.0
character map (as defined in the XSLT 2.0 spec) but using it for
XSLT 1.0 processing. I think I considered xsl:key but ended up not
using because there was something that prevented it from being
usable with the design. I don't remember now.

> It's also sad that a C-based thing (libxslt) can be slower than
> a Perl script, operating on XML documents of about the same size
> (refentry documents and Man-XML documents).

Are you comparing XSLT processing in C (libxslt) to XSLT
processing in Perl? Or are you comparing libxslt to some non-XSLT
procssing in Perl. If you're comparing XSLT processing to non-XSLT
processing, then you're not comparing a C-based thing to a Perl
script. You're instead comparing two completely different things.

There are always going to be faster alternatives that XPath/XSLT.
Some of them are fine alternatives, especially if you're willing
to trade off flexibility and power for speed, and as long as you
are willing to build your application in such as way that it
requires end users to install additional dependencies (other than
just an XSLT engine) in order to use it.

> Well, the slowness of HTML and FO is even more crippling,
> because HTML and FO are used even more often than man pages.
> (one reason why I stopped using DocBook for my latest literate
> programming project)
> 
> > So the manpages stylesheet, slow though it may be, really isn't
> > very slow at all relative to processing for other output formats.
> 
> Well, I want to raise the standard :)

Good luck with that. I personally think that there are many better
uses to which you could put your energy and skills than in trying
to creating the world's fastest DocBook-to-man (or DocBook to
HTML, or DocBook to FO transformation system). Even if you manage
to achieve whatever performance goals you have in mind, the fact
is that most users are simply not going to care, because most do
not consider their current build times to be excessive.

> > I would like to test with your document collection and see what I
> > get. Could you tar up the whole set of source files you are
> > currently testing with, and put them online somewhere so that I
> > can download them?
> 
> Sure.
> http://docbook2x.sourceforge.net/man7.tar.gz

Thanks. I will download it and try it out soon.

  --Mike