Re: Autoconf for CRM114 fun and games

On Fri, Feb 13, 2009 at 12:14 AM, Nico Kadel-Garcia <[email protected]> wrote:
> Not quite: it doesn't show whether he's still tracking it privately,
> or interested in doing so. I'm finding his code much easier to follow
> than yours.

:-) Mine is just more of the same. And includes some features which
are not off the shelf for autoconf work (debug and release builds,
that sort of thing).


>> Had the intention to make it all support multiple regex libs when I
>> took up the autoconf work from Joost back in '07 -- have been using
>> multiple regex libs with other packages anyway -- but never got around
>> to it. TRE is very nice and still way faster than other stuff which
>> supports the mandatory subexpressions, such as pcre.
>
> Got it. I thought you had already done so, from its availability in
> the configure.ac, but I hadn't tested it. Integrating PCRE is an
> intriguing concept, but I'm afraid that due to the distinctions in the
> various regex.h files or their equivalents, it would be safer to
> continue to compile distinct binaries.

That stuff is/was a half-hearted copy&paste from another project I'm
working on (pavuk - also available on SF). Since pavuk already had all
this in place it was a 'what the heck, way easier to take that route'.
The wrapper code already exists, so 'all it takes' is making
configure.ac cope (copy&paste) plus make copies of the crmregex
wrapper code and jack in each of those regex engines).

So far the idea at the time. Then I took stock and guess what:
multiple regex lib support shot to the background at something close
to the speed of light.
crm114 had enough other trouble that was important to get fixed for my
intended use of the thing: the idea was to bring crm114 to a state
where the code could be 'librarized / daemonized' so I could have my
'crm114 daemon'. Alas, the big mistake I made was to not doublecheck
my own implicit decision to bring the script language along to such a
server: a plethora of global variable dependencies welcomed me, I got
to work and then I discovered I hadn't been smart enough to backpedal
and kick the whole bloody mess out the door, strip the classifiers to
their very core, take the idea of the cached mmap-ed CSS databases and
ditch 90% of the code. Enter the libcrm114 RFC by Bill, which kicked
me awake again: been doing the dumb thing all along.
crm114 is nice as a 'sample application' but once you start measuring
binary invocation overhead and such (check out the numbers; there's
performance test .crm files mentioning the data in ./tests/ ) and
ponder the additional drawbacks you get that way (the CSS cache is
essentially disabled, for one), there can be only one way forward when
you need to pump quite a bit of data through this kit: daemonize,
multithread CSS access and when that's not enough, go distributed
(server park).

As such, the 'public port', pardon, branch, was rather an accident,
due to my need for both 32- and 64-bit platforms (Win + UNIX) due to
the way the other bits were coming to fruition. Ah well, I'm in it
now. And by now it's at a stage where it indeed can serve as a sample
app for any libcrm114 to come.
A crm114d server is another example.



> I did notice a whole *set* of regexp variants: did you have any reason
> to consider anything other than TRE and PCRE when you were looking at
> this?

I know from my work on pavuk and my use of that tool in various
environments that several of those regex packages are on par with TRE,
at least with regards to their regex abilities. Yet I know TRE is
quite a bit faster than several of those, especially when going beyond
the easy regexes, so it's rather funny really: I use pavuk quite a
bit, and guess what: these days I always make sure I have TRE with it.
And pavuk had this multi-regex-lib support down pat from long before I
entered that project...

For the rest, see above. Twas a Nice Idea(tm). Execution stopped in
its tracks at less than 50% progress. And I'm glad it did.


> Or plain old filename.txt, which is what Bill uses, and leave it in
> /usr/share/doc/crm114/. I've personally lost much of my interest in
> writing documentation format of the week throughout my career, and
> find nroff to be unfortunately unstable among various platforms using
> antique SVR5 nroff, groff, or who knows what.

Check that.

Given the long-enduring XML hype ;-) (old idea, new execution), I
think DocBook will stay around as well, as it's XML (one less item to
explain to marketing) and is endorsed by a few big corps (not that
/that/ is any guarantee of longevity). And when it dies, I can always
XSL-transform the value in the documents to the winning format
(Microsoft Office 2K7? <evil grin>).


The whole thing is rather irrelevant, as the main requirement, at
least in my book, regarding documentation format, is that I can
transform it automagically to on-line and off-line document formats:
that is things like, indeed, man pages, but also HTML output for web
and PDF for printing. From a single source, whenever possible. Saves
me a lot of nauseating work.

That's why I moved the pavuk documentation to DocBook. And, yes,
there's a lot of hickups along that road. pavuk bears the brunt of
that as it's my trail project for DocBook.


In my younger days I'd have advocated TeX (or TexInfo), but it is
severely lacking on the image ability front (yeah, yeah, EPS embedding
is all the rage, I know). A real pity because I've never met a
typesetting quality that would match it (unless you count Quark
Express and its ilk as feasible [partial] solutions). And I don't
mention 'web publishing' as it ain't got any of that.

I've tried my hand at roff and groff and let's just say I am quite
impressed with the works of W. Richard Stevens (R.I.P.) in more ways
than one (he wrote his books using roff). I am not worthy.

So I went on, I've tested the lot, the obscure and the common, but
have settled for either plain old HTML (I am still looking for a wiki
which just takes HTML or XHTML instead of their own 'easy' format -
one more 'standard' ;-) ) or DocBook.
Both are machine readable, where HTML has quite the upper hand when it
comes to WYSIWYG tools for authoring.

I use Microsoft Expression to edit my HTML (some may shudder, but when
I document, I tend to discard geek notions such as 'edited with
Notepad' and that OS WYSIWYG tool-wattsisname just doesn't cut it.
Visual render in-editor is simply yech. Instant turn-off). Then to
make it palatable I pull the whole thing through one or more rounds of
HTMLtidy and maybe a bit of awk tag filtering to eradicate any
Microsoft 'specialty tags' in there. This toolchain is a remainder of
my old 'side job' as editor in chief of the company magazine for
several years. Talk to publishers, have a look at their place, have a
chat with the DTP guys and suddenly you'll grok why, till this very
day, they still crave plain text, unformatted, and start to open their
gun cabinets as soon as you start to submit in, say, MS Word.


So I'd say, if DocBook is too fancy/risky a format or when you like
your authoring tools to act as smoothly as I like 'em, the way forward
is plain old HTML.

Tools abound to author HTML in an easygoing way, zero geek required.
Convert it to XHTML (HTMLtidy!), and from there, given a bit of CSS
class assignment discipline while writing your stuff, we can go
DocBook, PDF, etc., all automated.
So type once, publish many ways.
Oh, and there's also HTML to ASCII plaintext formatters, so README.txt
and QUICKREF.txt can be generated as well for old farts like me. ;-)
Heck, I might do a conversion of those to HTML anyway. Easy.




And on another HTML related note, you may consider introducing Doxygen
to the 'techie' toolchain for extracting documentation (ahem) from the
codebase. I know, most Doxygen results are crap, but it's not the tool
but due to the coders not commenting their code nor organizing the
output. (It's not click-and-go-Libero diapers.)  The blurbs Bill puts
at the top of each classifier sourcefile are prime candidates for a
bit of Doxygenation and since he's into lib building now, the tech
folks out there waiting for it might like the generated (API) docs
that way as well: it's the closest you can get to 'up-to-date' when it
comes to developer documentation output anyhow. ;-)
All the kids these days have seen the similar tool during their Java
classes (JavaDoc) so there's a habit waiting to be used. All it takes
is a needle. The only advantage you get when programming is being
taught by using Java instead of C, IMO. :-}  But only do this when the
devs are willing to document their code in a rather detailed manner.
Otherwise it's just another bunch of Kafka-esque web pages.   Lots of
click to go, yet little 'click' to have, if you get my drift.


-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H