Re: a slightly updated rpm spec file for Ger Hobbelt crm114 branch

On Fri, Jan 30, 2009 at 2:56 PM, Bill Yerazunis <[email protected]> wrote:
> The only difference I'm aware of that is substantiative is that the
> Ger fork does not implement the "right thing" with respect to
> assertions detecting internal errors; an internal error that is
> recoverable is specced to cause a FAULT / TRAP execution, and the Ger
> assertions don't do that.

The vanilla crm114 contains no equivalent nor a replacement for those
'assertion' checks. They do not exist in vanilla crm114.
A matter of differences in taste? Probably. (
http://research.microsoft.com/apps/pubs/default.aspx?id=70290 )

The 'ASSERT' statements in the GerH C source code serve a very
particular purpose: they exist to detect *C* coding failures, e.g.
off-by-one errors, functions being fed inputs which don't match their
assumptions, ditto for outputs, etc.
It's all about preconditions and postconditions, if anyone can recall
those CS software proof 101 classes ;-)  (always seemed to me most
IT/CS students had bigger headaches due to those classes than from
their liquor intake during the rest of the year. But then, some accuse
me of having no sense of 'fun'. [And an odd sense of humor to go with
that.])

These 'assertion' failures are NOT [crm114] script failures, but
*core* programmer failures. Are these 'internal errors'? Yes. But
there's two different brands of those: due to C programming mistakes
(A) or [interpreted crm114] script mistakes, which result in the wrong
data being fed to the core routines (B).
The assertions are specifically introduced and used to catch errors of
type A. As usual in the 'C' landscape, assertions can be compile-time
removed for 'release builds' (which you have to specify explicitly
when running GerH's ./configure, by the way).
These assertions therefor have NO alternative nor an equivalent in the
vanilla crm114 code base, because, as I understand it, Bill clusters
errors type (A) and type (B) into a single group and calls them
'internal errors' all. Which would mandate more rigorous run-time
checking anyway: an area where crm114 begs to improve. The 'right
thing' for 'type (B)' is indeed to have run-time checks for those and
fail/trap back to script level (which a lot of [GerH added] run-time
checking code does). The 'right thing' for 'type (A)' is to sound some
/serious/ alarms as the core developer(s) f*cked up (and you betcha:
GerH builds by default 'abort' on type (A), because I consider such
failures severe enough to declare the software as-is untrustworthy
following such f-ups, at least for the current run, and, yes, you can
alter the default behaviour to 'Bill preference' fail/trap on type
(A), i.e. assertions, by setting a single #define (there's even a
./configure parameter for it IIRC), so your run-time does the same for
(A) as it does for (B) from then on. Either flavor.

Trap-to-script on *C* coder mishaps (contrasting *script* writer
mistakes) is not the GerH build default, but it's been offered as an
alternative from day 1.)

Lacking good 'contract coding' assists in 'C', I have fallen back to
the paradigm of defensive programming in places. And I do not like
code which does not check [important] return codes, but instead
decides to recalculate the probable result. Or functions which very
subtly rely on the fact that they are only called from a particular
set of other functions in the current codebase. Thanks to the low
profile of crm114, nobody with an eye for such things has yet had a
really good look, as we don't seem to feature on the CVE list or
SecurityFocus. Yet.

For anyone wondering why certain changes are in there: I've code
reviewed the whole lot, with the planned intent / assumption that
lateron I would be extending / adapting the crm114 code base (moving
it towards becoming a full fledged daemon/server): a file-based I/O
approach is simply not of this age, when you're going to integrate
such a beast into a high performance server architecture meant for
real-time data processing. And don't get me started on fork()ing and
the lot, puh-lease.

high perf server park, that's where GerH was (and is) headed. First
stage: kick the software up to a level of reliability that's at least
tolerable for such an environ. Mostly done.
Second stage: deamonize. libcrm114 is a very good incentive. Too bad
it's still file I/O based. As such, it's useless for server
deployment.
(And raise your hands, 'web' folks who feel I'm overstating the
necesity for rugidized (modern word: 'security-aware'), streaming
software. Sure you can make a mess in a web server serving pages to
your marks, but backend financial systems (to name but one area)
require that 'extra touch' of serious software developing
professionalism.)


> Other than that issue, the Ger version has a lot going for it,
> including more rigorous compile-time and runtime checking.

That was the main drive to 'do' the GerH builds anyway: I cannot have
any software sitting in my production pipeline which silently ignores
errors, irrespective if they're due to C coder or script writer
mistakes / ignorance; the numbercrunching performed is way too
delicate for that, so, yes, I enforce the highest possible production
standards on each piece of software in there. (That's also why GerH
builds are still marked 'alpha / beta' level in my book: megatest
doesn't even get close to a good quality production test set; the GerH
'make check' test set derived and extended from that origin isn't
cutting it either. (Yet.) The basic script commands are tested more
rigorously now, including script 'fringe cases' and 'error
conditions', but a _serious_ attempt to test the classifiers is still
missing from vanilla megatest & GerH 'make check': there's a whole
range of errors/'oddities' which is not caught by those (e.g. input
series fed to train / classify). We're just darn lucky they don't
happen... don't they?


> Note: the 64 v 32-bit issue is going / gone away; with Nico and Kurt's
> help, we now have scrubbed those problems.  We also fixed a couple of
> bugs in the process.  :-)
>
> And- a heads-up: we're going ahead with libcrm114 ( callable CRM114
> classifiers ) which will be licensed LGPL so people writing
> proprietary code can still call libcrm114 and not have to reveal
> their proprietary code, just the libcrm114 code they're calling
> (and not even reveal _how_ they're calling it, or with what args).
>
>    - Bill Yerazunis

I'm glad you've considered LGPL for this; at least that'll allow some
wider use base for libcrm114.

See above about the API: as long as it requires a filesystem, it's
still flawed. (Nope, don't need a file system at all. Not for the CSS
nor for anything else. Take the mmap-ing out of the core, because all
it provides is a pointer + length and that's exactly what those
classifiers need: a series of pointer+length, one for each 'CSS
database'.)
The updraft of this: suddenly you're interesting for the [deep]
embedded market as well - worked on enough embedded OSs to know
filesystems are not 'default' there. Yet a Flash-based Bayesian
(Markovian, etc.etc.) filter is *quite* interesting for those folks as
well. Think: intelligent routers and packet filter hardware. Need
filesystem. Nyet. Want libcrm114? You betcha.

My benefit if you change to that stance? --> I don't have to roll my
own libcrm114: once I have a classifier core lib which accepts chunks
of memory on all input / outputs, I can use this baby inside both
socket-based machines and as SQL database connected functionality. Not
only 'crm114 classifier X' as a stored procedure, but also as part of
pipelines which are big on pushing around BLOBs. I don't have the time
for application invocation (did you run tests recently? Starting
crm114, let alone script-forking it, is a prohibitive cost factor I
can do without) nor unneeded file I/O (I'm not in the business of
testing my /tmp partition I/O performance).

(And for anybody else out there who likes their own script languages
(perl, PHP, Python,. Lua, Ruby, you name it), a non-file-based
libcrm114 is a real boon (language add-on!) as well as they don't need
to learn crm114 script, which was once referred to as TECO on acid,
IIRC. crm114 script as a language design a cool, geeky idea, and for
that, I like it and respect it. But it's holding back crm114 as a tool
for statistical filtering/dissemination, as not everybody is willing
to cope with another language, even when it comes preconfigured.
Especially in business/professional environments.)





-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt


PS: Sorry Eric, Eric, Alexander, Inge, Henk and Hans, who all advised
me on this: I've been quiet long enough. It had effect. Some. Alas,
that right there was the remainder of the rant you advised me not to.
I still agree with you.
Mea culpa. You're all better/stronger than me. Maybe it takes another
4 decades... I just chose to accept the consequences anyway.
GerH is a fork. And apparently, that's official.


--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword