Re: a slightly updated rpm spec file for Ger Hobbelt crm114 branch

On Mon, Feb 2, 2009 at 9:29 PM, Bill Yerazunis <[email protected]> wrote:
> So, the current plan is ./src/lib and ./src/engine where lib contains
> the stuff that's LGPL (that is, the classifiers and classifier support
> code) and engine is the GPL code and contains the JIT and
> language-specific code.  The engine (GPL) code calls the lib (LGPL)
> code to do classifications.

Sounds fine with me. Copy&paste configure.ac and the makefiles and
make crm114 configure 'see' libcrm114 and it's a go. Hm, have to think
this over again. The usual way for libs and apps is to
cd libcrm114
 ./configure
make
make install <-- so it ends up in a known place, say, /usr/local/lib

then the app
cd crm114
./configure   <-- which finds the libcrm114 static or dynamic lib
make
make install

Since there's two licenses and, in my PoV, two purposes/markets as
well, it means libcrm114 is split off crm114 and crm114 is then a user
of libcrm114, just like it is a user of libTRE (the regex lib).
OTOH, one could keep them together (no migration effort for me) but
then that would make it much harder for those who just like to use
libcrm114 /without/ crm114 -- like me.
Anyway, separate dirs is very good for this.


> What happens when you hit a CRM_ASSERT?  With no I/O, your
> embedded device hangs.

Depends. And hanging isn't necessary bad -- if you do it in a
controlled way. Fatal disaster is fatal, after all.

Mind you, CRM_ASSERTs are not a user check, they are developer checks.
That's also why they are compile-time removable, and, yeah, I often
leave them active in production environments as well, but that's more
to do with my experience regarding corporate test quality then
anything else.
I always joke rather cynically that "it compiles, hence it works".
Unfortunately, recent years in the corporate environment have shown me
IT managers increasingly take this adagium literally (as if they had
invented it on their own already, but lack the required odd sense of
humor) and are so eager to 'push' the s/w you get development vs. test
ratios of over 5:1 (and don't start me on design time vs. dev time.
It's all done Britney Spears style: "can't sing? no worries, mate.
We'll fix it in the mix."  Video people (hi!) will know the equivalent
for film: f*cked up the shot? no worries, we'll fix it in post!
(meaning some poor sod will have to tweak the colors for weeks till it
looks somewhere close to okay or get down to tedious rotoscoping.
Happy happy, joy joy.) Design (e.g. scenario analysis: what's the
software got to do; what's it got to do in case of X or incase of
anomaly/error Y?) is 'fixed' by brute-force coding ('the mix'). So IT
has this equivalent: "concept doesn't work? Heck, no sweat! We'll
fiddle the code till we pass the test." (Too hard? Delete the test.
Get the related delivery item scratched. Yes, I've seen it happen.
More than once. "Over here, we don't test for /errors/! Our software
works! We're not interested in /errors/!" [says CTO] )

But let's forget that experience for a bit (which indeed hints at
'best practices' being keeping the CRM_ASSERTs active for ever and
always): then CRM_ASSERT is meant as a checkpoint to validate basic
designer/developer assumptions.
As such, assertions should be active (and may/will be hit) during
component ~ , system ~ and integration tests.
No, CRM_ASSERTs are not meant to check assumptions
(pre/postconditions) which [may] vary due to end user activity (you
don't assert on a crm114 script programming error coursing through the
parser or interpreter, as that's a user-based error), but assertions
instead are meant to catch your own obvious ('C' programming)
glitches.
(See the Microsoft paper; see other software development books that
concern themselves with writing software in general, not just language
X (that's be Java today))

Put in another way: an assertion should never trigger, no matter what
you feed the software under test. Not in 'regular', nor in 'irregular'
/ 'erroneous' conditions.

For example, yesterday I was happy to have assertions in there: I did
something stupid in the Markovian code, fubar-ed it such that in
effect the run-time tried to unmap an *already* unmapped mem area.
Code flow like this:

ptr = mmap(css)
...
unmap(ptr)
...
unmap(ptr)

This is quite stupid, but it happened. I sprinkle Assertions
throughout the code to help me catch this kind of stupidities: that's
another reason why I like extended test sets WHICH INCLUDE ERRONEOUS
TEST SCENARIOS: during 'make check' my code went nuclear, and instead
of a hard to track down weird error (or a coredump) I got an assert
statement telling me which condition failed on me. From there, it was
a few brain minutes between the 'huh?!?!' and the 'oh sh*t! no!'
moment, when I found what I had done and could fix it.

By the way: currently GerH is configured to, by default, keep the
CRM_ASSERT code active for ever and always, *and* throw a crm
trap/fault instead of printf+abort/exit. But that's just a feature of
my macros & configure script. To protect the natives, you'll have to
use a special ./configure option when you want those cycle-nibbling
CRM_ASSERTs outa there!


I know you are 'against' assert, you've vented that often enough, but
CRM_ASSERT is more than default assert() (= print line + abort()).
Also, have you considered what the [libcrm114] code flow is to be when
you hit on of that set:

nonfatalerror
fatalerror
untrappableerror

? Because they too exhibit the same problem from a code flow
perspective: the question how to recover from / continue after such
failures. Another place where a callback is the way to go, so the code
calling libcrm114 gets to decide what those buggers are to do
_exactly_. The outer layer knows better how to handle faults anyway,
oftentimes.

Pop stack to original question: what to do in an embedded environment:
that's precisely the type of development where actually _testing_ your
software is important. During testing, I would 'activate' the
CRM_ASSERTs. As they call a custom function, such a function can log
the issue in any way feasible for the given embedded device under test
(log to flash, sends syslog message over IP stack, etc.) and once that
action completes -- we already know the developer has made a grave
mistake, so any 'trustworthiness' of the software as a whole is
already gone -- we can decide to either continue as if nothing
happened and wait for the coredump/lockup to happen, OR halt the test
device in a controlled manner. For nonfatal and fatal my initial pick
would be 'continue & pray', for untrappable, it's definitely 'wait for
the watchdog to kick our booty'. CRM_ASSERTs would probably, by my
choice, land in the fatalerror 'continue and pray' camp, until I hit
'weird issues', where I will promptly recompile after setting them to
act like untrappable, so to make sure the error log gets through and
we've got controlled reboot going on. Most embedded devices have a h/w
watchdog these days, so that would mean a few (mili)seconds of
twiddling thumbs (after we made a best effort to get our error report
stored or sent somewhere) and a reboot follows.

Remember that, before you reel at this, an assert in any form is there
to catch grave programmer mistakes, which would otherwise corrupt the
software. null pointers in places you'd never expect them; double
free()s (detectable once you adjust the free() interface using a
wrapper, which also NULLs the pointer which was just free()d, for
example), etc.


As counter example, assert-ing on a FILE* != NULL following an fopen()
call is /worse/ than Bad Form(tm). One-way trip to the firing squad,
I'd say. (And if you go and look deep enough, I /might/ be candidate
for that firing squad myself ;-) )
When it comes to NULL pointer checking and such, the edge [between
ASSERT and untrappableerror()] becomes a little hazy: for speed's
sake, one may like to dispense with checking /all/ input args in
/every/ call or at any stage in between code sections -- as long as
you can argue that those NULL pointers would never reach down there in
such speed-demanding routines. The assert is then added to prove that
argument, at least empirically. As such, it must be combined with a
good test set (that's why every bug report ever should get its own
test script in the permanent test set as well, BTW), i.e. a test set
which has a good 'coverage' of the software whole, _plus_ a thorough
code review.


Since you seem to have had a look at the latest published GerH code
(finally! good!) you will have found that the error checking, which I
strongly advocate everywhere, is enhanced in two ways, when compared
to vanilla:

First, indeed, there's the CRM_ASSERTs all over the place. I haven't
run any statistics on this, but I'm willing to bet that the areas
where I have been busy extended crm114 will show a larger number of
asserts per LOC than other parts. Take, for example, the built-in
debugger: compare your quickref and mine and notice the documented
differences: I can, a la gdb, 'track/view' variables in a crm114
script, even record a sort of debugger macros (view current script bit
context + variable values + step to next opcode in script), modify the
code flow by jumping to other locations, e.g. when I want an 'alter'
in the script to behave 'the other way', etc.
Lots of CRM_ASSERTs in there: all of them there to ensure my code
doesn't screw up the fragile script interpreter state vector ('program
counter', etc.).

(Yes, the debugger language, which was already quite 'on acid' (which
Eric S. Johansson, who I understand coined the phrase that crm114 is
awk on acid (or was it teco on acid) will surely appreciate) is now,
following my own additions, definitely something to be 'experienced'
through a glass pipe. Heck, ever seen a debugger command as gorgeous
as
      nie;v>5
?) See old mailing list post form last year for this and more.



Second, and as important, I've added error return value and other
run-time checks related to script and other user-inducible faults, all
of which trigger one of these forms

nonfatalerror
fatalerror
untrappableerror

... as they should. I've been rather loose on picking nonfatal or
fatal, yes, but at least it's a huge improvement already. And those
checks should /stay/ in an (optimized) 'release executable', as these
all check for failures which can be caused by some external element or
other (crm script, file system, OS itself, heap management, etc.etc.)

The one thing where GerH and vanilla can 'shake hands' regarding error
handling is our common crumminess in actually *following up* on those
nonfatalerror and fatalerror-s (untrappableerror just aborts the
runtime, so nothing to recover there; best thing to do in an embedded
setup (where you don't have the benefit of UNIX-like process cleanup)
is a hardware reset): nonfatal and fatal both return signal codes (not
error codes) and even when it's a nonfatal, often times the best
follow up would be to clean up the current call and return to the
script interpreter. In actual reality, most of the time, the code just
barges on.
Ouch.
Definitely an area waiting for someone to sweep a broom, but I too
have a limited amount of energy reserved for fixing error checking
paths for existing codebases: I needed some functionality up and
running in a trustworthy, stable manner. That's why I don't like
'intern done' software (and 'interns' can range from age 12 to 65 it
seems): when you don't start out writing your code with a focus on
error handling and propagation, next to implementing the process
itself, you'll quite often find someone (yourself) having to do it all
over again. (CLUMP comes to mind: still it coredumps and it irks me no
end the lame excuse is 'it's experimental'. My foot. Experimental,
that's NN reporting it can't converge, but coredumping software is not
experimental. coredumping software is crap. And, yes, CLUMP coredumps
on /all/ platforms I can test it on. Not somewhere in the middle, but
quite near the start. That's vanilla code, so no 'Ger' influence
there. Twas probably a rush job for a conference or a paper to get out
posthaste, but you've been darn lucky the thing did something on your
h/w.)


And, yes, I know others have fed terabytes of mail through crm114 and
never complained about any of this. Well, I got the Mano Negra because
all ever happened to me was errors, errors and then some more spurious
errors. So I reverted to a full fledged code review back in '07 and
everything which was found to can-go-wrong, was tagged and fixed as
best as possible, with a CRM_ASSERT or, almost as often, a run-time
condition plus fatal or nonfatal in the vicinity. That got me out of
the errors arena and into the rather more entertaining 'why the heck
is the classifier underperforming?' arena. Which I still haven't left,
alas. <snif> I shouldn't get me an MBA, I should get myself a top
notch statistical analysis brain add-on. But that's another [William
Gibson] story.)







> This is why I'm truly concerned by error testing that is either turned
> on during testing only ("you now know how to drive, so you can stop
> wearing the seat belt") and off during production, or asserts that do
> not _always_ allow the user's code to do cleanups, such as gracefully
> terminating connections, cleanly aborting a mySQL query, using an
> application-defined error logging and fixup facility, etc.

assert: It's rather using a test dummy to check what a crash does to
mr. crash test dummy. When done, taking the test dummy out of the
trial volvo, then mass-produce them ('release build') and putting
actual humans in them (I drive Volvo - good cars). And, yes, those
humans sometimes die in accidents.
That's why there's legislation about how much testing you /have/ to do
/at least/. We have it for cars. We don't have it for software. So
everybody from floor 0 upwards ditches their ethics out the window,
hoist their laziness and pushes spot checked software out as product.
End result: a planet full of crash test dummies. And, hey, this is
bliss! These dummies are even free of charge! P.T. Barnum /was/ right.
(Last year's software-similar example: china company exports new cheap
SUV to Europe. German magazine and a few others pay for the crash
test. Conclusion: drive faster than a horse can run and you, your wife
and all your kids go SPLAT, one way ticket to St. Peter guaranteed.
Exit SUV. Exit China company. Come back when you've done a better job.
Same types you find in IT: European importer and others linked to
China company are affronted and call it discrimination and market
protection and other names that should rub PC people the right way.
Everybody is amazed at their attitude. But when we use or make
software, all of us act like that China corp. And nobody is amazed. Au
contraire.)




> This isn't to say that there aren't bugs in mainline; there assuredly are.
> However, there's no _intentional_ failure to allow the maximum
> level of recovery possible.

There's recovery from error and there failure on error. The former, I
relate to VMS. Awesome OS, superb software. If I am going to be
reincarnated, get my soul to a VMS world. Whenever you can recover
from an error, or even in case of error within error, it is done.
That's maximum level of recovery for you.
The latter (fail on error anyway) is, to me, what UNIX is. Your stuff
goes dung-of-monkey? Give it a guru meditation (okay, that's Amiga),
pardon, core dump. Error value is being nice; exit value is even cool,
but nothing fancy on the recovery front. Simple: no recovery. Redo
from start.
Okay, that's exaggerating, but keep that visual.

When I was young, I was VMS minded. In a way, I still am.
Yet, for some odd reason, I don't want to be reincarnated to that
parallel VMS world just yet.

crm114 on the other hand feels like neither.
crm114 benefits the most, IMO, from failing - in a controlled way.
Currently, it doesn't fail. It doesn't recover either. Because there's
quite a few spots where it simply does not check. It just plods on
after [hopefully] saying something like "hey, dude. This is some bad
Afghan' we're smokin' here, so things may become a wee bitty wobbly,
but no worries, eh?"
To the point: vanilla and GerH are still in need of a review targeted
at inspecting failure nodes and deciding what to do at each of them: a
lot of the nonfatals are rather fatal - at least to the call they're
in - so my feeling is those should all just exit the call and pop
upwards back to the script interpreter: that bugger can then handle
the issue and decide (script assisted) whether to exit or retry or do
something else entirely.

>   That implies ONE assumption/restriction for pluripot files in my
>   approach -- as long as you want to mmap them --: each part has to be
>   one block, so 'growing' them is a no-go. So it's fine for OSB et al,
>   but no-go for Hyperspace as that bugger *appends* when training.
>
> OUCH!!!!  I had not thought of that.

Then you're welcome to that one. It's on me. A freebie.

It's scenarios. Take an idea, then plonk in each of those classifiers
and ask 'what happens if I train?' 'what happens if I classify?' etc.
Yes, that takes time. But before you start writing code, it's quite
useful to do, because you'll know up ahead about several snags that
are waiting to bite you. Makes for more robust software.


> "Real" databases have very high overheads compared to the hash-and-go
> that CRM114 uses for most things.  Compare the per-token speed of
> Spambayes (which uses a very lightweight DB) versus CRM114, and you'll
> see a huge differential.

Yech, not like spambayes. Heck, I wasn't talking about lightweight DBs
anyway. And the hash tables underlying all the bayes classifiers is a
darn smart move, so we should stick with it.
Sorry, today is Exaggeration Day here, but I was referring to 'real'
databases. The ones which support 'BLOBS'. binary large objects. Of
the size of say, 40MByte per record.

I.e. stuff where you'd put a single feature set (CSS 'file') in a
single record, then do

  SELECT css.content_blob FROM css_collection AS css WHERE css.type =
'osb' AND css.group_id = 'my_bloody_mailfilter'

or something like that, so that those CSS files as we have them now
come straight out of a database - which, yes, acts as a kind of 'file
system' then.
That's the very basics: your pluripot is the recordset coming out of
that query, and you can take this to the bank: that query will be
/fast/. Because all it does it pick a few (huge) records from a measly
table and that's what DBs are darn good at. Drawback to this is that
that bloody BLOB is mem-copied, so mmap wins over that one.
When you need to append, what you can do, for instance, is just insert
the chunk to append as a record; then it's a little harder, but the
SELECT up there can have an extra SORT BY so the DB can [re]order
those chunks in such a way that to us, it'll [almost] look like
several continuous CSS files again.

But then you put that in the context of a big server, have a query
cache (which will thus spit out the very same blob records on
recurrence of the query) and a streamed or batched feed of messages to
classify (and train) and that blob memcpy() overhead will spread
across all those messages. On large numbers of messages, a memcpy() is
indiscernible from a mmap as in the mmap scenario all 'pages' get hit
by all those probes (multiple probes per message, multiple messages in
a single run: it adds up coverage-wise, so hit coverage % will go up
asymptotically). For large numbers, the recordset-memcpy will probably
equal the mmap collective, as the page misses take time too: those
happen once, and are, under the hood, DMA'd memcpy''s, so it'll be
cutting close.

Sure, this is not for home-grown mail filters, but for big boys, doing
lots of messages: no matter how you are feeding such bastards, mmap or
otherwise, in the end it'll only perform when the entire CSS set sits
in memory. For ever.
(See why I like memory-based APIs so very much? This is my growth path
here, and I can do without the intermediate file I/O; it'll only slow
me down no end.)


Bottleneck is (and remains) classifier.train activity. That action
locks all activity to that CSS, so train is (and will remain)
expensive. In database terms, its impact is similar to a table lock:
everything stops until daddy's done.





> Admittedly, CRM114 does it with no regard to hash-clash errors; that's
> intentional; after all, half the time, it's "bank error in your favor".

I was planning (as a side project) to try a cuckoo-hash based layer
instead of the current linear-probe hash table, as cuckoo has, I
believe, so very nice characteristics which would help cut down the
linear probe lengths when fill rates go up (that's generally
published) AND due to its way of storing things, there's a nice little
boon waiting to be had for microgroom: now, linear probe, of course,
puts the latest feature at the end of a chain. That's bad for
microgroom as you have to move the chain back to start wehn you groom.
Re-insert all those remaining feature hashes again.
With cuckoo (and especially with the extended, modern variants) you
can store the latest feature AT THE FRONT OF THE CHAIN. Heck, that's a
basic behavioural treat of cuckoo as it comes.
So what's microgroom in such a world? Simple! Walk down chain. *Chop*.
And you're done. (Modern cuckoo has dual or N entries per hash index,
so no moving around required on groom.)
Saves on re-placing feature hashes in microgroom chains - which
happens, and must happen, in linear probe tables. You can
automatically do aged-based microgroom. On the fly. Just limit the
'chain length' in cuckoo and you have microgrooming in effect.

The idea is there because at a certain point in life of the CSS, the
table will fill up and a periodic groom is in order. As linear probe,
at fill rates nearing 50% (and exponentially worsening beyond that -
it's exponential all the way, but most say @ <50% it's 'negligible')
you've got to do multiple probes per hash anyway, so average cost per
hash probe should be about equal for n-bucket cuckoo versus linear
probe (once it fills up, which it will, quickly). Ergo: cost savings
in microgroom allow for either more speed on average or more & better
grooming at sustained speed.
Besides, cuckoo behaves favorably at higher fill rates, so CSS's which
are in use for a little longer and fill up will see far less
perforrmance reduction over time then with linear probe.

That's the idea.

"All I need to do" is build it. That's why I've been looking at
C++-ing this stuff: classifiers use hash table.  base class = hash
table. derived class #1 = linear probe hashtable, swap for derived
class #2 cuckoo hash table, and rerun test set, check timing and
ha-hum about the results. Maybe I'm right, maybe I'm wrong. When I'm
right, I've got a nice alternative for osb and friends, where
featuresets need to be packed in the CSS (= desired higher fill ratio)




> But there's an easier way to do it for LEARN than having the user
> create a callback.  We know what has to happen, so let the classifier
> do the heavy lifting:

Actually, that's harder, because now you must tell the user up front
how much space he's got to reserve for the write (hyperspace: in
essense, it's arbitrary dcument length there. Okay, limited to 500K
features right now, but that's nitpicking). (And forget about both
sides jointly malloc/realloc/free-ing this sort of thing - that's
total disaster waiting to happen.) They can as easily screw that up as
when they do a callback, so what's the difference? Flexibility.
Callbacks are natural for this thing as it's a 1:1 replacement of the
current RTL function calls.

Okay. I just suppressed a looong sigh.

Don't be scared of callbacks. And if the reasoning is (yes, you've
said that before, but somehow, ah well...) is that the libcrm114 '
'users' cannot cope with a callback', well, let me be blunt here: the
*user* of libcrm114 will be a 'C' programmer. Nobody else can use
libcrm114, as it is a library. At the very least, someone will need to
write a wrapper around it, if you're going to offer it to anyone BUT a
'C' programmer.

And let me put it this way: any 'C' developer who CANNOT cope with a
callback, i.e. a function pointer, should pick up his goods, clear his
desk and go back to filling shacks at Wall Mart.

Shocking?

No. Been there. I've had a time when I didn't grok function pointers.
Admittedly, that was a very short time, but then there's also been the
time when I didn't understand recursion. /THAT/ time was about a year,
give or take a month. I still remember. Because I saw the Light when a
friend explained it to me. I was 16 at the time.
Was I a programmer then? /No/. I was, irrespective of my age at the
time, a *student*.
I don't learn 'on the job'. I learn *before* the job.

When you code 'C' and are not able to 'grok' what, say, the possible
use of a 'void ***p' is, or go numb when you see


typedef int fprintf_like_callback114(const char *msg, ...);
...
int libcrm114_classify(fprintf_like_callback114 *cb)
{
  ...
  if (internal_trace)
  {
    (*cb)("malloc failed. Stick a fork in us. We're done. (size =
%lu)\n", (unsigned long int)malloc_size);
  }
  ...
}


then you should pick up something else. (Try Java. Less chance of
running into me on the floor. ;-) )

So if you use interns (you did before, right?) and they go 'huh?' at
that example bit (basic interview question. maybe?), then there's a
nice little book for them to add to their curriculum. Written by
Kernighan & Ritchie. It's all in there. (Okay, 'stick a fork' is
yours, not Kernighan's, but you get the point.)



(And let me cut off the KISS response beforehand. If 'simple' is so
important it forecloses the requirement to learn anything new (and
some times it sure looks like that), then wonder why we took up the
effort, each of us, to learn to read and write. It took us years (16
or more, and still you're not done) and lots of agony to achieve an
active vocabulary which surpasses that of the Teletubbies. That'd make
us pretty much 'overachieving complexity lovers' all, because as far
as I can tell, those Teletubbies communicate as well as humans in a
wide range of subject matter. Ever heard an elephant talk, or type an
email? Still, they get older than any human, so dexterity with
grammars isn't a survival treat either. So what the Hell did we learn
human languages for? Aga aga ag agg KISS.)



> So, the classifier codes are free to malloc up a bigger space and copy
> into it, or just use available internal space via realloc.

Yech. one side of the API does the heap allocation thing, same side
has to clean. Or realloc.
Especially important once you drop this kind of thing in dynamically
loadable libs and environments where mixed run-time libs happen. (Say:
Windows. But UNIX as well: different libc's)

Use callbacks and you abstract it all out to caller: you feed him the
data chunk, he can do whatever he likes. copy, alloc in own space,
forward to socket, whatever. Don't try to circumnavigate callbacks.
It's part of the idiom. It's in the standard runtime lib. So it's
nothing fancy. Get over it.



> We still need to put in the capability of a policy that tells
> the classifier to trade off accuracy <-> speed <-> storage footprint,
> but I want to think about that first.

Hm. I thought I had the different classifiers for that. ;-)  (speed
and accuracy and footprint)
You've been telling me I am complexotron, but heck if this ain't going
to be one with capital C, that way.

The idea with the different hash table (cuckoo vs linear probe) is
only easily doable once you move on to C++ (and, yes, the embedded
world has C++. All over the place. As long as you don't go
fancy-footing by plonking in STL or other template-based libs
everywhere you possible can, it's perfectly okay.)

And then, such 'cuckoo hash' ideas and such, are, as far as I am
concerned, R&D lab *only*. It's good if the code allows things like
that to be done to it, makes for a nice R&D _and_ production platform,
but when I want to tune accuracy vs performance vs. footprint, I'll
just cut down on the size of my CSS, set up groom or whatever the
equivalent, to keep fill rate well below 50% in the hash table and
that leads to good performance speed-wise, much less so accuracy-wise.
But all those parameters are already in place today, for current
crm114, so nothing extra needed.

Want more? Pick another classifier, heck, drop in a custom tweaked VT
matrix if you've got the time to research those, that's at least where
I see the performance and accuracy matter (and differ).




> yeah.  Hmmm... given the usual grand disproportion between
> LEARN and CLASSIFY (1 learn per million classifies, or so it seems)
> making CLASSIFY as fast as possible is probably the win.

Errr... not only Exaggeration Day for me, then. Well, I do get the
point. Yes, making classify /fast/ (and keeping it going /fast/) is
the ticket. When needed, I can always dedicate a machine to training
only and swap CSS's in and out, so the classify boxes can keep running
at full speed. Training is delayed, so be it. Decision to train is
always after the fact any way; only for real-time apps which
auto-train may it matter, and then it's still not a big problem, IMO,
if a few messages pass through classify on box A while box B trains a
previous item, only to swap the CSSs when done, so that a few messages
'slipped through' the system before the latest training was 'in'.

And if I find I cannot live with that 'slippage' anyway, then I,
indeed, need to get me some fancy new, faster, gear.



>   Do yourself a favor, don't make it all separate function arguments,
>   just have an interface struct (yes, I would call that a çlass', but
>   that's me) where all those ptr+len & callback args can be filled in.
>   All of them are related anyway; can't leave out half and still use a
>   classifier succesfully anyway.
>
> Matter of taste, really.  I still have to type in all those assignments
> and whether it goes to slots in a struct or into a calling sequence
> directly makes little difference.
>
> De gustibus non disputandum est.

Yup. But the struct way has a little advantage: it's the easy way to
have 'default arg values' in C a la C++ and other languages. (Okay,
with restriction).

memset(&struct_var, 0, sizeof(struct_var)); /* zero all elements */
struct_var.arg3 = xyz; /* fill in only the args which you find important */
call(&struct_var);

For the rest, it's indeed taste.


> Um, yeah.  Those should be CRM_blotz #defines... and actually, they
> already are defined and everything.  They're bits in a 64-bit field.

Currently, yes.


> Is the 64-bitness going to cause problems in the embedded system
> market?

Depends if you are targeting high end only or not. Some of them won't
like it, that's for sure.


But allow me to point you at a VERY nice 'C' feature, which you
haven't used yet: bit fields.

Second, when looking at those defines, a few are mutually exclusive in
a way which suggests they're rather better served with an enum. Where
an 'enum' has the added benefit that most compilers perform
type-checking, adding that extra bit of safety/propriety coding
validation at compile time, for free.

The classifiers should go in an enum:

typedef enum
{
  CRM_CL_UNKNOWN = 0, // always handy to help detect morons who didn't
spec this item
  CRM_CL_MARKOVIAN,
  CRM_CL_OSB,
  ...
} crm_classifier_t;

so that cuts out several bits worth' of flags.

Then we have all those 'attributes' which we can mix & mash, such as
'unique', 'bychar', etc.
Those can remain bit-value #defines, or, when you want to do it
another way, without the preprocessor assisting you:

typedef struct
{
  unsigned unique: 1; // one bit: boolean!
  unsigned bychar: 1;
  unsigned unigram: 1;
  unsigned microgroom: 1;
  unsigned refute: 1;
} crm_classifier_flags_t;

The beauty of bitfields is that they 'pack' into integers, and the : 1
there clearly states to anyone reading it, it's a boolean. Besides,
you can have as many as you want in there. No worries about 64-bit /
long long support either.

(You can also do things like 'int bitfield: 3;' for a 3-bit sized
variable, but that's more for when you want to construct protocol
packet headers or some such.)

Personally, I use #defines with hex values, but that's mostly historic
influence: PureC (~ 20 years ago) for MC68K (Atari) produced way
better code for your hand-crafted

if (flags & BIT_MASK_X) { ... }

condition check than the equivalent

if (flags_struct.bitfield_x) { ... }

while the others produced (close to) identical code. As PureC was my
favorite, bitfields became 'fancy' because you knew up front you'd get
more opcodes and slower run-time, so I did it only when I wanted to
annoy a friend of mine.
Ah, those were the days. Though the best thing was when I finally had
the money to buy my first ANSI C compiler, so 'the old days' are not
what they're played up to be.
Today, PureC is long gone (though a good friend of mine still uses it
for all his MC68K work) and bitfields are treated like those hex mask
#defines, like they otherwise were and are and should.




-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com