Re: Mixed 64-bit system GerH binaries / BillYscripts

> when a LEARN can make use but 1 single class at a time; consider a 128+
> classes case.

I did & do.

Given 60 classes (= CSS files), Paolo can have his KISS and I can eat
my pie too. Simple.

The set passed to classify is a set and should be passed to learn as
well. Be it 1, 2 or even 60 classes. Indeed the <learn_spam> flag is a
total dud (I couldn't think of something dumber at that moment, I
totally agree), so optional index is MUCH BETTER, but still the _set_
of classes sticks together in both classify _and_ learn and that's
what I like about it.

Which makes the code even easier as now you can specify the set ONCE:

isolate (:c:) /class1 | class2 | and so on .../

and then all of a sudden you've got 'unified' use all over without any
additional cost:

classify (:*:c:) [message]

and

learn (:*:c:) <optional_index_whatever_you_want> [message]

or?

learn (:*:c:) (index) [message]

Both look good to _me_. ;-)


Because you always pass along the whole set at script level, the
classifier code (both learn and classify implementation) gets to pick
what they want/need, you get the chance to apply filters & processes
in learn that are simply impossible right now PLUS you don't have to
worry anymore either which classifier you're gonna use because today
all the bloody buggers require their own particular incantation when
it comes to number of css files (classes) passed to learn.

Saves a few lines in mailtrainer.crm and mailconfig.cf as well.

So no unified ... mess; I'd say it's unified ... structure / design.
Which, in this case, is particularly kissable, also at C
implementation level, not just for any future libcrm114.


Cost for Trever @ 60 classes? nil.

You like a nitty gritty nitpick like I do? Okay. one array index
access in 'C' extra. That's about, what, 12 clockticks @ < 1 nsec each
(and no cache miss penalty because you just filled the array)?
Okay, okay, lemme cut the crap and be honest and realistic for a sec
here... assuming you already use an 'isolated' variable for the
argument (so no extra there, and even when, it's about a string scan,
malloc and mem copy away) --> worst-case extra: the regex currently
used to munch the script call arguments has more text to consider
because there's (60-1) * N characters extra to digest (N is length of
one CSS filepath as you specify it plus surrounding whitespace), so
that's several hundreds of clockticks extra per character, depending
on the TRE lib code and ability of your optimizer (in the compiler) -
which can be much improved by replacing the regex by custom code which
just looks for the pipe in there like some classifiers already do.
Only after _that_ do you get that one array index @ 12 clocks and off
you go for OSB and friends. (And the others as well.) Sure, might be a
few microseconds, but if you worry about that for every *learn* call,
I should not even want to consider using CRM114 as-is at all, because
the executable startup cost alone (executable invocation, tokenizing,
etc.) is prohibitive in comparison. Heck, I'm willing to bet on the
lemma that the :*: use of a variable instead of a 'hardcoded' class
list in the classify arg is costlier than actually splitting the arg
at the | pipe symbol and picking the one class you crave in your learn
C code in the learn function :-) ... Right around *now*, managers will
develop a throbbing headache and ask: "how _much_ would it _cost_?
_Relevant_ _much_ please!" and I would say:

"nil."


At 1M messages incoming each day, I checked what the bits 'n pieces of
crm114 actually cost me long ago.

You save far more time when you find a way to reduce disc I/O cache
misses on your memory-mapped CSS files, even when you achieve such a
feat for learn alone (which would be rather weird and besides, unless
you 'Train Everything', optimizing classify is the winner). I have a
test using adapted cuckoo hashing lining up for when I find time to
play again: a different hash table organization with significantly
reduced chances at hash collisions can save your bacon on loaded CSS
files (>= 50% fill rate) compared to the current linear probe
algorithm. With [adjusted] cuckoo hashing, you can even full-auto
microgroom on the fly if you like as old entries shift towards the
'back' and can be dropped automagically.
Want some real, achievable gain? convert crm114 to play 'server', i.e.
permanently loaded and CSS files (close to) permanently mapped in
memory, that is: across processing multiple messages. Would cut my
'startup' costs significantly at least. Think along the lines of
spamassassins spamd + client code here.
On Win32, I get the fancy number of 2000 per second (that is 'two
thousand per second') as an UPPER bound for crm114.exe invocations per
second. Zero training, nada, zilch, none, just a single statement
script, which yells: "everybody out!" (exit). Didn't fare much better
on my 64-bit Linux box: I recall it was around 10K or 20K there-abouts
when the planets aligned. May be wrong, but it were devastating
numbers for me anyway.
With my feed at >1M messages a day (estimated to reach 10+M in a year
or so if things go well), imagine my 'joy'. My feed (tiny messages at
around 20..30 words each), my hardware (definitely not the very latest
nor the pricey racked server shite I'd love to buy when the money
train stops and UNloads at my station for once) classification and
learning was 80% of the load ('load' here: wall-clock time
consumption). 12% was in startup cost: that's the time between
invocation of crm114 and the moment the script *tokenizer* kicks in.
You're not even *executing* script yet by then! The rest (8%) is
spread across tokenizing ('compiling the [small!] script'), tokenized
script code execution, wrap-up and unidentified fluff elsewhere.
Believe me, if I'd see an easy way to kick that bugger into higher
gear, you'd already have it.
Serverizing (is that a word?) is on the list. High hopes, not enough
time - or should I say: not angry enough yet. Implementing it means
it's Hiroshima for significant chunks of code and I'd like a serious
rethink of the script interpreter to go with that, ma'm. And hopes
that GerH and BillY builds will be one again one day are down the
drain for sure.
I already took down and built the script interpreter up again: now
it's almost pedantic in it's syntax checking and error reporting, like
I want all my compilers and interpreters to be, while profiling showed
me 'delayed parsing' of any statement arguments is a total _loss_ so
GerH already 'does' the full arg package munching at initial
tokenization time. Saves a bundle for looping scripts and close to nil
cost addition for one-time scripts -- unless you have very deep
conditional code execution trees (read: if within if within if ad
nauseam - or should I say: long match+fail series and deep condition
trees?

Maximum gain is expected when you 'serverize' crm114 to allow it to
run as a multithreaded server which has the desired CSS file in r/w
accessible _shared_ memory and cached all the time. I've been
seriously considering hacking crm114 into becoming mod_crm114, i.e. an
Apache2 plugin: you get the server, the socket I/O and the
thread-pooled multithreading for FREE; add to that some lovely logging
and the admins might even start to like you yet. crm114 script can
live in there like a wicked PHP-alike server-side scripting language
and you will definitely achieve instant notoriety. ;-)
If only I would take the time to discard all the global variables in
crm114. Relatively easy, but it would jeopardize my 'sync' efforts
with Bill's code in a very bad way.


Anyhow, I don't see any good reason why the learn (classes) argument
cannot be identical to the related classify (classes) argument, except
ONE: strict adherence to 'backwards compatibility' at CRM114 script
level.
For sure, run-time cost does not come into play.



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/