Re: a slightly updated rpm spec file for Ger Hobbelt crm114 branch

   From: Ger Hobbelt <[email protected]>

   You're welcome. That's mostly what the earlier discussion regarding
   the RFC was about as well. To be entirely accurate: libcrm114 should
   contain the classifiers (classify + train [+ maintenance routines])
   and classifiers only.

That's the plan, at least.

This is also the time for us to restructure the source tree.  Nico
points out that the current arrangement makes it very hard to describe
what's licensed under GPL and what's under LGPL.  

So, the current plan is ./src/lib and ./src/engine where lib contains
the stuff that's LGPL (that is, the classifiers and classifier support
code) and engine is the GPL code and contains the JIT and
language-specific code.  The engine (GPL) code calls the lib (LGPL) 
code to do classifications.


   Anything that reeks of I/O in some form or other should be abstracted
   out. That's where the callbacks are useful: also on UNIX, you don't
   /always/ have stdin/out/err available (daemons don't have them, unless
   you've created a look-alike which is riding on top of inetd, but
   that's no real server software ;-) ) - so those internal/user-diag
   fprintf()s become simple callbacks (and, please, try to use elipsis
   callbacks a la printf() -- GCC even provides very nice

What happens when you hit a CRM_ASSERT?  With no I/O, your
embedded device hangs.

This is why I'm truly concerned by error testing that is either turned
on during testing only ("you now know how to drive, so you can stop
wearing the seat belt") and off during production, or asserts that do
not _always_ allow the user's code to do cleanups, such as gracefully
terminating connections, cleanly aborting a mySQL query, using an
application-defined error logging and fixup facility, etc.

This isn't to say that there aren't bugs in mainline; there assuredly are.
However, there's no _intentional_ failure to allow the maximum 
level of recovery possible.

   Something like that. Little more 'complexotron'for you: since you're
   goign to feed those classifier animals an unknown number of CSS files
   (== CSS pointer+len chuncks) it's (and recall our int/long/size_t
   session, yes? thanks)

   >          void ** statistics_list_ptr, /* it's not ''char', so keep it 'void'. Cast internally. After all CSS format is only determinable once you know which sort (CRM_XYZ flags) you've got today */

   >          size_t *statistics_list_chunk_lengths, /* one length per ptr */
   >          size_t statistics_list_size, /* the number of CSS ptr+len chunks ou're feeding today */

   That way you can do all crm114 is capable of today: two CSS, three CSS, etc.etc.

   >
   > Now, here's the question... what about multiple statistics files (that is, more
   > than one?)  The "new" statistics files are pluripotent, that is, they will contain
   > both positive and negative examples.
   >
   > But should we have a way to put more than one of them into a call?  I vote "figure
   > that out later, probably in a wrapper function".

   See above: my basic approach doesn't mind pluripotent vs. multiple
   single-potent CSS dbs: as long as each 'CSS' - as we know them today -
   can stay ONE CONTINUOUS CHUNK OF MEMORY - it's just a job for the
   outer layer: when fed a pluripot, mmap and chop it up in chunks.

   That implies ONE assumption/restriction for pluripot files in my
   approach -- as long as you want to mmap them --: each part has to be
   one block, so 'growing' them is a no-go. So it's fine for OSB et al,
   but no-go for Hyperspace as that bugger *appends* when training.

OUCH!!!!  I had not thought of that.  

   Sure, there are solutions around that problem, but then we start
   talking about mechanisms which are so close to what *real* databases
   do, it's a very fine line between 'pluripotent hyperspace format'and
   just any regular database which can handle huge blobs (PostgreSQL,
   Oracle, DB2, MSSQL, etc.) and have those serve such content.
   Which leads to the easy way out: pluripot for non-appending classifier
   CSS formats (size once, train forever), and use separate CSS files for
   appending classifier formats (Hyperspace, etc.)

"Real" databases have very high overheads compared to the hash-and-go
that CRM114 uses for most things.  Compare the per-token speed of 
Spambayes (which uses a very lightweight DB) versus CRM114, and you'll
see a huge differential.

Admittedly, CRM114 does it with no regard to hash-clash errors; that's
intentional; after all, half the time, it's "bank error in your favor".

But there's an easier way to do it for LEARN than having the user
create a callback.  We know what has to happen, so let the classifier
do the heavy lifting:

   1) inbound data is:

        const void *start, const u64_t max_length, const u64_in_use_bytes

   2) outbound result is:

        void *newstart, u64_t *new_max_length, u64_t *new_in_use_bytes
  
So, the classifier codes are free to malloc up a bigger space and copy 
into it, or just use available internal space via realloc.

This also means that the classifier itself can decide when to 
microgroom versus when to grow the file.

The caller needs to know that all of these things might change, and
they need to be properly acted upon.

Another nice feature of this coding method is that if you supply it with
a NULL start, the classifier knows it needs to build a new statistics
buffer.  

We still need to put in the capability of a policy that tells
the classifier to trade off accuracy <-> speed <-> storage footprint,
but I want to think about that first.

   The other road towards pluripot for appending classifiers means you're
   going to çhunkify' the file, i.e. the old 'good.css' and 'bad.css' in
   there each are chopped into multiple chunks. On append another chunk
   is filled, or added on overflow. But that would break the current
   classifier code in a big way anyway. And then, IF you choose to go
   this way, the whole pluripot format becomes an *internal* issue for
   the classifier, which can then be fed one mmap-ed ptr+len for the
   whole thing.


   One caveat to it all: you can't ptr+len for appending classifiers
   (which now do their train() I/O through fwrite() for instance. That
   where you need another callback, which takes over the job of the
   fwrite() of old: libcrm114 sends memory block to append to CSS
   _entity_ (often, that'll be a file)through callback, outer layer picks
   it up and does the (system dependent) fwrite() equivalent.
   That's what you need to abstract out all platform I/O from things like
   Hyperspace train. (Hyperspace classify accepts a ptr+len for each CSS,
   no sweat).

yeah.  Hmmm... given the usual grand disproportion between 
LEARN and CLASSIFY (1 learn per million classifies, or so it seems)
making CLASSIFY as fast as possible is probably the win.


   Do yourself a favor, don't make it all separate function arguments,
   just have an interface struct (yes, I would call that a çlass', but
   that's me) where all those ptr+len & callback args can be filled in.
   All of them are related anyway; can't leave out half and still use a
   classifier succesfully anyway.

Matter of taste, really.  I still have to type in all those assignments
and whether it goes to slots in a struct or into a calling sequence
directly makes little difference.

De gustibus non disputandum est.

   result:

   >    crm_classify
   >        ( CRM_IO_STRUCT *base /* CSS typecode+flags+ptr+len + fwrite callback + diag callbacks */,
	       void *msg_ptr,
	       size_t msg_len, /* no use for a start_index: outer layer
   can do that pointer arith; we don't */
	     CRM_RESULT_STRUCT *results
   >        );

   And regarding that ösb ..." string: thanks for using enum's or
   #define's to feed that sort of thing into libcrm114: strings is for
   humans, integer codes and bits are for machines (they love it), and
   libcrm114 is a machine-machine layer API. Any human-related parsing is
   done way before in an outer layer (e.g. crm114 script parser, which
   already produces a flag set for this string today anyhow: keep it
   there).

Um, yeah.  Those should be CRM_blotz #defines... and actually, they
already are defined and everything.  They're bits in a 64-bit field.

Is the 64-bitness going to cause problems in the embedded system
market?

	- Bill Yerazunis

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general