Re: a slightly updated rpm spec file for Ger Hobbelt crm114 branch

On Mon, Feb 2, 2009 at 2:57 PM, Bill Yerazunis <[email protected]> wrote:
> Indeed.  Nice thing about forks is you don't feel bad about looking thru their
> code.  :)
>
> Also- Ger's point about having the _classifers_ separated from the filesystem is
> really quite important.  We'll do it that way.  (thanks, Ger.)

You're welcome. That's mostly what the earlier discussion regarding
the RFC was about as well. To be entirely accurate: libcrm114 should
contain the classifiers (classify + train [+ maintenance routines])
and classifiers only.

Anything that reeks of I/O in some form or other should be abstracted
out. That's where the callbacks are useful: also on UNIX, you don't
/always/ have stdin/out/err available (daemons don't have them, unless
you've created a look-alike which is riding on top of inetd, but
that's no real server software ;-) ) - so those internal/user-diag
fprintf()s become simple callbacks (and, please, try to use elipsis
callbacks a la printf() -- GCC even provides very nice
analysis/checking attributes for such stuff -- so the internal idiom
can stay fprintf()-like, while it just calls a callback which feeds
the whole shebang to the outer layer (crm114 app or other libcrm114
úser' code
[ú == 'u ...@#$% bloody Dutch keyboard layout - I want my US key
layout right /now/ :-(  dang corporate user profile]

>    ... blah blah blah...
>    //  some code that gets me a pointer to the in-memory statistics bytes,
>    //   like crm_mmap(), mmap(), even stat()+malloc()+fread()
>    crm_classify
>        ( "osb unique",
>          char * unk_ptr,
>          long unk_length_bytes,
>          long unk_start_index,
>          char * statistics_1_ptr,
>          long statistics_1_len,
>          CRM_RESULT_STRUCT *results
>        );

Something like that. Little more 'complexotron'for you: since you're
goign to feed those classifier animals an unknown number of CSS files
(== CSS pointer+len chuncks) it's (and recall our int/long/size_t
session, yes? thanks)

>          void ** statistics_list_ptr, /* it's not ''char', so keep it 'void'. Cast internally. After all CSS format is only determinable once you know which sort (CRM_XYZ flags) you've got today */

>          size_t *statistics_list_chunk_lengths, /* one length per ptr */
>          size_t statistics_list_size, /* the number of CSS ptr+len chunks ou're feeding today */

That way you can do all crm114 is capable of today: two CSS, three CSS, etc.etc.

>
> Now, here's the question... what about multiple statistics files (that is, more
> than one?)  The "new" statistics files are pluripotent, that is, they will contain
> both positive and negative examples.
>
> But should we have a way to put more than one of them into a call?  I vote "figure
> that out later, probably in a wrapper function".

See above: my basic approach doesn't mind pluripotent vs. multiple
single-potent CSS dbs: as long as each 'CSS' - as we know them today -
can stay ONE CONTINUOUS CHUNK OF MEMORY - it's just a job for the
outer layer: when fed a pluripot, mmap and chop it up in chunks.

That implies ONE assumption/restriction for pluripot files in my
approach -- as long as you want to mmap them --: each part has to be
one block, so 'growing' them is a no-go. So it's fine for OSB et al,
but no-go for Hyperspace as that bugger *appends* when training.

Sure, there are solutions around that problem, but then we start
talking about mechanisms which are so close to what *real* databases
do, it's a very fine line between 'pluripotent hyperspace format'and
just any regular database which can handle huge blobs (PostgreSQL,
Oracle, DB2, MSSQL, etc.) and have those serve such content.
Which leads to the easy way out: pluripot for non-appending classifier
CSS formats (size once, train forever), and use separate CSS files for
appending classifier formats (Hyperspace, etc.)

The other road towards pluripot for appending classifiers means you're
going to çhunkify' the file, i.e. the old 'good.css' and 'bad.css' in
there each are chopped into multiple chunks. On append another chunk
is filled, or added on overflow. But that would break the current
classifier code in a big way anyway. And then, IF you choose to go
this way, the whole pluripot format becomes an *internal* issue for
the classifier, which can then be fed one mmap-ed ptr+len for the
whole thing.


One caveat to it all: you can't ptr+len for appending classifiers
(which now do their train() I/O through fwrite() for instance. That
where you need another callback, which takes over the job of the
fwrite() of old: libcrm114 sends memory block to append to CSS
_entity_ (often, that'll be a file)through callback, outer layer picks
it up and does the (system dependent) fwrite() equivalent.
That's what you need to abstract out all platform I/O from things like
Hyperspace train. (Hyperspace classify accepts a ptr+len for each CSS,
no sweat).


Do yourself a favor, don't make it all separate function arguments,
just have an interface struct (yes, I would call that a çlass', but
that's me) where all those ptr+len & callback args can be filled in.
All of them are related anyway; can't leave out half and still use a
classifier succesfully anyway.

result:

>    crm_classify
>        ( CRM_IO_STRUCT *base /* CSS typecode+flags+ptr+len + fwrite callback + diag callbacks */,
            void *msg_ptr,
            size_t msg_len, /* no use for a start_index: outer layer
can do that pointer arith; we don't */
          CRM_RESULT_STRUCT *results
>        );

And regarding that ösb ..." string: thanks for using enum's or
#define's to feed that sort of thing into libcrm114: strings is for
humans, integer codes and bits are for machines (they love it), and
libcrm114 is a machine-machine layer API. Any human-related parsing is
done way before in an outer layer (e.g. crm114 script parser, which
already produces a flag set for this string today anyhow: keep it
there).


-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword