Re: libcrm114 ready - but can't get into sourceforge

Eugene Crosser <[email protected]> writes:

>
>> The juicy bits (specifically, the classifiers) are now=20
>> C-callable (thanks to Nico and Kurt Hackenberg).
>
> Hello Bill,
>
> a few things:
>
> 1. we are sufficiently teased already, please give us the code! ;-)

I want to!  I just can't get Sourceforge to let me do so!

So, until that gets figured out, I will mail it out to anybody who
asks.  Who wants it?  Send me email!  Be advised that it's not
a tiny .gz file.  :)

> 2. Imagine I am doing many "classify" operations having the p_db
> permanently open. More seldom, I run "learn" operations, that update the
> p_db. What is the suggested strategy for this case, signal the
> classifier to re-read the p_db at the next opportunity? Can I mmap() the
> copy of p_db for the classifier? I guess that I can, but then I must
> refrain from updating it, correct?

No need to mmap it.  As long as your process stays in memory, the
p_db will too, and you'll access it just fine. 

> 3. I don't see an example of initializing a p_db from a *binary* file.
> What is the procedure for that? Call crm114_new_db(), read the data, and
> fill the pointer and length by hand?

Easier than that.  IF (and big IF) you are going to restore the
data back onto a machine with the same architecture (i.e. the same
host, running the same OS and same libC layout of structures) then
a binary read/write works fine.

In fact, the recommended procedure is to write (or read)
p_db->datablock_size blocks of data to the file system of your
choice. 

With the one exception that you need to "peek" at p_db->datablock_size
before writing, and to stat() the file to know how big a space to
allocate before reading, it's all handleable by your OS.

So, to answer your question, yes you can mmap it too.  But you don't
have to any more, and since Windows doesn't "mmap" the way Linux does,
we've taken that out of the equation completely.

Note that because during a learn some things need to grow (i.e.
call realloc()) it can be unhealthy to mmap() a db you're doing 
active learning into, because mmap does not necessarily play
well with realloc() [[IIRC, they don't know each other.  At all.]]
So if a realloc() happens during a learn, the mmap will stay but
it will have stale data in it and not get updated.  Best to
avoid this particular complication if you can.

WARNING: if you want to move to another architecture (i.e. 32 bit to 
64 bit, Windows to Linux, et al), then reading and writing a binary
WILL NOT WORK!!!!!   For these data moves, there is an ASCII export
format as well.  Use crm114_db_write_text() and crm114_db_read_text()
to write and read in a straight ASCII format that looks suspiciously
like spreadsheet CSV.

Note that in order to do this, we have to "hook" libSVM to output
ascii into our file, which is why you need to use the _particular_
version of libSVM that we use.  Sorry, someday we'll fix it.  :(

> I am looking at 20100726, maybe these things have changed in the current
> version?..

Yes, changed to make life easier.  :)

     - Bill Yerazunis

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security 
threats, fraudulent activity, and more. Splunk takes this data and makes 
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2