Re: libcrm114 ready - but can't get into sourceforge

Chris Babcock <[email protected]> writes:

> --20cf305e25c941ab4e04a75ca418
> Content-Type: text/plain; charset=UTF-8
>
> Will CRM-114 use libcrm to power the CRM scripting language or is that a
> non-goal?

No, it's a goal.  Reason: that way I only have one set of 
classifiers to keep up with, and behavior of one maps to behavior
of all.

However, the regrow methods differ and I'll have to hack the 
internals of the language a little bit to make that work out
(at the very least, the on-disk binary format changes; we may
end up ditching the mmap in the language or at least having to
hack it some more.

But (near as I can tell) there's at least two ways to make it
work (do a full open/read/write/close on LEARN and CLASSIFY, or
do a cached version of the above), modulo two things: 

  - nonportability of old on-disk classifier files into the new format
  - syntax of the language

Both of these are due to the old design versus the new:
the old language classifiers were one-file-per-class and the
new C-callable classifiers are one-block-of-memory (or alternatively)
one-file-per-classifier.

So, the in-memory format is drastically changed; there isn't really
any _easy_ way to move from old to new except in a few rare cases
and I'm disinclined to write (and test!) such a tricky tool that 
will be used so seldomly.

There's also the syntax issue: CLASSIFY takes N arguments, each
is a class file in the old system and we only have one classifier
"file" in the new one; similarly LEARN took only one argument- the
name of the class file, in the new one it will need two args,
the name of the classifier file and the name of the class to train 
into.  And- the new format allows multiple interlaced success and
failure files.  And the output format is slightly different to
accomodate that.

AND - there's the issue of initial classifier class count; 
the library needs to know how many classes a newly created 
classifier will have.  

AND - we may need to extend the language a bit if you want
to import/export from the cross-platform ASCII format - or 
perhaps that is better handled by a separate utility, like 
"cssutil" does for Bayesian/Markov/OSB files.

So, it'll be a "minor recode and retrain" day.  But hopefully 
not worse than that.

> That may be the wrong question. I'm asking from the perspective of someone
> who may be interested in language bindings whether you have any
> recommendations or concerns for library language bindings over and above the
> demo and the how-to. If CRM was being maintained that way, there'd obviously
> be additional tools on the road map. If not, there still may be wisdom about
> the API to be gleaned.

The road map has several goals; in no particular order:

  * Winnow classifier
  * Neural Network classifier(s)...  [1]
  * crm114 language update to use libcrm114
  
[1] I keep thinking that Hopfield networks are the way for the future
in this, rather than the classic (and implemented) three-layer
feedforward ANN.  

And there's a possibility of more Mitsubishi-supported work in this.

So, there you go.

    - Bill Yerazunis

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security 
threats, fraudulent activity, and more. Splunk takes this data and makes 
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2