Re: I need a critique: using crm114 to train on (very) limited word set - or are there better ways?

   From: "Ger Hobbelt" <[email protected]>

   Ok, what I think I really need now is a kick in the head, because I
   have a hunch I've been focused so much on the crm114 hammer that every
   problem starts to look like a 9 inch nail and my local math
   wizzards[*], well, they do qualify.

   [...]

   What I am after are two things:

   1) Find out /if/ and when if, /which/ words are the 'significant' ones
   that have the most influence on the output.

   2) With any luck regarding (1), a hint at /how much/ significance
   (weight factor) or something like that might be nice.

Yes and no.

What you probably want is something like TF/IDF (Term Frequency / 
Inverse Document Frequency), or something like the EDDC from
OSBF.

(speaking of which, Fidelis:  What's the _currrent_ BEST 
coefficients and formula for the best-behaving EDDC so far?)

Now, CRM114 does not directly output these values; OSBF calculates
EDDC internally, uses them, and then discards them.  



   What's the [desired] output?

   I'll know that after some (significant) delay, so I will be able to
   tell if an event was 'it' or if it was a 'nit'. Thus I want to
   discover if a particular message has the probability of being 'it'
   instead of 'nit', preferably when the event occurs.

   That's where crm114 popped up in the brain: train the sucker based on
   older event series where you know who's 'it' and who is not; then
   continue feeding it event messages like that and see what pR says
   regarding the probability of an event being 'it' nor nit. You can
   improve (I hope) on the fly by backpropagating (training) wrong
   predictions once you know what should have been.

Now, _that_ is something CRM114 can do; _any_ of the classifiers can
do it.  

   I realize this sort of thing could theoretically be done as a
   multi-variable correlation, but I am kinda lazy and I think crm114
   might be able to do this for me, and I like to hear if I am smoking
   Afghan or Dog here. ;-)
   Especially that 'on the fly' bit seems appetizing to me as I don't
   think I'll have sufficient data to start with and sifting through
   hundreds of parameters (a.k.a. words) to find out which of them are
   significant, while a whole bunch very probably are not, is on my most
   wanted list of things to do, so I think that if I grokked crm114 well
   enough that I should be able to let crm114 do this by simply feeding
   it lines-as-messages in a nice script and then have a look at the pR
   coming out the rear end for each and regarding significance of certain
   parameters then: it should be present in the form of the weight
   numbers in the CSS files (OSB[F]) per hash (= word) or otherwise and
   from there on out it's: no worries, mate!

Yes, that would work.  Feed in a single word, get back a pR.  That's
the significance OF THAT WORD ALONE.

BUT - to do this, you must remember that by default CRM114 uses 
OSB phrasing.  You can turn OSB off by the keyword <unigram>, and 
that's what you need to do.

   Some additional details that matter (I believe):

   the current 'general idea' about the matter is that some parameters
   will certainly show to 'strengthen' each other (i.e. a kind of A AND B
   operation), while a few may independently lead to the same result (A
   OR B), but we also think that the chance that certain parameters
   actually 'cancel each other out' is low (A XOR B) - which is important
   as I understand that the classifiers (apart from NN? SVM? SKS?) cannot
   cope with an A XOR B input situation.
   [The only way I can see a XOR happen in my mind when Markovian or
   other classifier picks both A and B in the same chain to produce a
   single hash which could counterweight the learncounts on the A hash
   and the B hash. I don't grok SVM et al yet, so can't see anything
   there. And maybe I still don't grok Bayes/Markovian either... I'll
   find out now.]

Yeah, that's called a nonlinear separation situation.  Hyperspace can handle
arbitrary nonlinearities as part of it's design.  The neural net
can handle a fairly high degree of nonlinearity.  SVM and SKS are
purely linear (beyond what the short OSB phrases can generate).

I would say stick with the fast-solver linear classifiers (OSB, OSBF, 
Winnow) until you know for sure that you need a more powerful 
nonlinear separator.

   For some tech detail: every parameter has a name and since it's on/off
   (show / no-show), you either see the name as a word, e.g. 'P1', in the
   message line or not. So a viable event message might be:

   P1 P7 P11 P12 P20 P21

   and another might be

   P1 P3 P7 P11 P19 P20 P21

   and so on.

   For a test set, I'm thinking about testing this by adding the
   'expected result' to each line, strip it off before classify or learn,
   then train half of them using TOE or THTTR and then see how well the
   next half fares; kinda like the spam/ham email test sets where you
   train on one series and test with another section.

   I hope I explained it well enough for you to see what's going on.

   I can provide example sets (I checked and this is at least feasible as
   a public test set so future GerH builds might even include it) but
   there's no definite ETA on those because this is prio #2 as patting
   the production box on the head once in a while and writing reports is
   #1.

OK.  Sure, put it up.  I have a machine that can grind on it.

I would do a 10-fold validation rather than just the half-n-half thing;
it's a much more comprehensive test.


   Anyhow: my question to y'all:

   Is using crm114 to kinda 'adaptively correlate' a *severely limited*
   input set to an output decision value another of my more, er, retarded
   ideas (and if so: what's a better way of doing this? I'm kinda running
   around in a crm114 titled rut right now. :-(( ) or is this a perfectly
   good idea?
   Don't hesitate to take out that riot gun and pull!

No, it's perfectly reasonable.

I would also suggest (if you only have 5000 words in the vocabulary
and they are always in a particular order and so will never form
meaningful phrases) to use a smaller statistics file (for OSB, maybe
a "-S 10000" or so) and be sure to use <unique> to turn of phrase
building.


   Thanks for your time,

   Ger


   PS: bonus question: I've been thinking about it and have a glimmer of
   an idea, but are there 'smart' ways to detect that a classifier has
   run into an A XOR B situation (apart from continuing training
   operations being performed)? I will not be able to 'detect' A XOR B
   based on 'LOWER quality pR results' because I don't know where my
   upper bound for achievable pR quality lies, except that I have the
   very strong feeling it won't ever get near 95% - contrary to email
   claims / SPAMTREC benchmarks - as there's just way too much noise in
   the inputs obscuring signal.

Yes.  Several ways.

The easiest (but not quite efficient) is to form a cross product of
words and then use _THAT_ as the input to the classifier.  Example:

Message is:

   P1 P8 P12

Preprocessor result is:

   P1_P1  P1_P8  P1_P12  P8_P1  P8_P8  P8_P12  P12_P1  P12_P8  P12_P12 

thereby allowing the presence (or absence) of each pair to 
become visible to an ordinary, linear, <unigram>-using classifier.

       - Bill Yerazunis


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/