I need a critique: using crm114 to train on (very) limited word set - or are there better ways?

Ok, what I think I really need now is a kick in the head, because I
have a hunch I've been focused so much on the crm114 hammer that every
problem starts to look like a 9 inch nail and my local math
wizzards[*], well, they do qualify.

[*] No spelling error. Who's read Terry Pratchett will know what I
mean. Wizzard ~ expert in escapology. Don't need Reebok to break speed
limits in getting away from thorny issues.


What's this about?

Something I like to try as I guess it might work out. I'm not looking
for a 100% solution but for the 80/20 that works 'well enough'.

Let's say you've got a set of parameters, which I like to call 'words'
for the sake of crm114. Together, they form a 'message'. One message
per event. And an event is either a hero (ham?) or a zero
(spam/noise). Each parameter is of the on/off kind, so it's either
there (on) or it isn't.
As I realize that OSB and friends detect 'similarity' by actually
looking for 'word identicalities' and checking how much of them there
are, I am sure crm114 will be able to recognize the words, so no
problem there.

Now for the fun stuff. The number of parameters (a.k.a. 'words') in
all possible messages is known (requires a bit of work though) AND
severely limited. To talk numbers: I'm pretty sure I've got a
'vocabulary' of more than 100 words, but I am also pretty sure I'll
never surpass the 5000 words upper limit. In email analogy terms, that
would mean I know the lower and upper limits of the number of
*different* words (no l33t spellings and other tricks for this guy!)
PLUS I know that each message will:
a) only contain each word ONCE, and
b) only contain a limited SUBSET of all the possible words, and
c) all words will appear in a predetermined, FIXED ORDER in each
message. (Did I hear a <unique> there? well, maybe not.)

In other words: each message can be defined as a sequence of all words
W(i) where index i = 1..N and N is the number of possible words, where
each 'word' W(i) may or may not appear.
(That's pretty limited, compared to your usual email input, where you
have multiple occurrences, undetermined order, etc..)

Some words are interrelated, some are not. Which ones are
interrelated, I do not know up front. When I'm done, I probably still
won't know.


What I am after are two things:

1) Find out /if/ and when if, /which/ words are the 'significant' ones
that have the most influence on the output.
2) With any luck regarding (1), a hint at /how much/ significance
(weight factor) or something like that might be nice.


What's the [desired] output?

I'll know that after some (significant) delay, so I will be able to
tell if an event was 'it' or if it was a 'nit'. Thus I want to
discover if a particular message has the probability of being 'it'
instead of 'nit', preferably when the event occurs.
That's where crm114 popped up in the brain: train the sucker based on
older event series where you know who's 'it' and who is not; then
continue feeding it event messages like that and see what pR says
regarding the probability of an event being 'it' nor nit. You can
improve (I hope) on the fly by backpropagating (training) wrong
predictions once you know what should have been.

I realize this sort of thing could theoretically be done as a
multi-variable correlation, but I am kinda lazy and I think crm114
might be able to do this for me, and I like to hear if I am smoking
Afghan or Dog here. ;-)
Especially that 'on the fly' bit seems appetizing to me as I don't
think I'll have sufficient data to start with and sifting through
hundreds of parameters (a.k.a. words) to find out which of them are
significant, while a whole bunch very probably are not, is on my most
wanted list of things to do, so I think that if I grokked crm114 well
enough that I should be able to let crm114 do this by simply feeding
it lines-as-messages in a nice script and then have a look at the pR
coming out the rear end for each and regarding significance of certain
parameters then: it should be present in the form of the weight
numbers in the CSS files (OSB[F]) per hash (= word) or otherwise and
from there on out it's: no worries, mate!

Some additional details that matter (I believe):

the current 'general idea' about the matter is that some parameters
will certainly show to 'strengthen' each other (i.e. a kind of A AND B
operation), while a few may independently lead to the same result (A
OR B), but we also think that the chance that certain parameters
actually 'cancel each other out' is low (A XOR B) - which is important
as I understand that the classifiers (apart from NN? SVM? SKS?) cannot
cope with an A XOR B input situation.
[The only way I can see a XOR happen in my mind when Markovian or
other classifier picks both A and B in the same chain to produce a
single hash which could counterweight the learncounts on the A hash
and the B hash. I don't grok SVM et al yet, so can't see anything
there. And maybe I still don't grok Bayes/Markovian either... I'll
find out now.]


For some tech detail: every parameter has a name and since it's on/off
(show / no-show), you either see the name as a word, e.g. 'P1', in the
message line or not. So a viable event message might be:

P1 P7 P11 P12 P20 P21

and another might be

P1 P3 P7 P11 P19 P20 P21

and so on.

For a test set, I'm thinking about testing this by adding the
'expected result' to each line, strip it off before classify or learn,
then train half of them using TOE or THTTR and then see how well the
next half fares; kinda like the spam/ham email test sets where you
train on one series and test with another section.


I hope I explained it well enough for you to see what's going on.

I can provide example sets (I checked and this is at least feasible as
a public test set so future GerH builds might even include it) but
there's no definite ETA on those because this is prio #2 as patting
the production box on the head once in a while and writing reports is
#1.


Anyhow: my question to y'all:

Is using crm114 to kinda 'adaptively correlate' a *severely limited*
input set to an output decision value another of my more, er, retarded
ideas (and if so: what's a better way of doing this? I'm kinda running
around in a crm114 titled rut right now. :-(( ) or is this a perfectly
good idea?
Don't hesitate to take out that riot gun and pull!


Thanks for your time,

Ger


PS: bonus question: I've been thinking about it and have a glimmer of
an idea, but are there 'smart' ways to detect that a classifier has
run into an A XOR B situation (apart from continuing training
operations being performed)? I will not be able to 'detect' A XOR B
based on 'LOWER quality pR results' because I don't know where my
upper bound for achievable pR quality lies, except that I have the
very strong feeling it won't ever get near 95% - contrary to email
claims / SPAMTREC benchmarks - as there's just way too much noise in
the inputs obscuring signal.




-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/