Re: I need a critique: using crm114 to train on (very) limited word set - or are there better ways?

On Wed, Sep 10, 2008 at 10:29 PM, Bill Yerazunis <[email protected]> wrote:
>   That's where crm114 popped up in the brain: train the sucker based on
[...]
>   predictions once you know what should have been.
>
> Now, _that_ is something CRM114 can do; _any_ of the classifiers can
> do it.

Yup, that's why I was thinking of applying CRM114. Thought it could do
the correlations and stuff for me, saving me a bit of handwork hassle.
:-)

[...]
> Yes, that would work.  Feed in a single word, get back a pR.  That's
> the significance OF THAT WORD ALONE.
>
> BUT - to do this, you must remember that by default CRM114 uses
> OSB phrasing.  You can turn OSB off by the keyword <unigram>, and
> that's what you need to do.

Yeah, I know about the phrasing. Just hadn't realized that I could
extract particular word significance by feeding CRM114 just that word
and look at the pR.

I started out thinking about classifying each message (= set of words)
to one pR (because that's what's most important to me anyhow: can the
bugger be taught to recognize 'it' happening or not this way with
sufficient accuracy, i.e. how often would CRM114 be 'on the mark' once
it's past initial training and give a high-noise input: I can't take
out the noise in the parameters (= words)).

>   cope with an A XOR B input situation.
[...]
> Yeah, that's called a nonlinear separation situation.  Hyperspace can handle
> arbitrary nonlinearities as part of it's design.  The neural net
> can handle a fairly high degree of nonlinearity.  SVM and SKS are
> purely linear (beyond what the short OSB phrases can generate).

Ah, thanks for clearing that one up; I thought NN was the only one
capable of handling XOR.


> OK.  Sure, put it up.  I have a machine that can grind on it.

All right. It'll have to wait though until I'm done with all the
report building business here. Yakking on email is faster start/stop
than getting this out the door (and screened before sending it off).

> I would do a 10-fold validation rather than just the half-n-half thing;
> it's a much more comprehensive test.

Hmmmm... I seem to recall there was some 10-fold validation test
script around somewhere; I might just take that and rip it apart
instead of building one from scratch.
Very good suggestion, thanks!

> I would also suggest (if you only have 5000 words in the vocabulary
> and they are always in a particular order and so will never form
> meaningful phrases) to use a smaller statistics file (for OSB, maybe
> a "-S 10000" or so) and be sure to use <unique> to turn of phrase
> building.

Check. And check. Thought of both. Good to hear I didn't miss those.


>   PS: bonus question: I've been thinking about it and have a glimmer of
[...]
>   operations being performed)? I will not be able to 'detect' A XOR B

Well, not really an answer to the bonus question, because I was
wandering if you could somehow detect at run-time that your choice
classifier was going bonkers over an actual XOR situation it can't
handle (say you're using OSB <unigram>, which doesn't cope with XOR,
and you feed it input where you do not yet know there'll appear a XOR
situation down the road, but it happens and then I imagine it pops up
as continuously changing false pos/false negs which get trained every
time, resulting in a cycle where training NOT(A AND B) cancels out the
training for A OR B and vice versa for as long as the XOR situation
shows up in the input feed. I thought you might possibly maybe be able
to detect it by monitoring hash learncounts going up and warn when a
series of trainings ups particular hash learncounts on both sides of
the fence most of the time: those would be suspects regarding the
hashes representing A and B. (Phew, hope this makes it a bit clearer.)

Anyhow, as an ounce of prevention ...etc., I must say the cross
product is a brilliant idea to prevent A XOR B from bothering the
classifier. Sure, it's got a bit of a blunt edge to it, but I can see
it work and as I have a very limited message length and ditto
vocabulary anyway, this is perfectly doable.
I was thinking about hooking up the VT code into OSB and blow the
gaskets on that one, which would, when stretched alarmingly, end up
with a cross-product producing custom matrix, but this is much neater
as the P1..Px production is a preprocessing phase anyhow and that
phase can do a cross-product when it looks like such is necessary (I
hope not, but I'd rather have the escape route preplanned before going
in, you see).



Bill, thanks for the feedback and the new ideas. :-)


-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/