Re: Mixed 64-bit system GerH binaries / BillYscripts --> two-sided training? YES!

dang. 550 This message matches a blacklisted regular expression
(pharma ban) (state 18). :-( Okay, find&replaced the bloody blue pill.
Hoped the system was using CRM114 instead of brain-lobotomized
hardcoded rules, but no such luck.

---------- Forwarded message ----------
From: Ger Hobbelt <[email protected]>
Date: Sun, Aug 31, 2008 at 4:27 PM
Subject: Re: [Crm114-general] Mixed 64-bit system GerH binaries /
BillYscripts --> two-sided training? YES!
To: [email protected]


Oh boy!

two-sided training, Bill? As in the API def: "you have to specify
*all* CSS files in both your 'classify' and 'learn' statements?"
EXACTLY what I dearly want to see in CRM114. (a) it's a congruent
design (which always scores points with me) and (b) it'll allow me /
others to test a few really nasty ideas regarding the OSB[F]/Markov
classifier group: because I have VERY little time for this now, here's
the general idea:



Goal: idea to improve (I hope!) the OSB/Markov-based classifiers by
reducing the number of 'noise learn entries' in the CSS files. Noise
reducting ==> better and more 'extreme' pR values.



Why: You'll see it happen with email too, but with my own datasets
it's really terrible (and caused me to drop CRM114 for the 1.0 release
of my proprietary toolset, which should go 'live' tomorrow morning
(ironing out some bugs that make you turn grey at 40 right this
minute).
What I saw was that for 'common words', each of the two CSS files gets
a hash entry-plus-learn-count - as you might expect.
Multiple rounds of training will +1 that counter in one CSS file each
round. This message's training it'll be left, the other message's
training it may be the right one (I don't work with spam, so for me
it's left/right instead of ham/spam). Nothing new so far.
However, 'bad words' also occur on the 'good side' and vice versa
(think: you get email from a friend as he sends you a [blue pill] joke
(bang! '[blue pill]'! right there! bad word!) and for some reason the
message gets trained: +1 for a bad word on the 'good side'! Yikes.
Which will pull pR towards zero (== 'I dunno' says CRM114) on
subsequent classify runs. (Yes, yes, extreme case, only happens when
occurring multiple times, but shut up, it's happening to me all the
bloody time and the point is not pR=0.0 but look at the DIRECTION this
action is taking you: it's a 'pull' _towards_ 0.0.)
Remember that I use different 'messages': few words each, so this is
far more significant for me than for the email folks, I assume.
Anyway.
'Common words' like 'a' or 'the' appear in both 'bad' and 'good'
messages, so get trained on both sides of the fence: if you use the
system a while, training continues, so the 'count' for the word 'the'
in both left and right CSS files gets higher and higher with every
darn training. Yes, the numbers pan out - IFF you have a balanced
training method == train as much spam as ham (where did we hear THAT
one before, eh?) AND the number of messages (better: 'occurrences')
with 'the' in them is about equal for left/right (ham/spam).
So some may say: yeah, what's the issue? The issue is that high counts
for 'the' will have an ever so slight effect on pR, again pulling it
towards the 0 midpoint. When you have *small* messages, it gets worse
fast.
You think this doesn't count for you? Well, think about what will
happen when you do UNBALANCED training and each message has the word
'the': after several trained messages the system will TELL you the
word 'the' is BAD (assuming you are training more 'bad' than 'good'
messages, like it can easily happen in Email Union country. Which is
VERY worrysome because 'the' is a very NEUTRAL word and shouldn't push
pR towards either side, good or bad. To compensate for this, you'll
have to train more 'good' mail than strictly necessary, just to ramp
up the weight for 'good words' and 'common words' on that side. Which
ends you up in a kind of 'balanced training' regime, but it also means
you have to train more 'unknowns' to be good then the theoretical
minimum and manual effort to click and have it trained also results in
frustration. Anyhow, the number of training rounds is higher than
really need-be, which is another very subtle can of worms when you
work with (huge!) crazy feeds like me.
 Ger's Lemma #X+1: More Training = Faster Aging.
And I want my CSS files to stay young and frisky.

When you look at such CSS files and 'read' the files, you can say that
in both CSS files the common word 'the' has an extremely high 'weight'
(the 'count'!) which only 'equals out' if your classification takes
the DIFFERENCE between left and right weight of a word[*] AND MOST
IMPORTANTLY: you stuck religiously to the 'balanced training' dogma.
[*]= (does _your_ classifier do that? Check algorithm! Check code.)
I did not and I cannot: training with my data is fully automated: I
can say with 100% certainty if a message X *was* good or bad and the
system can then train CRM, but for my data type 'manual training' is
ludicrous: compare to a SINGLE mailbox receiving over 1M messages a
DAY, where future events (say a few weeks after 'today') will tell us
if we SHOULD HAVE BEEN looking for 'signal' or not; here 'signal' can
be read as 'ham', where anything else can be considered 'spam'. In
other words: I will know if it's 'ham' in a few weeks; can't see this
by just 'reading the text'. It's bloody hard to explain and I won't
take up your time doing so, don't want to either, but the fact of the
matter is that OSB et al are gathering hashes which should be
discarded ASAP: millions of hashes which prove to be 'common words'
(like the 'the' word) as they accumulate close-to-equal counts in both
CSS files in a controlled environment where I can stick to that
'balanced training' dogma.

Suffice to say I have been wrecking my brain how I could adapt CRM114
OSB et al to get rid of these buggers PRONTO.



The 'solution' (which will require extensive field testing, but I can
hope, can I?):

First of all, I need the training code to 'know about' the COMPLETE
set of CSS files - as they are fed to the 'classify' statement as
well. Right now, 'learn' only gets to see ONE of the buggers. Dang!

Because what I want to do is this:
( and the thought behind it is: "prevent CRM114 from pulling pR
towards zero over time" == find a way to delay 'aging' of the CSS
files as they gather 'wisdom': this may be a way to NOT have them end
up like the Zen master said to the apprentice: "gather enough
knowledge and you will learn to know that you know absolutely nothing
(pR=0.0)" )
when training a 'side' (left or right // spam or ham) and I get a hash
( a 'word' in CRM114) usually I would just +1 its count on this side.
That's the current code.
New code: first check if the hash ('word') exists ON THE OTHER SIDE:
if it does, DECREMENT (-1) the weight OVERTHERE (and we're _done_ for
training this hash!). Otherwise (= the other side doesn't know the
hash == has weight zero on the other side) do the usual: +1 overhere.

How do I expect this to work out, keeping in mind that 'unbalanced
training' I was yakking about?
Well, THIS way, a 'common word' will not get a chance to reach high
count=weight values. You can easily see that happen for BALANCED
training, right? Well, it works for UNBALANCED training as well (me
thinks ;-) ) when you apply repetitive training methods like THTTR
(Thick Threshold Training) because your training loop counts will
'shift' towards a behaviour where 'common word' hashes will decrease
and increase weight until a new equilibrium is found. My EXPECTATION
is that thanks to the 'decrease one on other side' approach, the
result at each moment T in the production lifeTIME of the system will
be a 'rather low count=weight' for common words, ALL of the time, as
new messages' training rounds 'pull down' weight that has been
collected during previous training previous messages. And low weight
for 'common words' is a HUGE benefit for ME, as the 'really bad words'
like '[blue pill]' will keep on collecting weight, despite that one-time
'-1' decrease as you OKayed that [blue pill]-joke message from your friend.

Bottom line: this way I think I can ditch a LOT of 'noise word
weights' from my CSS files and when this idea works for me, it should
work for email and a lot of others as well.

Notice the 'side effect' that the
'decrease-on-other-side-before-increase-here' approach will thus
ensure that each hash=word will !! only exist in one CSS file !! at
any moment of time. (Which is different from the current
code/algorithms in CRM114 where you will find 'common word' hashes in
BOTH CSS files.) --> The technical savvy folks will quickly see an
additional _technical_ optimization here as you can dispense with two
CSS files and make it one. But I digress.



I wrote this now because I do not see when I'll have time to code&test
it myself and maybe someone else does find that time. If the idea is
stupid/flawed or otherwise lacking, I'd like to hear about it, because
without a 'noise reduction system' like this, classifiers are
currently NOT production-useful for my purposes.

I do not 'grok' SVM/SKS/NN/BE at a sufficient level of detail to see
what I should do there, but suffice to say that they didn't help me
yet. And, yes, my data feed is HIGHLY unusual and VERY quaint in a
zillion ways and there's a whole _chain_ of issues there that must be
resolved before you can even consider using a statistical classifier
_at_ _all_, but suffice to say that it does work well for me with a
sorta 'hard-coded CSS'-like approach. (huh?! 'hard-coded'? think like
this: 'prepare two CSS files' and then use those for classification.
NO training ANYthing ANYtime. Just classify. How you 'prepare the CSS
files' is up to you.  No need to discuss it, as it is VERY SPECIFIC to
my workfield. If you want an analogy, think 'weighted email rules'
like spamassassin had them at the very beginning: "if email header
contains '[blue pill]', count +1 for the bad guys" kinda rules. Check the
rules. Count the weights. Get your pR verdict and off you go.)

The precieved benefit of using statistical / trainable filters for me
was the idea that it would make the system
'auto-tuning'/'auto-adjustable' as I can backpropagate 'knowledge'
into the system in full-auto. Kinda like this: you're SETI listening
for extraterrestial signals and you just don't know what to look for
at first, but you get visits from Aliens who say "hi! Didn't you get
my phone-call last week? Oh, you didn't? Well, here I am!" and your
local geek can run to the listening box crying 'oh shit!' and check
the log, then feed the missed phone-call bit to the listen device so
it'll know what to look for next time Bubba from Sagitarius IV comes
by for a cuppa. With my feed, it's no sure thing Mr. Alien will visit
me (some aliens don't like it when you don't check the answering
machine and call back next day), but the number of visits which I *do*
get are frequent enough AND as far as I can tell all the aliens show
at a limited number of K-Marts for their phone equipment too. And my
task is to report 'phone calls' to the authorities because there's
nothing like a flustered politico who couldn't make it to the
'welcoming reception' because _you_ just screwed up by not telling him
a VIP is coming by for a 'press moment' next week. (Imagine my
survival rate on the Obama or McCain team. Exactly, _not_ _good_ _at_
_all_...) Anyhow, when I _get_ a visit, the fella always consistently
tells me when he/she 'phoned ahead'. From there, thinking about
'full-auto training' is a very small step for mankind. And meanwhile I
get a raise as it's minus one IT geek on the salary expense account
and you all know what _really_ makes the bean counters go 'oooooohh!'
and 'aaaahh!' like they just got a private cigar moment with Monica L.



Anyway, already spent too much time writing this up in hopes someone
might say 'hey! that's somethin' you got right there!' and write it
for us/me!, so I don't have to wait till I discover the end of the
rainbow where I get time to develop this idea into working code plus
test results. Back to my bugs. <Here, critter critter critter. Got
some nice little poison dart for you!> <Here, ....>



PS: I'll listen in and push for a bit of time to yak about portability
and coding details y'all shouldn't be bothered about in an ideal world
when the need arises. But not now. Try me around next weekend. If the
launch goes well this week, I'll have a bit of time. (please, Lord,
Jahweh, Allah, Shiva, Maria, and all you other mightily awesomely good
'n great fellas and fellettes up (and down) there, care to spare a
nickle of good fortune on me, please? Thank you kindly, sirs and
ladies!)


--
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/