Re: I don't get it anymore (if ever): OSB pR increases linearly with message size. Is that by design?

Thank you very much for the explanation. I had completely overlooked that.

>   PS: Same phenomenon for bigger messages. I found it because the
[...]
> That sounds like a bug.
>
> Any suggestions?

Not yet. I'm thinking. Got to get my 'thought model' straightened out
first, before I can trust any 'fixes' I come up with.

The problem is twofold: I think I can explain why this also happens
for bigger messages: if a three-word message 'A B C' has pR P1 and 'A
B C A B C' gets pR P2 ~ 2*P1, then that is caused by the 'counting'
done in the classifier code; I'd say then that the length of the
message doesn't matter; just the fact that the 'duplication' _always_
means HALF your features will be dupes so completely correlated, will
hence always cause the same effect.
 <unique> would sensibly work for messages of any length as it would
remove any glaring correlation in the features.


What worries me as well (if not more) is the second fact I mentioned
in passing. That is:

learn('A B D', bad)
learn('A B C', good)
classify('A B C') -> c1
learn('A B C', good)
classify('A B C') -> c2
learn('A B C', good)
classify('A B C') -> c3

does not significantly 'push' message 'A B C' into the 'good' zone
(incrementally with each 'learn')): pR value c3 is quite close to c1.
Keep this up a few hundred rounds even and you still are stuck with a
relative low pR (progression is asymptomatic, as I'd expect, just not
the asymptote value I wished for ;-) ). This is because in the code
the two cpcorr[] correction factors pressure my 'good' Ps down (and
lift 'bad' up) as much as I can train them (because cpcorr[] assigns
global weights based on the number of trainings in each CSS); so any
added 'importance' I give my features by additional learning (which
runs up their learncounts correctly) is (almost) lost.

I have been thinking about adjusting those learncount-per-feature
based weights, based on their occurrence in _both_ camps, but not sure
about it. Too little clarity about what is _really_ going on
math-wise.

I fear this is another error of mine due to (over)simplification of my
thought model and that really bothers me.



I'm working on a constructed testset with known features (and
differences) so that I can check this more thoroughly.




On Fri, Oct 3, 2008 at 2:35 PM, Bill Yerazunis <[email protected]> wrote:
>
>   From: "Ger Hobbelt" <[email protected]>
>
>   [[ description of repeated examples in a text causing increases
>      in pR elided ]]
>
>   So where's my error?
>
> The problem you are seeing is the Bayesian Assumption being violated.
>
> Bayes theorem says that you can do renormalization to find the
> overall probability of something being true when you have a bunch of
> independent tests you can run.
>
> HOWEVER... note the word "INDEPENDENT".  Words in human languages
> are not independent.  They are highly correllated.  This
> breaks Bayes' rule.  (the coherence length is about six words).
>
> That's the whole principle behind Markov and OSB filtering- to
> take advantage of the correlation between words.
>
> One way to partially fix this is to use the UNIQUE keyword; with
> this turned on, the system does not consider repeated identical phrases
> as independent evidence.
>
> However, the slightest change in a phrase (just enough to make a
> change in the hash) will let the second phrase pass through the
> UNIQUE test.
>
> Some other filters get around this lack of independence by only using
> a small number of "significant" words (like 7 or 13 words) which are
> hopefully far enough apart in the text stream to be independent
> events.  However, those filters throw away huge amounts of evidence
> in the common, less significant words (and thus typically don't
> do as well in spam filtering accuracy).
>
>
>   PS: Same phenomenon for bigger messages. I found it because the
>   mailstripper in mailreaver is _quite_ different from the one in
>   mailtrainer, resulting in mailtrainer working on basically
>   'duplicated' messages, resulting in doubled pR scores used to
>   determine THTTR training regime. Which ended up in mailtrainer NOT
>   training messages (or stopping the repeat cycle for them) while
>   mailreaver itself would still incorrectly classify them when using the
>   same threshold values. Gives me the heebie-jeebies for sure.
>
> That sounds like a bug.
>
> Any suggestions?
>
>    - Bill Yerazunis
>
>



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/