Re: I don't get it anymore (if ever): OSB pR increases linearly with message size. Is that by design?

   From: "Ger Hobbelt" <[email protected]>

   [[ description of repeated examples in a text causing increases
      in pR elided ]]

   So where's my error?

The problem you are seeing is the Bayesian Assumption being violated.

Bayes theorem says that you can do renormalization to find the 
overall probability of something being true when you have a bunch of 
independent tests you can run.

HOWEVER... note the word "INDEPENDENT".  Words in human languages
are not independent.  They are highly correllated.  This 
breaks Bayes' rule.  (the coherence length is about six words).

That's the whole principle behind Markov and OSB filtering- to 
take advantage of the correlation between words.

One way to partially fix this is to use the UNIQUE keyword; with
this turned on, the system does not consider repeated identical phrases
as independent evidence.

However, the slightest change in a phrase (just enough to make a 
change in the hash) will let the second phrase pass through the
UNIQUE test.

Some other filters get around this lack of independence by only using
a small number of "significant" words (like 7 or 13 words) which are
hopefully far enough apart in the text stream to be independent
events.  However, those filters throw away huge amounts of evidence
in the common, less significant words (and thus typically don't
do as well in spam filtering accuracy).


   PS: Same phenomenon for bigger messages. I found it because the
   mailstripper in mailreaver is _quite_ different from the one in
   mailtrainer, resulting in mailtrainer working on basically
   'duplicated' messages, resulting in doubled pR scores used to
   determine THTTR training regime. Which ended up in mailtrainer NOT
   training messages (or stopping the repeat cycle for them) while
   mailreaver itself would still incorrectly classify them when using the
   same threshold values. Gives me the heebie-jeebies for sure.

That sounds like a bug.

Any suggestions?

    - Bill Yerazunis

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/