Re: combined prob func

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

k b wrote:
> I'm curious about the 'normalScoreMessage()' function in SpamProbe.
> Did you come up with this Graham-like function?

SpamProbe has always used basically the same equations described in A 
Plan for Spam.  My only change to the equation was to take the nth root 
of the ptoducts to smooth out the resulting scores.  Using straight 
Graham SP always wound up with scores of 0 or 1.  It's much more useful 
to have a more evenly distributed score.

Where I deviated from Paul's article substantially was in the 
tokenization algorithm, token sort criteria, top token array size, 
allowing terms to repeat in the top terms array, use of phrases, etc. 
Also I record the total number of times a term appears in all messages 
in the database rather than just the number of messages in which each 
term appears.

There are other differences but those are probably the most significant 
ones.


> What are your general thoughts about the different combined
> probability functions?
> (graham's original, robinson's geometric mean, and your modified graham-like).
> Have you tried the much touted Fisher-Robinson's Chi-Square function?

Back when I first wrote SP (right after Paul's article appeared) 
Robinson had just published his crtique online.  I read his article and 
felt that he was basically complaining that Graham's algorithm wasn't 
theoretically pure enough.  He suggested two alternatives in his 
article.  I implemented one and found that it was horribly inaccurate 
(that's the alt1ScoreMessage() method).  I didn't bother with the other 
method because it required tuning of the constants in the equation based 
on an individual's email.  That seemed very impractical to me.

Since Paul's algorithm worked insanely well with no tuning (and still 
does years later) I decided to follow my engineer's instincts and stick 
with what works rather than what's theoretically sound. :-)

At the time Chi-Squared was not very well documented.  I knew that 
bogofilter and spam-bayes were experimenting with it (and later adopted 
it) but I never found any evidence that it generated substantially 
better results than Paul's equation.  SP holds up well in comparative 
studies so I still don't see any need to change.

One of the speakers at the MIT Spam Conference said that the selection 
of tokens can have a much greater impact on filter accuracy than the 
algorithm used to combine their probabilities.  I think there is some 
truth to that.


> Thanks for a great product!!!

Thanks for the thank you! :-)

All the best,
++Brian

-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.0.4 (Build 4042)

iQEVAwUBQ9RTrDxRyEoJfXIFAQhnJAgAuBvyLjM7Ao5llWGofSBKrjPDEZGuDLyl
hy5YpvQ2KENcC7jgDi8Ah+zaM4VzOcMGBjGCV1RI/95u0bKLUeW8yV1H5ElnQwsu
GE+JF65LejOCIicumFk9pjaQlZ9iqPRqgKfA/UeICqnRYg0Di9Ee7xzYu1LlOIF6
z6uJem8uZBFPS4jSbQBEFzVY4xaDEnYNihV+mnTItb1BkWh3t1UiOAEWuxvEhuiE
C6bUSJfP+nYAXRA/cMxwAdMhmShymRRE/UqcBw2ZQ0t0NFay4Md4T5Whfhf27K9p
+yLLwSTCJqao3pBC9SKVLEJQUdJnYEUnV1iaXzoX1onPA7L5Igif6A==
=hlnL
-----END PGP SIGNATURE-----


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642