Re: Some questions

   From: Ger Hobbelt <[email protected]>

   For nitpicks...

   Few self-corrects here and an augment:

It's far better than that, and far worse than that, actually.

First off, the noise injection is actually fairly benign since the
noise is uncorrelated; thus it adds with the square root of the number
of passes and centers toward zero.  The spam/nonspam signal, however,
is self-correlated and thus adds linearly with the number of
passes.

Now, the far worse part- we're actually using the wrong formula
(and so is everyone else!).

Consider the actual renormalization- it's Bayes Theorem.  Yes,
I recast it in the quantum format ("add up all the ways it can go")
but it's still Bayes.

Now, one of the assumptions of Bayes is that the events being measured
are _independent_ of each other.  But with respect to features
as generated by the OSB or Vector Tokenizer (VT) engine, that
assumption is completely violated.  

In short, Bayes (renormalization) is the wrong tool to use!  

We compensate for this in part by the local probability formula, which
is heavily biased toward 0.5.  But it's a hack, and provably 
incorrect.  The one saving grace is that it works anyway.

Here's a concrete example: consider a strangely shaped "coin"
that for every ten flips, gives nine "heads" and one "tail".
You'd rightfully conclude that this "coin" has P(heads) = 0.9 
and the expected value of this coin, flipped once, in Bayesian
series, would be Plocal = 0.9

But the actual formula, evaluated in line 1250 of crm_osb_bayes.c,
is (translated to heads and tails; heads = 9, tails = 1, total is 
therefore 10): 

   Plocal = 0.5 + ( heads - (total - heads )) /   16 * (total + 1)

          = 0.5 + ( 9 - (10 - 9)) / 16 * ( 10 +1)

          = 0.5 + ( 8 ) / (16 * 11)
      
          = 0.5 + 1 / 22

          = 0.54545

In other words, we pretend our 90%-heads coin only has an extra 4.5%
"heads".  This keeps us from going too far astray.

Interestingly, this works _better_ than the Paul Graham / Erik Raymond
method of only keeping the most extreme N probabilities, which are
probably fairly independent (for N = 11, 17, 23, in various
implementations).

One thing to try - use an "extremum pipeline" that slides along
in synch with the feature pipeline; at each point, only the one most
extreme value can be used in the pipeline and once used, it doesn't
get _re_used.  Then, set the length of the extremum pipeline to be 
the correlation length of your language (in English, about six or
seven words, which is 24 to 28 OSB VT features in the default
configuration), use a more reasonable probability function like

     Plocal = (in_this_class + 1) / (total_times_seen + 1)

and see how well *that* does compared to the current "use everything
very gently" algorithm.

Anyway... enough for now.

   - Bill Yerazunis

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, & 
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com