CRM114 as a 1-way classifier

[[ Doug - I'm putting this onto the "CRM114 General" list because 
   other people might have a better idea of how to do this.  Doug
   should subscribe to the mailing list, but until then, please
   CC Doug on your ideas (or just "reply all" as I've set the headers
   to include Doug). ]]

Doug Beardsley says:

> Hi,
> 
> I've been looking into using CRM114 as a 1-way classifier (i.e. an
> anomaly detector).  The idea would be that if enough of the input
> matches what has already been learned, then it passes--otherwise it
> fails.  Are any of the classifiers suitable for something like this?
> In practice, I would probably automatically learn on everything that
> wasn't classified as an anomaly and use microgrooming so the behavior
> would evolve as the input evolved.
> 
> I've been playing around with some simple programs using the osb
> classifier and it appears that if I only supply one stats file, crm
> always gives a probability of 1 even when my input looks nothing like
> the stuff I trained on.  Do you know if anyone has used CRM for any
> kind of anomaly detection like this?  I'd really like to be able to
> leverage some of the unique capabilities of CRM's classifiers, but
> apply them to an anomaly problem.  Any help you can give me on this
> would be greatly appreciated.
> 
> Doug Beardsley

The problem is that until you have an anomaly to train on,
you have no real way to know what you're looking for.

Almost all of the classifiers use renormalization, so the 
final (top-level) result will always be "yes".  

On the other hand, if you're willing to hand-tailor to
a particular classifier, you can get somewhere.  

[[ NB: before we get into this: how are you ever going
to _test_ whether this "anomaly detection" actually 
works or not?  Do you have anomalies you can summon
on demand?  If so, train on them with a "leave one
out" algorithm to see if you're on the right track.]]

Anyway, the "status" variable contains a lot more than
pR of the matches; each classifier reports some oft-useful
values on the classification step.

For example, in OSB or Hyperspace, look at the ratio:

    total features in input file / total hits

here, a high value means "atypical".

In Bit Entropy, you will see a similar peak (although
less pronounced) in both:

    entropy / total features in input file
    jumps   / total features in input file

For FSCM (Fast Substring Compression Match a.k.a. infinite-window
LZ-77), the "peak on anomaly" will be on

    chcs / total features in input file

Finally, the L3 or L4 parameters of the bytewise correlation
classifier should also _dip_ strongly on anomaly compared to
the L1 or L2 parameters.  (these are the runlength-weighted
correlation lengths, with L1 being the lowest reward for 
long runs, and L4 being the highest reward for long runs, both
at constant total nubmer of equal characters)

The Winnow, Neural Network, SVM, and the SKS classifiers really need
(as in absolutely need!) two-sided training and so really should be
avoided for anomaly detection.

   - Bill

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july