Re: CRM114 as a 1-way classifier

On Thu, Aug 6, 2009 at 2:32 PM, Tracy Reed<[email protected]> wrote:
> On Thu, Aug 06, 2009 at 10:04:44AM -0400, Bill Yerazunis spake thusly:
>> The problem is that until you have an anomaly to train on,
>> you have no real way to know what you're looking for.
>
> This is exactly why I have not implemented crm114 for analyzing system
> logfiles yet even though I have wanted to for ages. I only know what I
> don't want to see. There are all kinds of errors which my various
> systems have not yet logged but certainly will in the future.

I definitely understand this problem, but it's still possible to come
up with with an idea of what's normal and what's not without having a
human-specified categorization.  Z-scores in statistics do just that.
Anything more than...say...two standard deviations away from the mean
is considered an outlier.  So instead of building a corpus of what
anomalies look like, you set a threshold which carries an implicit
definition.

Now admittedly this is much more simplistic and limited than a system
that can learn specific characteristics of anomalies, but it still has
some use in certain situations--hence the study of unsupervised
learning algorithms.  A k-means approach is a fairly obvious extension
of this idea where "anomaly" equals, "not close enough to any of the
clusters developed so far".  I was wanting to leverage some of
CRM114's infrastructure and existing algorithms--specifically SBPH and
hyperspace sound promising--to avoid reinventing the wheel (at least
while in the proof of concept phase).

> [[ NB: before we get into this: how are you ever going
> to _test_ whether this "anomaly detection" actually
> works or not?  Do you have anomalies you can summon
> on demand?  If so, train on them with a "leave one
> out" algorithm to see if you're on the right track.]]

Well, that's a good question.  At first, testing would be very
informal.  If I have some success, then the work required to compile
test data sets might be justified.  And of course, at that time, it
may or may not be useful to leverage the supervised aspects of CRM114
along with the test data.

> Anyway, the "status" variable contains a lot more than
> pR of the matches; each classifier reports some oft-useful
> values on the classification step.

I've been playing with hyperspace and am already extracting the
radiance value.  I'll look into some of the other fields in the status
variable for other classifiers and see what I can come up with.  I was
mainly interested in finding out if anyone else has done this kind of
thing with CRM114.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july