Re: too many false alarms

On Fri, Jan 25, 2008 at 07:36:51AM -0400, Andrew Hay wrote:

> As a follow-up(-on) to Ron's response, I think that the resource
> being accessed also plays a role in determining the "acceptable"
> false alarm rate and the perceived usefulness of an anomaly
> detection system.

Amen.  

Low false positive rate, low false negative rate, low labor/tuning
cost: pick any two.

If the resource you are trying to protect is important, you should be
budgeted appropriately, so you can spend the time to lower both your
false positive rate and your false negative rate.  But you will
probably never tune it perfectly, and will have to use some of your
budget for ops labor to deal with lots of false positives.  With
sufficient staffing, even a high false positive rate is acceptable
(i.e. 99%).

Or maybe the resource you are trying to protect isn't so important.
So if you don't have much tuning budget or ops budget, you can usually
configure the product to be insensitive at a high level, so you have
fewer false positives, but also more false negatives.  [But if the
resource isn't so important, why bother to do this at all?]

If the resource you are trying to protect isn't so important, and you
have lots of ops budget but no tuning budget, you configure to be
sensitive.  Your ops get to deal with lots of false positives, but at
least you have low false negatives.  Better make certain your ops are
motivated and well-trained, though -- they'll tend to ignore the vast
spewage.

If the resource you are trying to protect is important, and you don't
have labor budget to handle events correctly, report this in writing.
If management overrules you, it's probably time to leave, since you
will probably be blamed when the inevitable successful intrusion is
not detected.

- Morty