Re: open source artificial ignorance-like systems

To make a long story short, I'll summarize here in a single mail
multiple answers, sorry if I do create entropy but it's better than
firing off a salve of answers :D

Safier, Adam * wrote:
> I have to wonder if anyone has approached the Artificial Intelligence
> crowd with log analysis questions like this. 

Yeah, they have tried each and every algorithm in the book, blindly, on
each and every log sample ever submitted.

> right up their alley.  I would think that LISP might be decent at

Any reason to suggest LISP over, say, any other decent language ?

It's not like you need to use LISP to do AI, or vice-versa... :)

This observation goes also to other posters who suggested LISP or PROLOG
or XML-based-thingies. It's not a matter of the TECHNOLOGY here.

The matter, as MJR correctly puts it, is that we are talking of
(natural?) language processing here, and machines SUCK badly at
processing patterns in languages, in particular when grammars and
semantics are not well defined or not defined at all !

Tom Le wrote:
> The problem with most AI approaches is you have to train the network,
> and in order to do that you have to provide accurate examples (not to

You seem to think that AI means neural networks: just to let you know,
neural networks are a common instrument in data analysis, and are not
even strictly part of AI anymore...

Additionally, you seem to imply here that learning algorithms need to be
supervised (i.e. have pre-labelled samples to train on): this is untrue
as well.

> But when you start talking about host based messages or IDS/IPS events,
> it's just impossible to train the network

Once again, this is your perception, but it does not necessarily
coincide with reality ;)

> Almost all correlation today is either focused on anomaly/heuristics
> type detection or a priori pattern matching of some kind.

I'd say most of the second, and little of the first.

> Pattern
> matching works great, if you believe the vendor has accurate pattern
> matching and can factor in your tuning requirements

Since pattern matching does not, has never, and will never work on
simple intrusion detectors, it strikes me as a bit pointless to think to
solve the problems related with misuse-based intrusion detectors through
misuse-based alert correlators...

> none of the good stuff is open source.

And which of the closed source stuff would be good ? :)

Stefano