RE: History of Log Analysis and Modern Search Engine

 
> Since I am feeling pretty warlike now :-) I would like to explore this
> further: why do you think that there is a link between log analysis
> and search engines (apart from whatever historical one)? Searching
> logs, while necessary sometimes, is certainly not a fun thing and,
> just as certainly, not the most effective way to make sense of logs.

Wow, that's a pretty sweeping statement!

> Just today I was helping investigate this fun incident where logs from
> a compromised server  was the only evidence available. Figuring out
> "what to search for" is pretty much a non-starter...

Aren't the vast majority of signature based log analysis and reporting tools
-- swatch leaps to mind -- precisely based on what to search for? Whether or
not it's searching for terms to eliminate, or terms to locate?

For instance, Chris Brenton's article on using grep with iptables logs:

http://www.loganalysis.org/sections/parsing/application-specific/firewall-lo
gging.html#anchorsix

or http://tinyurl.com/2fzuna

So how exactly are you defining "searching"?

Surely on a compromised host, you would at least know where to *start* --
keywords like "panic" or "fatal" or "adduser"; applications returning error
codes; and you'd probably have clues provided by the way in which the
compromise was discovered.

[Disclosure here: I am doing consulting work for Splunk, which bills itself
as a search engine for IT data; and I do *love* the ease with which it's
automated the types of searches and "anti-searches" (eliminating the stuff
I'm not interested in)...I'm *vastly* better at quickly identifying relevant
search terms than I am at writing regular expressions. But that doesn't
change the nature of my position. Whoever is paying me, I would still
consider artificial ignorance techniques to be search methods, tailored to
simplify the job of identifying significant events, as well as
never-before-seen entries.]

cheers - tbird