RE: Syslog - monitoring the bigger picture

> > ...lather, rinse, repeat.
> 
> So, Tina, if there were a modularized and extensible open source app
> to do log analysis, folks will stop this hamster wheel of
> pain^h^h^h^hembarassment and develop modules for this mythical
> application and not their own scriptlets?

Probably not...because there will always be *some* people who think it's
easier (or more fun! if we stick to the hamster wheel analogy) to roll their
own than to learn someone else's.

Also note Marcus' experience with NFR...the way I understand it, NFR was
originally open source (for some def of open source) and there was an
expectation that "the community" would contribute their customizations, but
although NFR got used in a lot of places, very few people were willing to
post their own work. Some small amount of that may have been reasonable
fears about confidentiality; but a lot of it was just BS in my opinion.

So the model of community contributions of the type you imply may itself
have issues, although we can't really know beforehand.

But your question brings up another one of my favorite rants (having Marcus
on his regex soapbox has inspired me) -- one of the reasons we get all these
little tools is that the folks starting out to deal with this issue have
*so* little guidance to look to. That is, there's no where they can turn to
for learning what kinds of messages are important, what kinds are less
important, even what categories of messages or severities they should be
thinking about -- much less warning them about the pitfalls inherent in
believing what man pages tell you about "levels"...or even to tell them to
start small, with one or two servers, rather than immediately overwhelming
themselves by starting with the idea that syslog lets you get all the data
in one place.

Which is why my contribution to the "how do we solve this problem" always
comes back to building that kindergarten-level knowledge base of which
messages are important, for what reasons, on whatever kind of
device-OS-application you may be considering. 

[Disclosure: I am currently working for Splunk as a content developer for
SplunkBase.]

My *main* reason for enthusiastically writing content for SplunkBase (public
content, mind you, licensed under the Creative Commons) is that they're
paying me to build the knowledge base -- unlike Marcus and I trying to do
the same work in our off-hours with loganalysis.org...so that I am
contributing (and helping to guide, I hope) a repository that contains info
on how to get the logs, how to tweak the logs, where to go to figure out the
logs, all that good stuff. Not to denigrate the other folks who are building
similar sites...but my own personal experience suggests that having *some*
financial backing for the effort makes results a lot more likely.

This way, when that poor junior person gets started, there's a bunch of info
*out* there that just might help them get a better handle on what they want
to *do*, and then help them figure out how to do it.

[And lest anyone wonders: I will try to continue to maintain the list of
open source and commercial log parsing tools on loganalysis.org, so newbies
have a list of places to start...and I'll also be maintaining a list of
sites that are designed for the collection and explanation of logs; I think
I posted my current list here a while back.]

One of the things that's been a continual frustration to me, in maintaining
that list of parsing tools, is that I've had little luck getting the authors
of the various programs to tell me exactly what you asked: what
distinguishes your bit o' code from the other bits o' code out there? That
would be hugely beneficial. But I think a lot of programmers think that's a
little too close to documentation for comfort ;-)

enough for now -- tbird