standardized intermediate log file format

MrC wrote:
>> Of course now that we've optimized this, I'd like to voice my 
>> agreement with a comment made recently that dates and other "header" 
>> information should probably not get stripped off before handing the 
>> data to the service filter. As I noted in another message, having a 
>> common intermediate log format would permit building an infrastructure 
>> for doing common things, like splitting a line into a standard set of 
>> components (date, host, service, PID, message), making the job of the 
>> service filter just as easy, wile still giving it access to the 
>> greater detail if it needs it.
> 
> Great!  Let's see what we can come up with. 

I'm not entirely convinced that a unified intermediate format is the way 
to go, but I think the pros and cons are worth discussing.

I see logwatch already has an infrastructure set up to handle "drivers" 
for log files by placing filters in scripts/logfiles/<service>/.

Some obvious negatives to this approach include:
  -Greater complexity. Yet another file/script.
  -More processing overhead to convert formats. (Do we cache the 
converted output?)
  -Complications trying to shoehorn every conceivable log format into 
one unified format.

On the last point it isn't quite as complicated as it seems, as for our 
purposes we're only interested in identifying certain components in the 
log line, such as the timestamp, host, service, and then everything else 
gets lumped together as the message.

Another option to consider: instead of actually reformatting the log 
lines, the driver could merely provide an API for decomposing the 
foreign log line into standard elements. So for example:

while (my $line = $logfile->next) {
     if ($line->service eq 'dovecot') {
        ...

The date range filters, *OnlyService filter, and other filters, as well 
as the service filter would make use of this API to interact with the 
log lines. The entire original line would be preserved ($line->raw?) and 
available for the service filter to work with, if it wanted.

Though this is likely to be the worst option for performance. If the 
processing chain has multiple filters, the log line may end up being 
parsed multiple times. It also will be complicated to use the existing 
mechanism of separate processes connected in a pipeline, as the above 
really needs to permit passing an object to the report generator. (If 
the whole point is to have one report generator that can handle multiple 
input formats, then you need to be able to select the right driver for 
your particular format external to the report generator.)

A more OO approach opens up other possibilities as well. Consider the 
discussion of "logical blocks" in the documentation for Logrep, another 
log monitoring tool:
http://www.itefix.no/phpws/index.php?module=pagemaster&PAGE_user_op=view_page&PAGE_id=22

   ...one block corresponds to one line. However, there are some
   exceptions like postfix and sendmail files with multi-line blocks, and
   syslogs with last line repeated messages that contain many blocks in a
   one single line...

If you can store metadata about a log line, then you can do things like 
store a repeat count, or link multiple lines together representing a 
mail delivery transaction, and have the report generator treat the block 
as one item, instead of dealing with individual lines. Of course the 
report generator is free to do that kind of grouping itself. And pulling 
off something like this would require either a lot of memory, a more 
complex intermediate format with persistent storage, and/or multiple 
passes over the data.

I think I'd rather keep things on the simpler side, and resist the 
temptation to create something so complex that it starts to feel like 
the next logical step is to pipe all the logs into an SQL database.

Lastly, aside from the benefit of being able to mix and match various 
intermediate filters, the biggest advantage to the unified format is 
going to be for services that support multiple log file formats. Any 
service for which logging both through syslog and some other format 
isn't unusual. That may be a limited set.

Although there may be some side benefits, like moving platform-specific 
logic out of the report generator and into a file format driver (as is 
currently done with scripts/logfiles/cron/applydate to translate Solaris 
format to syslog format) or converting between the formats used by 
different versions of the same application (like Dovecot 0.9 to Dovecot 
1.0 format).


> I'd like to see:
> 
> 1) standard date 

I'd recommend using one of the ISO 8601 profiles:
http://en.wikipedia.org/wiki/ISO_8601

Probably the version with no punctuation.

Any of these formats (as long as you stick with one variation) provide 
the advantage of being fix width, and can be parsed with Perl's unpack().


> (syslog logs don't include year, vsftpd does, and clamd 
> host no time stamps at all.)

Yeah, the driver would need to incorporate heuristics for generating 
missing information. Perhaps getting the year from the file's timestamp, 
with some windowing logic to avoid picking the wring year on a file 
rolled in January, but containing December's data.


> 2) hostname

Not always present, so same as above.


> 3) service w/PID (amavis has some non-standard entries, essentially 2 
> ids').

Are the PIDs ever used by any logwatch filters or report generators?

If they are, consider making them a standard component, even if optional.


> 4) log line remainder

So something like:

20070628T110500Z:host:service:PID:message

I'd benchmark parsing of this with split() against a version using fixed 
width fields (for everything but message):

20070628T110500Zhost      service   PID   message

being parsed by unpack().


Mike Tremaine wrote:
> Probably the easiest way to start would be to knock of the syslog 
> standard and add a new config variable to the logfile group conf that 
> tells it to validate using "syslog" then add other validations as they 
> get worked out.

I wouldn't call these validations. The log format drivers should 
incorporate validation logic, but that's a side issue.

I expect 'syslog' to be the default driver. Although if we're 
maintaining backwards compatibility, the default would be to pass 
through the raw data to the service filter.


> Check out the Cron applydate to remind us of the horrors lurking in the 
> system. ;)

Thanks for the pointer. It doesn't look much worse than the format 
translator I created for Dovecot.


> Samba and Apache [http] are pretty big services that are also not Syslog 
> formatted. These should provide you with pretty examples of the what it 
> is going to take.

I'm more concerned with log formats that have records spanning multiple 
lines (like the MySQL query log). I think those are going to pose the 
greatest challenge to attempts at wedging them into a syslog-like format.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: http://tmetro.venturelogic.com/