Selecting logs based on time range in TAI

Casey Allen Shobe - SeattleServer Mailing Lists <[email protected]> Mon, 2 May 2005 05:04:22 +0000
Newsgroups gmane.comp.lib.libtai
Organization SeattleServer, Inc.
Message-ID <[email protected]>
I currently have my multilog size limit set very high (25MB), and issue an 
`svc -a` to /service/*/log/ at precisely midnight each day in UTC.

Also at midnight, a script starts to execute which generates a daily system 
report.  Amongst other things, it greps and counts through qmail, djbdns, and 
other multilogging services' log files, to report counts of email traffic, 
failed DNS requests, etc.  This is a bit messy right now, because I select 
the newest log file not named "current" based on the timestamp, which varies 
based on when it was last written.  I would also like to have better 
functionality, such as the ability to grep through multiple days or just a 
couple hours.

I have envisioned a cleaner way of doing things:

1.  Set multilog processes to log files up to 10MB in size and keep 15 of
    them.  A single file may span weeks on a small server, or several files
    may contain only a single day on a busier machine.  The size and count
    should be adjustable as desired.

2.  Generate TAI timestamps for a starting and ending time range (i.e. current
    UTC midnight and (current UTC midnight - 24 hours) for the daily report.

3.  Input timestamps to process which selects appropriate files from the
    log directory based on filenames (including current if necessary), then
    parses the file(s) and outputs only values within the given time range.
    This should be done efficiently as possible, i.e. by intelligently
    locating the first appropriate line in the oldest of the selected file(s),
    the latest line, and just spitting out everything in between.

With this output I can grep through for the results I want.

Any ideas on how to accomplish #2 and #3?

I plan to release a future version of this script as open source.

Cheers,
-- 
Casey Allen Shobe | SeattleServer, Inc.
[email protected] | cell 425-443-4653
http://www.seattleserver.com