Selecting logs based on time range in TAI
Casey Allen Shobe - SeattleServer Mailing Lists <[email protected]> Mon, 2 May 2005 05:04:22 +0000
| Newsgroups | gmane.comp.lib.libtai |
|---|---|
| Organization | SeattleServer, Inc. |
| Message-ID | <[email protected]> |
I currently have my multilog size limit set very high (25MB), and issue an
`svc -a` to /service/*/log/ at precisely midnight each day in UTC.
Also at midnight, a script starts to execute which generates a daily system
report. Amongst other things, it greps and counts through qmail, djbdns, and
other multilogging services' log files, to report counts of email traffic,
failed DNS requests, etc. This is a bit messy right now, because I select
the newest log file not named "current" based on the timestamp, which varies
based on when it was last written. I would also like to have better
functionality, such as the ability to grep through multiple days or just a
couple hours.
I have envisioned a cleaner way of doing things:
1. Set multilog processes to log files up to 10MB in size and keep 15 of
them. A single file may span weeks on a small server, or several files
may contain only a single day on a busier machine. The size and count
should be adjustable as desired.
2. Generate TAI timestamps for a starting and ending time range (i.e. current
UTC midnight and (current UTC midnight - 24 hours) for the daily report.
3. Input timestamps to process which selects appropriate files from the
log directory based on filenames (including current if necessary), then
parses the file(s) and outputs only values within the given time range.
This should be done efficiently as possible, i.e. by intelligently
locating the first appropriate line in the oldest of the selected file(s),
the latest line, and just spitting out everything in between.
With this output I can grep through for the results I want.
Any ideas on how to accomplish #2 and #3?
I plan to release a future version of this script as open source.
Cheers,
--
Casey Allen Shobe | SeattleServer, Inc.
[email protected] | cell 425-443-4653
http://www.seattleserver.com