Re: performance

MrC wrote:
> The biggest performance busters in my opinion are:
> 
> 1) copying log files to the temp directory...

Can you elaborate on where in the process this happens and why?

I'd expect everything to be ran in a pipeline, with no temp files except 
maybe to cache the output of the report, so it can easily be discarded 
if no report is generated.


> ...it takes just too long to copy an archive of 100meg email log
> files...

Why is the copy happening?

It appears that logwatch makes no attempt to track log file offsets for 
what has previously been scanned. I was a bit surprised, given that a 
less sophisticated tool (in my opinion), logcheck, does this.

To support processing arbitrary date ranges, you'd want to cache not 
just the byte offset of the last line you processed, but instead create 
an index that maps dates (days) to offsets. A DBM file could be used for 
that. One per log file. With an algorithm to regenerate the index under 
certain conditions, like inode change. (logcheck has some of this.)


> 2) needless use of pipelining to perform basic filtering. 

Doing everything in-process is obviously going to be faster, but I 
wouldn't expect it to be significantly better if you're still asking the 
code to iterate over tens-of-megabytes of log lines that aren't relevant.


One thing I've found a bit puzzling is that both logfile group config 
files and service filter config files can reference filters, though it 
isn't clear what order the filters are ran in. I would have expected to 
see something more like:

Filter = ApplyStdDate | OnlyService foo | RemoveHeaders


Also, it appears that there is no requirement that the output of a 
logfile group follows some standard log line format, yet such a 
requirement does get imposed if you plan to use some of the common 
shared filters. It seems like it might be better to formalize this so 
the data is consistent at certain points in the processing chain.

For example, if dates are always represented in the same way, you can 
then easily build into the framework the mechanism for filtering by 
date. A consistent format also permits the end-user to mix and match 
filters as needed.

(And I'd expect to see logfile group post-filters separated from service 
filter pre-filters by either a naming convention or subdirectories. I 
might also be inclined to call "service filters" "service report 
generators" or something like that.)


> An original goal of this project was to allow any scripting language
> or utility to act as a pre-filter. ... I think its time to just
> accept the fact that logwatch is written in perl, uses perl filters
> and services, and drop the any-scripting language goal.

Being a professional Perl developer, I'd have no problems with that, but 
before I'd vote for such a change I'd want to know how widely used other 
languages are, and exactly what benefits (such as benchmarks showing 
improved performance) would be gained from the switch to pure Perl.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: http://tmetro.venturelogic.com/