Re: performance

> MrC wrote:
> > The biggest performance busters in my opinion are:
> > 
> > 1) copying log files to the temp directory...
> 
> Can you elaborate on where in the process this happens and why?
> 

Look at the code block following:

  # Okay, now it is time to do pre-processing on all the logfiles...


> I'd expect everything to be ran in a pipeline, with no temp 
> files except maybe to cache the output of the report, so it 
> can easily be discarded if no report is generated.
> 
> 
> > ...it takes just too long to copy an archive of 100meg email log 
> > files...
> 
> Why is the copy happening?

Because the archives are often gzip'd or bzip'd, and this was easier I
suppose; I'm not sure what the original reason was for the tmp file
creation.


> 
> It appears that logwatch makes no attempt to track log file 
> offsets for what has previously been scanned. I was a bit 
> surprised, given that a less sophisticated tool (in my 
> opinion), logcheck, does this.

This is correct.

> 
> To support processing arbitrary date ranges, you'd want to 
> cache not just the byte offset of the last line you 
> processed, but instead create an index that maps dates (days) 
> to offsets. A DBM file could be used for that. One per log 
> file. With an algorithm to regenerate the index under certain 
> conditions, like inode change. (logcheck has some of this.)

Again, I don't disagree.  Generally logwatch was run in a --range yesterday
mode, and the arbitrary range stuff was implemented later.  The project's
goals have not been to generalize the utility.

> 
> 
> > 2) needless use of pipelining to perform basic filtering. 
> 
> Doing everything in-process is obviously going to be faster, 
> but I wouldn't expect it to be significantly better if you're 
> still asking the code to iterate over tens-of-megabytes of 
> log lines that aren't relevant.

Actually, the larger the file size, the more the savings.  You're still
talking about multiple context switches to push the data through the pipes.
Each pipe-sized read turns into 3 or more context switches vs. none.

> 
> 
> One thing I've found a bit puzzling is that both logfile 
> group config files and service filter config files can 
> reference filters, though it isn't clear what order the 
> filters are ran in. I would have expected to see something more like:
> 
> Filter = ApplyStdDate | OnlyService foo | RemoveHeaders

Yes, this could be a little more clear.

> 
> 
> Also, it appears that there is no requirement that the output 
> of a logfile group follows some standard log line format, yet 
> such a requirement does get imposed if you plan to use some 
> of the common shared filters. It seems like it might be 
> better to formalize this so the data is consistent at certain 
> points in the processing chain.

The scripts and logwatch's development have been somewhat ad-hoc: users
contribute patches and filters to make things work for their systems.
Personally, I don't see a problem here; until there is real-life demand to
solve a real-life problem, such an abstraction doesn't seem necessary (just
yet).  But you bring up a good point in general.

> 
> For example, if dates are always represented in the same way, 
> you can then easily build into the framework the mechanism 
> for filtering by date. A consistent format also permits the 
> end-user to mix and match filters as needed.

This is harder than it appears on the surface I believe.

> 
> (And I'd expect to see logfile group post-filters separated 
> from service filter pre-filters by either a naming convention 
> or subdirectories. I might also be inclined to call "service 
> filters" "service report generators" or something like that.)
> 

I agree.


> > An original goal of this project was to allow any scripting 
> language 
> > or utility to act as a pre-filter. ... I think its time to 
> just accept 
> > the fact that logwatch is written in perl, uses perl filters and 
> > services, and drop the any-scripting language goal.
> 
> Being a professional Perl developer, I'd have no problems 
> with that, but before I'd vote for such a change I'd want to 
> know how widely used other languages are, and exactly what 
> benefits (such as benchmarks showing improved performance) 
> would be gained from the switch to pure Perl.

Given that all filters are called with perl as the interpreter, I think we
can say that all the scripts are perl:

my $PerlVersion = "$^X";

$FilterText .= ("| $PerlVersion $ConfigDir/scripts/shared/$cmd
'$LogFileData{$LogFile}{$_}'" );


MrC