Re: performance

Tom Metro wrote:
> MrC wrote:
>> The biggest performance busters in my opinion are:
>>
>> 1) copying log files to the temp directory...
> 
> Can you elaborate on where in the process this happens and why?
> 
> I'd expect everything to be ran in a pipeline, with no temp files except 
> maybe to cache the output of the report, so it can easily be discarded 
> if no report is generated.
> 

Logwatch does a preprocessing run based on Logfile group to the temp 
directory. Then services are run against the temp file version of the 
logs. It saves some processing per run.

> 
>> ...it takes just too long to copy an archive of 100meg email log
>> files...
> 
> Why is the copy happening?
> 
> It appears that logwatch makes no attempt to track log file offsets for 
> what has previously been scanned. I was a bit surprised, given that a 
> less sophisticated tool (in my opinion), logcheck, does this.
> 

Correct each run is taken as new we make no assumptions about what has 
occurred before.

> To support processing arbitrary date ranges, you'd want to cache not 
> just the byte offset of the last line you processed, but instead create 
> an index that maps dates (days) to offsets. A DBM file could be used for 
> that. One per log file. With an algorithm to regenerate the index under 
> certain conditions, like inode change. (logcheck has some of this.)
> 

See above. If processing is really causing that much load on a service 
that taking byte offsets would be helpful then it would be easy to make 
a preprocessor that could set up the a tmp directory with.

Consider this snip

#logwatch --logfile maillog --logdir /tmp --range today --print
system("$logwatch --logdir \'\"$tmpdir\"\' $options --print");

Now imagine having the byte counter you want dumping the logs from the 
counter forward into tmpdir under the same name [so /var/log/message 
gets cut into /tmp/wrapper/messages] then you run logwatch with the 
--logdir /tmp/wrapper and you only end up searching your short logs.

[I do something like that but I also using Syslog-ng + Mysql and some 
other stuff so it super fringe.]


> 
>> 2) needless use of pipelining to perform basic filtering. 
> 
> Doing everything in-process is obviously going to be faster, but I 
> wouldn't expect it to be significantly better if you're still asking the 
> code to iterate over tens-of-megabytes of log lines that aren't relevant.
> 

Run some tests. I thought that also at times but then again the C 
versions of some of the Unix tools are really hard to beat plus since we 
have not used any threading models the piping lets us use multiple CPU's 
with out much effort, or trying to use perl modules that might not be 
installed.

> 
> One thing I've found a bit puzzling is that both logfile group config 
> files and service filter config files can reference filters, though it 
> isn't clear what order the filters are ran in. I would have expected to 
> see something more like:
> 
> Filter = ApplyStdDate | OnlyService foo | RemoveHeaders
> 
> 
> Also, it appears that there is no requirement that the output of a 
> logfile group follows some standard log line format, yet such a 
> requirement does get imposed if you plan to use some of the common 
> shared filters. It seems like it might be better to formalize this so 
> the data is consistent at certain points in the processing chain.
> 

You can also override shared version with custom versions look in

/usr/share/logwatch/scripts/logfiles/<service>

> For example, if dates are always represented in the same way, you can 
> then easily build into the framework the mechanism for filtering by 
> date. A consistent format also permits the end-user to mix and match 
> filters as needed.
> 
> (And I'd expect to see logfile group post-filters separated from service 
> filter pre-filters by either a naming convention or subdirectories. I 
> might also be inclined to call "service filters" "service report 
> generators" or something like that.)
> 
> 
>> An original goal of this project was to allow any scripting language
>> or utility to act as a pre-filter. ... I think its time to just
>> accept the fact that logwatch is written in perl, uses perl filters
>> and services, and drop the any-scripting language goal.
> 
> Being a professional Perl developer, I'd have no problems with that, but 
> before I'd vote for such a change I'd want to know how widely used other 
> languages are, and exactly what benefits (such as benchmarks showing 
> improved performance) would be gained from the switch to pure Perl.

We did make the move to "perlify" all of the scripts BUT I specifically 
wrote the ability to call other scripting languages because that was the 
original intent that Kirk had. Originally it was a mix of bash and perl 
and it seems like if someone wants to write a really cool service filter 
in ruby or python or whatever then they should be able to.

Now the real answers about performance. ;p Logwatch is 10 years old. 
Patches come in from all over and get applied as seen fit. Kirk Bauer is 
the original author of all this and graciously maintains the 
infrastructure for this project but his time for development is very 
limited. There are a few others who roll up their sleeves and push the 
code around as we are able but time being a big limiting factor here 
also no one has said "Wow this needs to be totally re-done." [Although 
Bjorn did that with the sendmail filter for which is gets high marks 
since it was a terribly mess ;)]

So if you can find a way to really improve performance without breaking 
cross-platform usage, and not decreasing the inheritant flexibility I 
say go for it.

My parting thought is. I have several hosts that have 500mb->1GB maillog 
files and logwatch still completes in 2 mins nightly.

-Mike