Re: performance

Mike Tremaine wrote:
> Tom Metro wrote:
>> Can you elaborate on where in the process this happens and why?
> 
> Logwatch does a preprocessing run based on Logfile group to the temp 
> directory. Then services are run against the temp file version of the 
> logs. It saves some processing per run.

What is it accomplishing with the preprocessing? Is it just to 
decompress files, as MrC writes below, or is it doing something else?


MrC wrote:
> Because the archives are often gzip'd or bzip'd, and this was easier I
> suppose;

You can always use zcat to work around the need for temp files.


Mike Tremaine wrote:
> If processing is really causing that much load on a service 
> that taking byte offsets would be helpful then it would be easy to make 
> a preprocessor that could set up the a tmp directory with.
...
> So if you can find a way to really improve performance without breaking 
> cross-platform usage, and not decreasing the inheritant flexibility I 
> say go for it.
> 
> My parting thought is. I have several hosts that have 500mb->1GB maillog 
> files and logwatch still completes in 2 mins nightly.

My test system has tiny logs, so it isn't an issue for me. I was 
commenting on performance due to comments MrC made and mostly because I 
was surprised that an apparently more sophisticated and mature product 
didn't incorporate the kinds of performance tweaks I saw in simpler 
packages.


My own interest lies more in the area of improving the naming of some of 
the config file directives to make them more intuitive, and seeing the 
documentation move into a wiki, so it can be collectively improved upon 
(easily). As a new user to the project, I have a window of opportunity 
to address some of these things while I still see the project as an 
outsider. (I'll start a separate thread on this. Please respond there. 
For any of the items below, feel free to split off a new thread.)

I'm also interested in seeing the "out of the box" usefulness of the 
Debian package improved. For example, I'd like to see it setup to run a 
low detail scan daily (to just report errors), and a medium detail scan 
weekly to report summary information. (Currently it does a medium detail 
scan daily, which just produces unnecessary noise and encourages the 
admin to ignore or delay looking at the logwatch reports.)

And lastly, I'd like to pick up on some of the points I raised on the 
user list about logwatch ignoring log lines that aren't being consumed 
by a service filter, and not reporting errors when configured logfile 
groups have no readable logs. I consider this to be a fairly important 
shortcoming of a tool designed for monitoring.

I'd consider any of these items a higher priority than performance at 
this point. Once they are addressed, I'd consider exploring the idea of 
unifying the intermediate log file format, and creating an indexing date 
range filter.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: http://tmetro.venturelogic.com/