Re: Dovecot

Over on the user list I wrote:
> ...I was surprised not to see a section in the filter where it had
> regular expressions for matching lines to be ignored...

In the latest version of the filter, I did find one regular expression 
that was for matching lines to be ignored.


> ...I have Dovecot's IMAP service logging to a non-syslog log file 
> to keep that clutter out of the mail delivery logs, and Dovecot's native 
> format is quite different from syslog:
> 
> I assume a filter needs to be constructed to transform this into
> syslog compatible format...
> Has anyone created such a filter?

Apparently not. I've written a draft filter, which I'll submit shortly. 
(I currently have the filter installed as 
/etc/logwatch/scripts/shared/dovecottostd. It seems a little odd to put 
a service-specific filter in shared, but given that it is needed to 
pre-process the log lines before they're seen by the service filter, I'm 
not sure where else it would go.)


Before I do, I'm also working on some changes to the Dovecot filter and 
I'm wondering:

The logic controlling what gets displayed at various detail levels 
didn't mesh with my expectations. Low should report only errors, but 
instead it was reporting summary information, like:

  Dovecot restarted 4 time(s).

  Dovecot disconnects:
     Inactivity: 4 Time(s)
     Logged out: 77 Time(s)
     in IDLE: 2 Time(s)
     no reason: 48 Time(s)

Then medium was reporting what I'd consider detailed information, such 
as a breakdown of the number of logins per IP address per protocol. And 
finally High was reporting the number of logins per user. These last two 
seem hardly distinguishable. (I'd expect a typical site to see similar 
results from both reports, as each user often has a unique IP address.) 
Whether they belong in Medium (summary) or High is debatable, but I'd 
expect them both to be ranked the same.

I'd suggest moving both to High, and add some summary information to 
Medium to cover logins. What would be useful to report? Quantity of 
logins per protocol? Something like:

  Connections:
     POP3:  20
     IMAP:  80
    Total: 100


There seems to be no code to track login failures, which in Dovecot 1.0 
produces lines like:

Jun 27 18:32:39 host dovecot: auth-worker(default): 
sql([email protected],127.0.0.1): Password mismatch
Jun 27 18:33:00 host dovecot: auth-worker(default): 
sql([email protected],127.0.0.1): unknown user

I've added code for this, but it appears that the line will vary based 
on the auth section ("default") and database type ("sql"). I ignored the 
auth section, and assumed that all database types would use the same 
(user,host) format. If others have sample log lines for login failures, 
please send them to me and I'll accommodate them.

Currently I'm generating a report like this:

  Login failures:
     Password mismatch:    1
          unknown user:    3

in low, and like this:

  Login failures:
     Password mismatch:    1
            [email protected] from       127.0.0.1
          unknown user:    3
            [email protected] from       127.0.0.1
            [email protected] from       127.0.0.1
            [email protected] from       127.0.0.1

in medium.


The first line into the while loop the code does:

    # remove timestamp.  We can't use *RemoveHeaders because we need the
    # service name
    $ThisLine =~ s/^\w{3} .\d \d\d:\d\d:\d\d [^ ]* //;

and it ignores lines that don't match. I modified it to add non-matching 
lines to the %OtherList hash so they'll get reported as "Unmatched Entries."


There was a note in the code:

# TODO:
# - use printf features to align text in table

and I've done that for the summary info shown above, but I'm wondering 
if Perl's classic format functionality might be the more appropriate 
tool for doing the bigger tables used for the detailed report. Is there 
any policy or recommendation regarding use of format? I thought there 
are some OO modules that clean up the interface to format, but a search 
on CPAN didn't turn up anything useful. Any policy or recommendation 
regarding dependencies?

In any case, I'll at minimum refactor the detail reporting code that is 
currently calculating string lengths and padding to use printf.


I noticed there is no bang line at the top of the script. An oversight?


Would it make sense to split the filter into a .9 and a 1.0 version? Or 
perhaps have a config variable to enable the .9 regular expressions so 
you aren't running through a list of irrelevant expressions for every 
log line? Using two separate scripts would permit using more of the 
stock logwatch infrastructure, like *RemoveHeaders. (As the comment 
above notes, the script currently doesn't use *RemoveHeaders because the 
sub-services of Dovecot show up in the service field of .9 logs.)


I've updated the code to comply with 'use strict;'

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: http://tmetro.venturelogic.com/