Re: sendmail blackhole

Gulikoza,

Got it.  In that case, I'd like to suggest searching for [^\/\s], which
will include any characters that are not a slash nor whitespace, which
also applies to newline.  That will also take into account the form
	http://example.com info
where info is additional string after the URL and a space or a tab.

Attached is the patch against the repository version of sendmail.  Can
you let me know if that works?


On 12/31/2014 09:50 AM, gulikoza wrote:
> Hi,
> 
> Perhaps there was some misunderstanding, my patch actually makes regex match
> more lines not add more info to the hash.
> For instance, current code does:
> 
>    } elsif ( ($Relay,$BlSite) = ($ThisLine =~
> /^ruleset=(?:check_relay|check_rcpt), arg1=([^,]*),(?: arg2=[^,]*,)?
> reject=55\d\s*[\d.]*\s*.*http:\/\/([^\/]*)\//) ) {
>       #Example 553 error with NO RELAY -mgt
>       #ruleset=check_relay, arg1=s010600402b39ee29.vf.shawcable.net,
> arg2=127.0.0.2, reject=553 5.3.0
>       #Spam blocked see: http://spamcop.net/bl.shtml?70.68.8.182: 1 Time(s)
> 
> So it expects the sendmail message to include the full url:
> "http://spamcop.net/bl.shtml?70.68.8.182: 1 Time(s)" including the
> client_addr as explained in the comment.
> 
> This regex actually needs 3 slashes in the url to match:
> "http:\/\/([^\/]*)\/" and assigns the string before the third slash to
> $BlSite. Precisely because sendmail configuration can differ greatly and the
> blacklist provider might not provide public url to full lookup with client
> address, my suggestion was to also match lines where only site name is
> displayed without the client_addr, so to end the regex with either '/ or
> newline' match. The modified regex would match lines like: "- see
> http://www.myblacklistprovider.net" (as an example) without the client_addr
> at the end. In both cases only the domain name part is used in $BlSite.
> 
> I don't believe such a change would significantly impact sendmail reports in
> the negative way, if I'm not mistaken.
> Happy New Year!
> 
> Regards,
> gulikoza
> 
> 
> -----Original Message-----
> From: Bjorn [mailto:[email protected]] 
> Sent: Tuesday, December 30, 2014 10:04 PM
> To: [email protected]
> Subject: Re: [Logwatch-devel] sendmail blackhole
> 
> 
> While I didn't write the blackhole reporting code in Logwatch sendmail, I am
> familiar with this service script, so I wanted to provide some
> thoughts:
> 
> I think the original intent was to have Logwatch identify the sources of the
> blacklist (the sites from which the blacklist was obtained).
> 
> One issue is that this code is not only for spamcop, the blacklist provider
> you are using, but any blacklist.  And there is the assumption that the
> blacklist website is listed on the error message that is returned by the
> sendmail MTA.  This error message is something that each user can configure
> on its own in the pertinent mc file (or even the cf file).  Also, additional
> variables can be inserted, such as the client_addr you are using on your
> configuration file, making it very hard to know what the format or contents
> of the error message is beforehand.
> 
> Placing the whole string in $BlSite might cause the Logwatch report to
> display all this other information that may not be suitable for a summary,
> and also break out individual error lines that otherwise would be reported
> as a single summary line with a larger error count.
> 
> Looking at the Logwatch sendmail code, it appears that the Blackholed hash
> is the one that gets displayed when $Detail >= 10, while the contents of the
> BlackHoles hash is displayed at $Detail >= 5.  So if additional info is
> added, it would seem preferable to use the BlackHoled hash only instead of
> both BlackHoles and BlackHoled.  Actually, it would seem preferable to use a
> hash of hashes, as was done with other error categories (such as
> $BlackHoled{$BlSite}{$Relay}).
> 
> [As an added comment, it seems there is some commingling on checking of
> check_relay and check_rcpt; and arg2 and relay; but it appears that the
> resulting report displays the available data.]
> 
> In summary, I think that the patch suggested may adversely impact other
> people's Logwatch sendmail reports.  There might be a more universal
> solution, such as putting additional info in the BlackHoled hash only (or in
> hash of hashes), but I don't have blacklist error reports to check or test.
> 
> As a final note, I'll mention $MatchFilter and $ReportFilter, which are
> described in the configuration file.  It allows individual sites to
> customize the sendmail report, precisely because the sendmail MTA allows so
> much customization, whether through changes in the mc file, the cf file, or
> the many milters available.  These two variables allow changing the Logwatch
> sendmail script without having to patch it every time a new sendmail script
> is released.
> 
> 
> Bjorn
> 
> 
> 
> On 12/28/2014 08:28 AM, gulikoza wrote:
>> Hi,
>>
>> Since sendmail seems to be on the spot, would it be possible to modify 
>> blackhole regex not to search for the third / at the end of the url.
>> My sendmail only prints sitename ("- see http://www.spamcop.net") not 
>> the full lookup url ("http://spamcop.net/bl.shtml?70.68.8.182"). 
>> Perhaps something like (match [\/\n] instead on \/):
>>
>> --- sendmail.org        2014-12-28 16:08:30.472681423 +0100
>> +++ sendmail    2014-12-28 16:18:28.294860546 +0100
>> @@ -823,18 +823,18 @@
>>        $Temp = "From " . $1 . " by " . $2;
>>        $BlackHoled{$Temp}++;
>>        $BlackHoles{$2}++;
>> -   } elsif ( ($Relay,$BlSite) = ($ThisLine =~
>> /^ruleset=(?:check_relay|check_rcpt), arg1=[^,]*,(?: arg2=[^,]*,)?
>> relay=([^,]*), reject=55\d\s*[\d.]*\s*.*http:\/\/([^\/]*)\//) ) {
>> +   } elsif ( ($Relay,$BlSite) = ($ThisLine =~
>> /^ruleset=(?:check_relay|check_rcpt), arg1=[^,]*,(?: arg2=[^,]*,)?
>> relay=([^,]*), reject=55\d\s*[\d.]*\s*.*http:\/\/([^\/]*)[\/\n]/) ) {
>>        $Temp = "From " . $Relay . " by " . $BlSite;
>>        $BlackHoled{$Temp}++;
>>        $BlackHoles{$BlSite}++;
>> -   } elsif ( ($Relay,$BlSite) = ($ThisLine =~
>> /^ruleset=(?:check_relay|check_rcpt), arg1=([^,]*),(?: arg2=[^,]*,)?
>> reject=55\d\s*[\d.]*\s*.*http:\/\/([^\/]*)\//) ) {
>> +   } elsif ( ($Relay,$BlSite) = ($ThisLine =~
>> /^ruleset=(?:check_relay|check_rcpt), arg1=([^,]*),(?: arg2=[^,]*,)?
>> reject=55\d\s*[\d.]*\s*.*http:\/\/([^\/]*)[\/\n]/) ) {
>>        #Example 553 error with NO RELAY -mgt
>>        #ruleset=check_relay, arg1=s010600402b39ee29.vf.shawcable.net,
>> arg2=127.0.0.2, reject=553 5.3.0
>>        #Spam blocked see: http://spamcop.net/bl.shtml?70.68.8.182: 1
> Time(s)
>>        $Temp = "From " . $Relay . " by " . $BlSite;
>>        $BlackHoled{$Temp}++;
>>        $BlackHoles{$BlSite}++;
>> -   } elsif ( ($Relay,$BlSite) = ($ThisLine =~ /reject=553\s*[\d.]*\s*<[^
>> ]*>\.\.\. +Mail from ([\d\.]+) rejected\;see http:\/\/([^\/]*)\//) ) {
>> +   } elsif ( ($Relay,$BlSite) = ($ThisLine =~ 
>> + /reject=553\s*[\d.]*\s*<[^
>> ]*>\.\.\. +Mail from ([\d\.]+) rejected\;see http:\/\/([^\/]*)[\/\n]/) ) {
>>        #This is the another blackhole tag -mgt
>>        $Temp = "From " . $Relay . " by " . $BlSite;
>>        $BlackHoled{$Temp}++;
>>
>>
>> ----------------------------------------------------------------------
>> -------- Dive into the World of Parallel Programming! The Go Parallel 
>> Website, sponsored by Intel and developed in partnership with Slashdot 
>> Media, is your hub for all things parallel software development, from 
>> weekly thought leadership blogs to news, videos, case studies, 
>> tutorials and more. Take a look and join the conversation now. 
>> http://goparallel.sourceforge.net 
>> _______________________________________________
>> Logwatch-devel mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/logwatch-devel
>>
> 
> 
> ----------------------------------------------------------------------------
> --
> Dive into the World of Parallel Programming! The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net
> _______________________________________________
> Logwatch-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/logwatch-devel
>

------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net

_______________________________________________
Logwatch-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/logwatch-devel

--- sendmail.orig	2014-12-24 10:42:43.529365661 -0800
+++ sendmail	2015-01-03 13:58:55.451690153 -0800
@@ -823,18 +823,18 @@
       $Temp = "From " . $1 . " by " . $2;
       $BlackHoled{$Temp}++;
       $BlackHoles{$2}++;
-   } elsif ( ($Relay,$BlSite) = ($ThisLine =~ /^ruleset=(?:check_relay|check_rcpt), arg1=[^,]*,(?: arg2=[^,]*,)? relay=([^,]*), reject=55\d\s*[\d.]*\s*.*http:\/\/([^\/]*)\//) ) {
+   } elsif ( ($Relay,$BlSite) = ($ThisLine =~ /^ruleset=(?:check_relay|check_rcpt), arg1=[^,]*,(?: arg2=[^,]*,)? relay=([^,]*), reject=55\d\s*[\d.]*\s*.*http:\/\/([^\/\s]*)/) ) {
       $Temp = "From " . $Relay . " by " . $BlSite;
       $BlackHoled{$Temp}++;
       $BlackHoles{$BlSite}++;
-   } elsif ( ($Relay,$BlSite) = ($ThisLine =~ /^ruleset=(?:check_relay|check_rcpt), arg1=([^,]*),(?: arg2=[^,]*,)? reject=55\d\s*[\d.]*\s*.*http:\/\/([^\/]*)\//) ) {
+   } elsif ( ($Relay,$BlSite) = ($ThisLine =~ /^ruleset=(?:check_relay|check_rcpt), arg1=([^,]*),(?: arg2=[^,]*,)? reject=55\d\s*[\d.]*\s*.*http:\/\/([^\/\s]*)/) ) {
       #Example 553 error with NO RELAY -mgt
       #ruleset=check_relay, arg1=s010600402b39ee29.vf.shawcable.net, arg2=127.0.0.2, reject=553 5.3.0
       #Spam blocked see: http://spamcop.net/bl.shtml?70.68.8.182: 1 Time(s)
       $Temp = "From " . $Relay . " by " . $BlSite;
       $BlackHoled{$Temp}++;
       $BlackHoles{$BlSite}++;
-   } elsif ( ($Relay,$BlSite) = ($ThisLine =~ /reject=553\s*[\d.]*\s*<[^ ]*>\.\.\. +Mail from ([\d\.]+) rejected\;see http:\/\/([^\/]*)\//) ) {
+   } elsif ( ($Relay,$BlSite) = ($ThisLine =~ /reject=553\s*[\d.]*\s*<[^ ]*>\.\.\. +Mail from ([\d\.]+) rejected\;see http:\/\/([^\/\s]*)/) ) {
       #This is the another blackhole tag -mgt
       $Temp = "From " . $Relay . " by " . $BlSite;
       $BlackHoled{$Temp}++;