svn commit: r1934332 - in spamassassin/trunk: . lib/Mail/SpamAssassin/Plugin rules t
[email protected] Mon, 18 May 2026 04:31:22 -0000
| Newsgroups | gmane.mail.spam.spamassassin.cvs |
|---|---|
| Message-ID | <177907868280.64701.4929705898591493717@svn03-he-fi> |
Author: fkento
Date: Mon May 18 04:31:22 2026
New Revision: 1934332
Log:
Redirectors: support per-path matching and simplify domain matching
- url_redirector and url_redirector_get accept an optional /path suffix;
a URL only matches when its path begins with that string and ends at
a path-segment boundary. Useful to follow click-tracking endpoints
(e.g. .r.af.d.sendibt2.com/tr/cl/) without chasing tracking-pixel
endpoints (/tr/op/) on the same host.
- Rewrote _check_redirector_uri: linear lookup (exact host, auto-www,
suffix walk) replaces the previous dot-counting branches. Leading-dot
entries match subdomains only, not the bare domain (matches prior
behavior). Bare entries still auto-match the www subdomain.
- clear_url_redirector understands the same domain/path syntax and
drops the host entry when its path list becomes empty.
- has_url_redirector_path version feature so rule files can gate the
new syntax with can().
- 25_url_redirectors.cf: restrict .awstrack.me to /L0/ (click) so the
open-tracking /O0/ endpoint is no longer fetched.
- t/redirectors_match.t: new unit tests (network-independent) cover
domain matching, path-prefix gating, segment-boundary handling, and
clear_url_redirector.
- t/debug.t: register the existing "Redirectors" debug facility, which
the new code paths surface more frequently.
Added:
spamassassin/trunk/t/redirectors_match.t
Modified:
spamassassin/trunk/MANIFEST
spamassassin/trunk/lib/Mail/SpamAssassin/Plugin/Redirectors.pm
spamassassin/trunk/rules/25_url_redirectors.cf
spamassassin/trunk/t/debug.t
Modified: spamassassin/trunk/MANIFEST
==============================================================================
--- spamassassin/trunk/MANIFEST Mon May 18 04:20:04 2026 (r1934331)
+++ spamassassin/trunk/MANIFEST Mon May 18 04:31:22 2026 (r1934332)
@@ -630,6 +630,7 @@ t/recips.t
t/recreate.t
t/recursion.t
t/redirectors.t
+t/redirectors_match.t
t/regexp_named_capture.t
t/regexp_named_capture_spamd.t
t/regexp_valid.t
Modified: spamassassin/trunk/lib/Mail/SpamAssassin/Plugin/Redirectors.pm
==============================================================================
--- spamassassin/trunk/lib/Mail/SpamAssassin/Plugin/Redirectors.pm Mon May 18 04:20:04 2026 (r1934331)
+++ spamassassin/trunk/lib/Mail/SpamAssassin/Plugin/Redirectors.pm Mon May 18 04:31:22 2026 (r1934332)
@@ -125,16 +125,46 @@ sub new {
=over 4
-=item url_redirector domain [domain...] (default: none)
+=item url_redirector domain[/path] [domain[/path]...] (default: none)
-Domains that should be considered as an URL redirector. If the domain begins
-with a '.', 3rd level tld of the main domain will be checked.
-The 3rd level starting with www with always be checked for every 2tld.
+Domains that should be considered as a URL redirector.
+
+Domain matching:
+
+=over 4
+
+=item *
+
+A bare domain (e.g. C<bing.com>) matches that exact host and also C<www.bing.com>.
+
+=item *
+
+A leading dot (e.g. C<.sendgrid.com>) matches any subdomain of the domain,
+to any depth. It does NOT match the bare domain itself; to match both, also
+list the bare domain (C<sendgrid.com>).
+
+=back
+
+An optional C</path> may follow the domain to restrict matching to URLs whose
+path begins with that string and ends at a path-segment boundary. C</tr/op> matches
+C</tr/op>, C</tr/op/foo>, and C</tr/op?x=1>, but NOT C</tr/open> — the
+configured prefix is treated as one or more whole path segments. Append a
+trailing slash (C</tr/op/>) to require at least one more segment after it
+(matches C</tr/op/foo> but not bare C</tr/op>). Multiple entries for the
+same domain are additive (allowlist); a URL is followed if it matches any
+entry.
+
+A bare-domain entry without a path is equivalent to C<domain/>, which matches
+any path.
Example:
url_redirector bing.com
url_redirector .sendgrid.com
+ url_redirector .sendibt2.com/tr/cl/
+
+The last line follows C<https://x.y.sendibt2.com/tr/cl/abc> but not
+C<https://x.y.sendibt2.com/tr/op/abc>.
=back
@@ -153,8 +183,8 @@ sub set_config {
if ($value eq '') {
return $Mail::SpamAssassin::Conf::MISSING_REQUIRED_VALUE;
}
- foreach my $domain (split(/\s+/, $value)) {
- $self->{url_redirector}->{lc $domain} = 1; # 1 == head
+ foreach my $token (split(/\s+/, $value)) {
+ _add_redirector_entry($self, $token, 'head');
}
}
});
@@ -215,11 +245,16 @@ Set Selenium port to use.
=over 4
-=item clear_url_redirector [domain] [domain...]
+=item clear_url_redirector [domain[/path]] [domain[/path]...]
Clear configured url_redirector domains, for example to
-override default settings from an update channel. If domains are specified,
-then only those are removed from list.
+override default settings from an update channel. If no arguments are given,
+all entries are cleared. If domains are specified, only those are removed.
+
+When an entry includes a C</path>, only that path is removed from the
+domain's allowlist; the domain entry itself is dropped only when its path
+list becomes empty. Use C<domain/> to remove the "match any path" entry
+added by a bare-domain configuration.
=back
@@ -230,10 +265,11 @@ then only those are removed from list.
code => sub {
my ($self, $key, $value, $line) = @_;
if ($value eq '') {
- $self->{url_redirector} = {};
+ $self->{url_redirector_exact} = {};
+ $self->{url_redirector_suffix} = {};
} else {
- foreach my $domain (split(/\s+/, $value)) {
- delete $self->{url_redirector}->{lc $domain};
+ foreach my $token (split(/\s+/, $value)) {
+ _clear_redirector_entry($self, $token);
}
}
}
@@ -241,11 +277,13 @@ then only those are removed from list.
=over 4
-=item url_redirector_get domain [domain...] (default: none)
+=item url_redirector_get domain[/path] [domain[/path]...] (default: none)
-Domains that should be considered as an URL redirector. If the domain begins
-with a '.', 3rd level tld of the main domain will be checked.
-The http GET method will be used to check those domains.
+Domains that should be considered as an URL redirector, accessed using the
+HTTP GET method instead of HEAD. Syntax and matching rules are the same as
+C<url_redirector>: a bare domain matches that host and C<www.>, a leading
+dot matches the domain and any subdomain, and an optional C</path> prefix
+restricts the match.
=back
@@ -258,8 +296,8 @@ The http GET method will be used to chec
if ($value eq '') {
return $Mail::SpamAssassin::Conf::MISSING_REQUIRED_VALUE;
}
- foreach my $domain (split(/\s+/, $value)) {
- $self->{url_redirector}->{lc $domain} = 2; # 2 == get
+ foreach my $token (split(/\s+/, $value)) {
+ _add_redirector_entry($self, $token, 'get');
}
}
});
@@ -805,11 +843,81 @@ sub redir_url_loop {
return $pms->{redir_url_loop} ? 1 : 0;
}
+sub _add_redirector_entry {
+ my ($conf, $token, $method) = @_;
+
+ $token = lc $token;
+ my ($domspec, $path) = split(/\//, $token, 2);
+ $path = defined $path ? '/' . $path : '/';
+
+ my $is_suffix = ($domspec =~ s/^\.//) ? 1 : 0;
+ return unless length $domspec;
+
+ my $bucket = $is_suffix ? 'url_redirector_suffix' : 'url_redirector_exact';
+ my $entry = $conf->{$bucket}->{$domspec} ||= { method => $method, paths => [] };
+ $entry->{method} = $method;
+ push @{$entry->{paths}}, $path unless grep { $_ eq $path } @{$entry->{paths}};
+}
+
+sub _clear_redirector_entry {
+ my ($conf, $token) = @_;
+
+ $token = lc $token;
+ my $has_path = ($token =~ /\//) ? 1 : 0;
+ my ($domspec, $path) = split(/\//, $token, 2);
+ $path = '/' . (defined $path ? $path : '');
+
+ my $bucket = ($domspec =~ s/^\.//) ? 'url_redirector_suffix' : 'url_redirector_exact';
+ return unless length $domspec;
+
+ if (!$has_path) {
+ delete $conf->{$bucket}->{$domspec};
+ return;
+ }
+ my $entry = $conf->{$bucket}->{$domspec} or return;
+ @{$entry->{paths}} = grep { $_ ne $path } @{$entry->{paths}};
+ delete $conf->{$bucket}->{$domspec} unless @{$entry->{paths}};
+}
+
+sub _entry_match_path {
+ my ($entry, $path) = @_;
+ for my $p (@{$entry->{paths}}) {
+ next unless index($path, $p) == 0;
+ # Require a boundary after the configured prefix so /tr/op does not
+ # match /tr/open. A trailing '/' in the configured prefix supplies
+ # its own boundary; otherwise the next char must be end-of-string
+ # or '/'. ('?' and '#' have already been stripped from $path.)
+ return 1 if substr($p, -1) eq '/'
+ || length($path) == length($p)
+ || substr($path, length($p), 1) eq '/';
+ }
+ return 0;
+}
+
+sub _lookup_redirector {
+ my ($conf, $host, $path) = @_;
+
+ if (my $e = $conf->{url_redirector_exact}->{$host}) {
+ return $e if _entry_match_path($e, $path);
+ }
+ if ($host =~ /^www\.(.+)$/) {
+ if (my $e = $conf->{url_redirector_exact}->{$1}) {
+ return $e if _entry_match_path($e, $path);
+ }
+ }
+ my $h = $host;
+ while ($h =~ s/^[^.]+\.//) {
+ last unless $h =~ /\./;
+ if (my $e = $conf->{url_redirector_suffix}->{$h}) {
+ return $e if _entry_match_path($e, $path);
+ }
+ }
+ return;
+}
+
sub _check_redirector_uri {
my ($uri, $conf) = @_;
- my $newuri;
-
local($1,$2);
# normalize uri only if it doesn't contain more explicit redirects
# encoded as parameters
@@ -824,6 +932,7 @@ sub _check_redirector_uri {
(.*)? # Some path wanted
}ix;
my $host = lc $1;
+ my $rest = defined $2 ? $2 : '';
# return early if host is not valid
if($host =~ /\=|\&|\?/) {
return;
@@ -831,85 +940,40 @@ sub _check_redirector_uri {
if(is_fqdn_valid($host)) {
$host = idn_to_ascii($host);
}
- my $has_path = defined $2;
+ my $has_path = length $rest ? 1 : 0;
my $levels = $host =~ tr/.//;
# No point looking at single level "xxx.yy" without a path
return if $levels == 1 && !$has_path;
- my $params = $2;
- if($has_path and defined $params and (length($params) > 2)) {
- dbg("Found url with host $host and querystring $params");
- }
return if $uri !~ /([^.]+\.[^.]+)/;
# skip wrongly parsed uris
return if $uri =~ /^([a-z0-9]+?)\@/;
- if (exists $conf->{url_redirector}->{$host}) {
- dbg("Found redirection for host $host");
- return {
- 'uri' => $uri,
- 'method' => $conf->{url_redirector}->{$host} == 1 ? 'head' : 'get',
- };
+ # Split path from querystring/fragment so query content can't sneak past
+ # the path-prefix test.
+ my ($path, $query) = ($rest, '');
+ if ($path =~ /^([^?#]*)(.*)$/) {
+ ($path, $query) = ($1, $2);
}
- # if domain is a 3rd level domain check if there is a url redirector
- # on the www domain
- elsif($levels == 2 && $host =~ /^www\.([^.]+\.[^.]+)$/i) {
- my $domain = $1;
- if(exists $conf->{url_redirector}->{$domain}) {
- dbg("Found internal www redirection for domain $domain");
- return {
- 'uri' => $uri,
- 'method' => $conf->{url_redirector}->{$domain} == 1 ? 'head' : 'get',
- };
- } elsif ($newuri = _check_querystring($params, $conf)) {
- return {
- 'uri' => $newuri,
- 'method' => 'head',
- };
- }
- }
- elsif($levels == 3 && $host =~ /^www\.([^.]+\.[^.]+\.[^.]+)$/i) {
- my $domain = $1;
- if(exists $conf->{url_redirector}->{$domain}) {
- dbg("Found internal www redirection for domain $domain");
- return {
- 'uri' => $uri,
- 'method' => $conf->{url_redirector}->{$domain} == 1 ? 'head' : 'get',
- };
- } elsif ($newuri = _check_querystring($params, $conf)) {
- return {
- 'uri' => $newuri,
- 'method' => 'head',
- };
- }
- }
- # if domain is a 3rd level domain check if there is a url redirector
- # on the 2nd level tld
- elsif ($levels == 2 && $host =~ /^(?!www)[^.]+(\.[^.]+\.[^.]+)$/i &&
- exists $conf->{url_redirector}->{$1}) {
- return {
- 'uri' => $uri,
- 'method' => $conf->{url_redirector}->{$1} == 1 ? 'head' : 'get',
- };
+ $path = '/' unless length $path;
+
+ if (my $e = _lookup_redirector($conf, $host, $path)) {
+ dbg("Found redirection for host $host path $path");
+ return { uri => $uri, method => $e->{method} };
}
- elsif ($host =~ /(\.[a-z0-9_]+(?:\.[a-z0-9_]+)?\.[a-z]+)$/i &&
- exists $conf->{url_redirector}->{$1}) {
- return {
- 'uri' => $uri,
- 'method' => $conf->{url_redirector}->{$1} == 1 ? 'head' : 'get',
- };
- } elsif ($newuri = _check_querystring($params, $conf)) {
+
+ if (my $newuri = _check_querystring($rest, $conf)) {
my $nhost = $newuri;
- if($nhost =~ /https?:\/\/(.*)/) {
- $nhost = $1;
- }
+ $nhost = $1 if $nhost =~ m{^https?://([^/?#:]+)};
+ $nhost = lc $nhost;
+ my $ne = _lookup_redirector($conf, $nhost, '/');
return {
- 'uri' => $newuri,
- 'method' => (exists($conf->{url_redirector}->{$nhost}) && ($conf->{url_redirector}->{$nhost} == 1)) ? 'head' : 'get',
+ uri => $newuri,
+ method => $ne ? $ne->{method} : 'get',
};
- } else {
- dbg("No explicit redirector host found for $host");
}
+
+ dbg("No explicit redirector host found for $host path $path");
return;
}
@@ -1301,5 +1365,6 @@ sub has_redir_url_maxchain { 1 }
sub has_redir_url_loop { 1 }
sub has_selenium_support { 1 }
sub has_url_skip_redirect_to { 1 }
+sub has_url_redirector_path { 1 } # path-prefix syntax in url_redirector / url_redirector_get
1;
Modified: spamassassin/trunk/rules/25_url_redirectors.cf
==============================================================================
--- spamassassin/trunk/rules/25_url_redirectors.cf Mon May 18 04:20:04 2026 (r1934331)
+++ spamassassin/trunk/rules/25_url_redirectors.cf Mon May 18 04:31:22 2026 (r1934332)
@@ -124,6 +124,9 @@ url_redirector_get share.google
url_redirector_get shared.outlook.inky.com
url_redirector_get t.nypost.com
-# generic list of likely active services
+if can(Mail::SpamAssassin::Plugin::Redirectors::has_url_redirector_path)
+ clear_url_redirector .awstrack.me
+ url_redirector .awstrack.me/L0/
+endif
endif
Modified: spamassassin/trunk/t/debug.t
==============================================================================
--- spamassassin/trunk/t/debug.t Mon May 18 04:20:04 2026 (r1934331)
+++ spamassassin/trunk/t/debug.t Mon May 18 04:31:22 2026 (r1934332)
@@ -17,7 +17,7 @@ my %facility = map( ($_, 1),
message metadata mimeheader netset plugin prefork progress pyzor razor2
received-header replacetags reporter rules rules-all spamd spf textcat
timing TxRep uri uridnsbl util pdfinfo asn geodb FromNameSpoof
- PHISHTAG resourcelimits https_http_mismatch DMARC ));
+ PHISHTAG resourcelimits https_http_mismatch DMARC Redirectors ));
my $fh = IO::File->new_tmpfile();
open(OLDERR, ">&STDERR");
Added: spamassassin/trunk/t/redirectors_match.t
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ spamassassin/trunk/t/redirectors_match.t Mon May 18 04:31:22 2026 (r1934332)
@@ -0,0 +1,85 @@
+#!/usr/bin/perl -T
+
+use lib '.'; use lib 't';
+use SATest; sa_t_init("redirectors_match");
+
+use strict;
+use Test::More;
+
+require Mail::SpamAssassin::Plugin::Redirectors;
+
+sub make_conf {
+ my $conf = { url_redirector_params => qr/(?:url|u)=(.*)/i };
+ Mail::SpamAssassin::Plugin::Redirectors::_add_redirector_entry($conf, 'bing.com', 'head');
+ Mail::SpamAssassin::Plugin::Redirectors::_add_redirector_entry($conf, '.sendgrid.com', 'head');
+ Mail::SpamAssassin::Plugin::Redirectors::_add_redirector_entry($conf, '.r.af.d.sendibt2.com/tr/cl/', 'get');
+ Mail::SpamAssassin::Plugin::Redirectors::_add_redirector_entry($conf, 'foo.example/tr/op/', 'head');
+ Mail::SpamAssassin::Plugin::Redirectors::_add_redirector_entry($conf, 'bar.example/tr/op', 'head');
+ return $conf;
+}
+
+my @cases = (
+ # Bare-domain entry: exact host + auto-www only, no other subdomains.
+ [ 'https://bing.com/', 'head', 'bare domain matches exact host' ],
+ [ 'https://www.bing.com/x', 'head', 'bare domain matches www subdomain' ],
+ [ 'https://mail.bing.com/x', undef, 'bare domain does NOT match other subdomains' ],
+
+ # Leading-dot entry: subdomains only, NOT the bare domain itself.
+ [ 'https://sendgrid.com/x', undef, 'leading-dot does NOT match bare domain' ],
+ [ 'https://www.sendgrid.com/x', 'head', 'leading-dot matches www subdomain' ],
+ [ 'https://a.b.sendgrid.com/x', 'head', 'leading-dot matches deep subdomain' ],
+
+ # Multi-label leading-dot: still excludes the bare-domain form.
+ [ 'https://r.af.d.sendibt2.com/tr/cl/abc', undef, '.r.af.d.sendibt2.com does NOT match its own bare form' ],
+ [ 'https://x.r.af.d.sendibt2.com/tr/cl/abc', 'get', '.r.af.d.sendibt2.com matches subdomain' ],
+ [ 'https://gfbgghj.r.af.d.sendibt2.com/tr/cl/abc', 'get', '.r.af.d.sendibt2.com matches deeper subdomain' ],
+
+ # Path-prefix gating, trailing slash in config (/tr/op/).
+ [ 'https://gfbgghj.r.af.d.sendibt2.com/tr/op/abc', undef, 'path /tr/op blocked when only /tr/cl/ configured' ],
+ [ 'https://foo.example/tr/op/abc', 'head', 'trailing-slash path matches /tr/op/abc' ],
+ [ 'https://foo.example/tr/open', undef, 'trailing-slash path does NOT match /tr/open' ],
+ [ 'https://foo.example/tr/op/', 'head', 'trailing-slash path matches itself' ],
+ [ 'https://foo.example/tr/op', undef, 'trailing-slash path does NOT match bare /tr/op' ],
+
+ # Path-prefix gating, no trailing slash in config (/tr/op).
+ [ 'https://bar.example/tr/op', 'head', 'no-trailing-slash matches bare path' ],
+ [ 'https://bar.example/tr/op/', 'head', 'no-trailing-slash matches with trailing slash' ],
+ [ 'https://bar.example/tr/op/abc', 'head', 'no-trailing-slash matches with sub-path' ],
+ [ 'https://bar.example/tr/op?x=1', 'head', 'no-trailing-slash matches with query string' ],
+ [ 'https://bar.example/tr/op#frag', 'head', 'no-trailing-slash matches with fragment' ],
+ [ 'https://bar.example/tr/open', undef, 'no-trailing-slash does NOT match /tr/open' ],
+ [ 'https://bar.example/tr/opfoo', undef, 'no-trailing-slash does NOT match /tr/opfoo' ],
+);
+
+# clear_url_redirector tests add 4 more checks
+plan tests => scalar(@cases) + 4;
+
+for my $c (@cases) {
+ my ($uri, $expect_method, $desc) = @$c;
+ my $conf = make_conf();
+ my $r = Mail::SpamAssassin::Plugin::Redirectors::_check_redirector_uri($uri, $conf);
+ if (defined $expect_method) {
+ is(($r ? $r->{method} : undef), $expect_method, $desc);
+ } else {
+ ok(!$r, $desc);
+ }
+}
+
+# clear_url_redirector behavior
+{
+ my $conf = { url_redirector_params => qr/(?:url|u)=(.*)/i };
+ Mail::SpamAssassin::Plugin::Redirectors::_add_redirector_entry($conf, '.sendibt2.com/tr/cl/', 'get');
+ Mail::SpamAssassin::Plugin::Redirectors::_add_redirector_entry($conf, '.sendibt2.com/tr/click/', 'get');
+
+ Mail::SpamAssassin::Plugin::Redirectors::_clear_redirector_entry($conf, '.sendibt2.com/tr/cl/');
+ ok(!Mail::SpamAssassin::Plugin::Redirectors::_check_redirector_uri('https://x.sendibt2.com/tr/cl/a', $conf),
+ 'clear removes /tr/cl/ path');
+ ok( Mail::SpamAssassin::Plugin::Redirectors::_check_redirector_uri('https://x.sendibt2.com/tr/click/a', $conf),
+ 'clear leaves /tr/click/ path');
+
+ Mail::SpamAssassin::Plugin::Redirectors::_clear_redirector_entry($conf, '.sendibt2.com/tr/click/');
+ ok(!Mail::SpamAssassin::Plugin::Redirectors::_check_redirector_uri('https://x.sendibt2.com/tr/click/a', $conf),
+ 'clear removes last path');
+ ok(!exists $conf->{url_redirector_suffix}->{'sendibt2.com'},
+ 'host entry dropped when paths empty');
+}