Re: URIBL plugin and 'semicolon insertion munging'

> This seems to be a big improvement at least on the 3 million lines of
> random traffic i tested with, and it's a smaller patch:
[snip]

Well, it may have been an improvement over my own data, but a colleague
pointed out the following case:

check out spamsite.com;it's awesome!

And this didn't deal with Devin's example of http://spamsite.com;.net either

At this point it's a bit beyond testing on the original plugin for me, but
here's what I came up with given my own modified hostname re loop (the
minimal entity decoding + stripping is still needed)

    while ( m{ ( (?: [a-zA-Z0-9:./-]+ @ )?
                 [a-zA-Z0-9][a-zA-Z0-9.;-]+\.$tld )
               (?! \.?\w ) }gxo ) {
        my $host = lc $1;
        # Deal with inserted-semicolon munging, e.g. 'http://foo;.com'
        if ( my @split = $host =~ /(.*?);(.*)/ ) {
            my @h = split /\./, $split[0];
            if ( $h[-1] =~ /^$tld$/ ) {
                # 'foo.com;.net', 'foo.com;.net;.foo', 'foo.com;it's great'
                $host = $split[0];
            } else {
                $split[1] =~ /([^.;]+)/;
                $split[1] = $1 if $1; # 'foo;.com;.net', 'foo;.com'
                $host = $1 ? "$split[0].$1" : $split[0];
            }
        }

This works at least for the test cases i've come up with, and doesn't
otherwise change processing from what _I_ had before for the other few
million lines i'm checking against, but is not yet in production and the
change alone is not tested against the vanilla code.  If anyone's
interested, the loop continues, dealing with excluding email addresses
while including user:pass@site:


        if ( $host =~ s/.*\.\.// ) {
            next unless $host =~ /\./;
        }
        if ( $host =~ s/^\w{3,16}:\/+// ) {
            # 'http://realsite.com/cgi/[email protected]'
            # 'http://user:[email protected]'
            $host =~ s/\/.*//;
            $host =~ s/.*@//;
        } elsif ( $host =~ /(.*):/ ) {
            # 'user:[email protected]'
            # 'mailto:[email protected]'
            next if $1 =~ /mailto/;
            $host =~ s/.*@//;
        } else {
            # 'realsite.com/cgi/[email protected]'
            $host =~ s/\/.*//;
            next if $host =~ /@/;
        }

This is the main reason for the different RE above.  Also it obsoletes the
need for two RE loops.

-Jared