Re: LogSQLMassVirtualHosting and mod_vhost_alias

Bob Apthorpe <[email protected]> Sun, 22 Jan 2006 00:37:33 -0600

Newsgroups	gmane.comp.apache.mod-log-sql
Message-ID	<[email protected]>

Hi,

Dice R. Random wrote:
> On 1/21/06, Dice R. Random <[email protected]> wrote:
> [...]
> Is there some ICANN documentation or something that will guarentee
> this is correct?  I didn't see anything when I took a cursory glance
> at their site.
> 
> It's not that I don't trust that what you said is true, Günther, I
> just want to be sure there's actually a policy in place so that this
> will work for all domains :)

Attached is a bit of perl to generate regexes to identify TLDs, 2LDs,
3LDs, and 4LDs from a recent (3.x) version of Mail::SpamAssassin.

The results should be PCRE-compatible so you should be able to use them
without perl. I originally used this script to generate code for a PHP
application to create DNSBLs from web probes, attacks, and blog spam but
it should be applicable to mod_log_sql as well.

Basically, if you match an xLD (top, second, third, fourth-level
domain), /(([^.]+\.)(xLD))$/ should match the base domain where you'd
store the log entry. www.example.co.uk and example.co.uk would match the
2LD '.co.uk' giving a base domain of example.co.uk for both. Similarly,
www.example.austin.ci.tx.us and example.austin.ci.tx.us would end up
under austin.ci.tx.us.

hth,

-- Bob

_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI@public.gmane.org

get_tld_regexes.pl (text/x-perl, 2.2 KB)

#!/usr/bin/perl -w
#
# Requires SpamAssassin v3.x or later
#
# This little bit of code extracts the top, second, third, and
# fourth-level domains from
# Mail::SpamAssassin::Util::RegistrarBoundaries. There's no point
# in reinventing the wheel here; if you're curious where the
# original data comes from, read the fine source code for
# RegistrarBoundaries.pm.
#
# Take the results of this script and add it to the function
# extract_domain_from_host() in analyze_host.php (or wherever it
# lives now.)
#
# ----- Example - FOUR_LEVEL_DOMAINS
#
# Here's the results of this script:
#
# -----
#
# (?ix-sm: (?:
#   # http://www.neustar.us/policies/docs/rfc_1480.txt
#   # "Fire-Dept.CI.Los-Angeles.CA.US"
#   # "<school-name>.PVT.K12.<state>.US"
# 
#   pvt\.k12\.(?ix-sm: (?:
#   ak|al|ar|az|ca|co|ct|dc|de|fl|ga|gu|hi|ia|id|il|in|ks|ky|la|ma|md|me|mi|
#   mn|mo|ms|mt|nc|nd|ne|nh|nj|nm|nv|ny|oh|ok|or|pa|pr|ri|sc|sd|tn|tx|ut|va|vi|
#   vt|wa|wi|wv|wy )
# )\.us
#   c[io]\.[^\.]+\.(?ix-sm: (?:
#   ak|al|ar|az|ca|co|ct|dc|de|fl|ga|gu|hi|ia|id|il|in|ks|ky|la|ma|md|me|mi|
#   mn|mo|ms|mt|nc|nd|ne|nh|nj|nm|nv|ny|oh|ok|or|pa|pr|ri|sc|sd|tn|tx|ut|va|vi|
#   vt|wa|wi|wv|wy )
# )\.us
# )
# )
#
# -----
# 
# PHP expects the regexp to be delimited; '%' was arbitrarily
# chosen as a delimiter in analyze_host.php so as long as the
# regexps don't contain '%' you should be able to use the perl
# regexps in PHP's PCRE functions without any trouble.
#
# The PHP code adds a \. (literal dot) at the beginning of the
# pattern and a $ (end of line) at the end of the pattern to
# anchor it at both ends of a domain. Otherwise something stupid
# like www.austin.ci.tx.us-porno.com would match as
# ci.tx.us-porno.com, not as us-porno.com

use strict;
use Mail::SpamAssassin::Util::RegistrarBoundaries;

print "##### 4LD regex #####\n"
 . $Mail::SpamAssassin::Util::RegistrarBoundaries::FOUR_LEVEL_DOMAINS,"\n";

print "##### 3LD regex #####\n"
 . $Mail::SpamAssassin::Util::RegistrarBoundaries::THREE_LEVEL_DOMAINS,"\n";

print "##### 2LD regex #####\n"
 . $Mail::SpamAssassin::Util::RegistrarBoundaries::TWO_LEVEL_DOMAINS,"\n";

print "##### TLD regex #####\n"
 . $Mail::SpamAssassin::Util::RegistrarBoundaries::VALID_TLDS,"\n";

__END__