Re: DNSBL and IPv6

On Fri, Oct 26, 2012 at 2:34 AM, Hal Murray <[email protected]> wrote:

>> I'm obviously biased since I run dnswl.org, but an IPv6-based whitelist may
>> work better than an IPv6-based blacklist. Enumerating the goodness is
>> generally easier than enumerating the badness.
>
> What fraction of email comes from hosts you have listed?  How hard would it
> be to scale your list up to cover the whole world?

We do store IPv6 addresses, but we don't publish them yet (since there
is no standard yet - the result of the discussion here may lead to the
emergence of a de-facto standard, or at least a first trial). So I can
not really tell what fraction we have listed in IPv6 world.

I can tell about our estimate on what we cover in terms of IPv4 based
on the dnswl.org stats. These stats are based not on SMTP trafffic,
but on the DNS traffic which we log (sample) on some of the public
mirrors. Larger senders with better cache utilization are likely to be
somewhat underrepresented in our data. We then do not use the absolute
numbers, but logarithmic magnitudes:

Magnitude	Percent
10.0	100%
9.0	10%
8.0	1%
7.0	0.1%
6.0	0.01%
5.0	0.001%
4.0	0.0001%
3.0	0.00001%
2.0	0.000001%
1.0	0.0000001%

Extract from our magnitudes report (I'll happily share more data on request):
1	9.63	IPs where we have no record
2	9.24	IPs which are in our DB*, not published - contains a lot of
"bad apples" ("DNSWL Id 0")
3	8.86	IPs which are in our DB*, not published - with fewer "bad
apples" ("DNSWL Id 1")
4	8.44	Yahoo
5	8.38 Google
6	8.36 Internally blacklisted (snowshoe ranges etc)
7	8.12 Hotmail
8	8.03 Facebook
9	7.91	Exacttarget
10	7.89	cheetahmail.com	

* Through imports from third parties or "learned" through the DNS logs

It's clear that key is the mag 9.63 of where we have no record. This
category contains all the residential/dynamic/botnet IPs; we do not
count the number of different IPs.

DNSWL Id 0 contains about 300k IPs, DNSWL Id 1 contains about 100k
IPs, all the other (published) DNSWL records contain about 200k IPs.
Records in 0 and 1 have quite some fluctuation (eg removed since they
are not present in the import source any more; promoted from "0" to
"1" based on some criteria; demoted from "1" to "0" based on the lack
of the same criteria). Entries are manually promoted from "0" and "1"
to one of the published DNSWL Ids.

Possibly 80k in each of the two "special" records may be considered
"good" (as in: not operated by spammer/bot herders), which would lead
to ~ 360k mostly legitimate SMTP-sending IPs.


> Assuming that you don't want to put all your eggs in one basket, how many
> white lists would you need and/or how would you decide the order to check
> them?

One? Definitely too risky. Two? Not sufficient, in my view. Three?
That may work. Four? Could improve diversity. Five? There is a point
where returns of additional lists become diminishing (eg higher
latency, larger overlaps).

-- Matthias