Rewritten URIBL plugin

[email protected] ("Jared Johnson")
Newsgroups perl.qpsmtpd
Message-ID <[email protected]> 
Hi all,

Attached is the essentially re-written URIBL plugin that I've just about
finished for my employer.  I say "just about" because once I go further,
it will soon be hopelessly forked from normal QP on account of
per-recipient config directives, etc. so this seemed like the best time to
drop it:  I've tested it pretty extensively, but piecemeal and not in
production, and although I've done my best to make sure it doesn't depend
on any of our site-specific QP changes, I may have missed some things.  I
realize it would be more helpful to put together piecemeal patches, but
unfortunately I don't have the luxury of time to do this with such a large
project.

The plugin has the following advantages over the original:

- Uses MIME::Parser to unpack message text so that we can look for URI's
in base64-encoded data, etc., and _not_ look for URI's in noise.
- Introduces support for URIBL services that may not have worked right, at
least out of the box, before.  Defines the subtle differences between
various known URIBL services in order to maximize compatibility. 
Specifically:
  - We always send as much of the hostname as possible to SpamHaus DBL, up
to 5 levels; this is because DBL supports wildcards and any level of
hostname, but we want to stop somewhere to avoid zero DNS cache hits
  - We don't send IP addresses to DBL, Spamhaus doesn't support IP's on
the DBL (and returns *positive* results when queried with IP addresses)
  - We send IP addresses to SBL-XBL, as recommended by SpamHaus. 
Additionally, we do NS lookups and A lookups and send the results of
those to SBL-XBL as well, as recommended
  - We send a single query to URIBL and SRBL which constitutes the right
number of hostname levels depending on their respective two and three
level TLD lists.  Before, if I understand the code correctly, we send
multiple queries to each service with different levels to try and make
sure we get a hit.
- Uses Net::DNS::Async to simplify the code, and also to ensure the
afore-mentioned A and NS lookups will prompt new URIBL lookups in an
efficient and simple manner via callbacks
- Faster and, I think, more accurate URI detection.  The old version
passed everything through a couple of regular expressions in order to
catch everything, this uses one and is optimized a bit for performance. 
Also, the old version caught hostnames on the right hand side of email
addresses mentioned in message content, and had some other false hits
which the new RE's avoid.  I left the IP address stuff pretty much alone.
- Supports different actions per mask specified in the config file.  I
don't think the previous plugin did this.

Attached also is tld_lists.pl, a companion file that needs to be dropped
in lib/Qpsmtpd/ which provides the list of first, second, and third level
TLDs that we care about.  It's derived from our URIBL datafeed as well as
the SURBL two-level and three-level lists -- but the SURBL lists are
pruned to exclude stuff that we're pretty sure is never going to actually
get listed -- when was the last time you saw a spammer use .mil? 
uribl.com never has, according to our datafeed :)

There are some disadvantages to this code.  MIME::Parser is probably heavy
in terms of resource usage compared to just scanning the message body
line-by-line.  This would probably not be that difficult to revert again
if desired, although I think the advantages of M::P outweigh the
disadvantages.  tld_lists is bulky and untried and will probably need to
be kept updated to work perfectly (although at least SURBL says their
lists hardly ever changed)  Due to the generally forked nature of our
code, it's hard to say that there aren't bugs in this submission that
we'll never find at our shop because we've fixed them elsewhere in the
code, etc.  At any rate, I've worked hard on this code and am thankful to
Devin and the rest of the QP contributors for the starting point, so I
hope that someone manages to find this useful and perhaps massage it into
upstream :)

-Jared
uribl (application/octet-stream, 18.8 KB) - not displayed
tld_lists.pl (application/octet-stream, 27.9 KB) 
# Lists of relevant one, two, and three level TLDs
# generated Sat Jul 24 10:44:14 2010 by gen_tld_lists on dcrbl1

# First-level TLDs that we care about.  From what we've gathered from our URIBL
# datafeed, there are no spammers out there with the guts to use .mil, .gov, etc.
our %firstlevel_tlds = map { $_ => undef } # 128 entries
    qw( ae aero am ar as at au az ba bd be bg bh biz br bs by bz ca cat cc ch cl cm cn co com cx cy
    cz de dk ec ee eg es eu fi fr gd ge gg gr gs hk hm hn hr hu id ie il im in info io ir is it je
    jp ke kg kr kz la li lt lv ma md me mk mn mobi ms mu mx my name ne net nl no np nu nz org pe ph
    pk pl pr pro pt py ro rs ru sa sc se sg si sk st su tc th tk tl to tp tr tt tv tw tz ua uk us
    uy uz vc ve vn ws za );

# URIBL Second and Third level TLDs
our %uribl_secondlevel_tlds = map { $_ => undef } # 222 entries
    qw( com.ar org.ar com.au net.au com.ba com.bd com.br etc.br eti.br gov.br ind.br inf.br nom.br
    org.br na.by co.cc com.cn fj.cn gov.cn hk.cn net.cn org.cn qh.cn sd.cn sh.cn tw.cn com.co
    110mb.com 150m.com angelfire.com blogspot.com br.com bravehost.com cn.com eu.com freehostia.com
    gb.com geocities.com googlepages.com idoo.com ifrance.com jimdo.com kimsufi.com livejournal.com
    ning.com no.com notlong.com proboards.com qc.com sa.com se.com selfip.com szm.com t35.com
    tripod.com uk.com unlugar.com uribl.com us.com uy.com webs.com weebly.com wordpress.com
    xanga.com za.com ath.cx com.cy wz.cz funpic.de gmxhome.de com.ec co.ee fie.ee pri.ee com.eg
    com.es iespana.es nom.es org.ge pvt.ge com.gr com.hk org.hk com.hn com.hr extra.hu co.id web.id
    co.il org.il co.im net.im org.im co.in firm.in ind.in org.in co.jp co.ke ac.kr co.kr go.kr
    ne.kr or.kr com.mk net.mk com.mx edu.mx org.mx com.my net.my cjb.net galocalculter.net gb.net
    homeip.net ovh.net selfip.net whsites.net za.net com.np co.nz altervista.org com.pe com.pk
    net.pk com.pl home.pl info.pl interia.pl kalisz.pl net.pl poznan.pl republika.pl rzeszow.pl
    waw.pl wroc.pl za.pl com.pt org.pt com.py com.ro go.ro home.ro co.rs in.rs by.ru com.ru
    front.ru hotbox.ru hotmail.ru kaluga.ru land.ru mail.ru msk.ru net.ru newmail.ru nightmail.ru
    nm.ru nov.ru nsk.ru org.ru oryol.ru pochta.ru pochtamt.ru pop3.ru rbcmail.ru smtp.ru spb.ru
    tomsk.ru tyumen.ru ucoz.ru vrn.ru com.sa com.sc com.sg edu.sk mail15.su mail333.su pisem.su
    net.tc ac.th co.th go.th in.th page.tl kickme.to bel.tr com.tr gen.tr gov.tr k12.tr org.tr
    uk.tt co.tv com.tw edu.tw idv.tw net.tw go.tz co.ua com.ua dn.ua dp.ua if.ua in.ua kiev.ua
    kr.ua ks.ua lg.ua mk.ua net.ua org.ua pl.ua co.uk me.uk org.uk com.uy com.vc com.ve gob.ve
    com.vn co.za );

our %uribl_thirdlevel_tlds = map { $_ => undef } # 16 entries
    qw( syearg.cjb.net hop.clickbank.net bkc.destructionrecline.com bkc.divisiongenerate.com
    web.fc2.com blog.friendster.com profile.live.com spaces.live.com blu.livefilestore.com
    sakura.ne.jp web.officelive.com bkc.planesvent.com cloud.prohosting.com com.sapo.pt no.sapo.pt
    homepage.t-online.de );

# SURBL Second and Third level TLDs
our %surbl_secondlevel_tlds = map { $_ => undef } # 2426 entries
    qw( ac.ae co.ae com.ae mil.ae name.ae net.ae org.ae pro.ae sch.ae accident-investigation.aero
    accident-prevention.aero aerobatic.aero aeroclub.aero aerodrome.aero agents.aero
    air-surveillance.aero air-traffic-control.aero aircraft.aero airline.aero airport.aero
    airtraffic.aero ambulance.aero amusement.aero association.aero author.aero ballooning.aero
    broker.aero caa.aero cargo.aero catering.aero certification.aero championship.aero charter.aero
    civilaviation.aero club.aero conference.aero consultant.aero consulting.aero control.aero
    council.aero crew.aero design.aero dgca.aero educator.aero emergency.aero engine.aero
    engineer.aero entertainment.aero equipment.aero exchange.aero express.aero federation.aero
    flight.aero freight.aero fuel.aero gliding.aero government.aero groundhandling.aero group.aero
    hanggliding.aero homebuilt.aero insurance.aero journal.aero journalist.aero leasing.aero
    logistics.aero magazine.aero maintenance.aero marketplace.aero media.aero microlight.aero
    modelling.aero navigation.aero parachuting.aero paragliding.aero passenger-association.aero
    pilot.aero press.aero production.aero recreation.aero repbody.aero res.aero research.aero
    rotorcraft.aero safety.aero scientist.aero services.aero show.aero skydiving.aero software.aero
    student.aero taxi.aero trader.aero trading.aero trainer.aero union.aero workinggroup.aero
    works.aero ac.am co.am com.am edu.am int.am mil.am net.am org.am tld.am com.ar
    congresodelalengua3.ar edu.ar educ.ar gobiernoelectronico.ar int.ar mecon.ar mil.ar nacion.ar
    net.ar nic.ar org.ar promocion.ar retina.ar uba.ar ac.at co.at gv.at or.at priv.at act.au
    asn.au com.au conf.au csiro.au edu.au id.au info.au net.au nsw.au nt.au org.au otc.au oz.au
    qld.au sa.au tas.au telememo.au vic.au wa.au biz.az com.az edu.az info.az int.az mil.az name.az
    net.az org.az pp.az pro.az co.ba com.ba edu.ba mil.ba net.ba org.ba rs.ba unbi.ba unsa.ba ac.bd
    com.bd edu.bd mil.bd net.bd org.bd ac.be belgie.be dns.be fgov.be 0.bg 1.bg 2.bg 3.bg 4.bg 5.bg
    6.bg 7.bg 8.bg 9.bg a.bg b.bg c.bg d.bg e.bg f.bg g.bg h.bg i.bg j.bg k.bg l.bg m.bg n.bg o.bg
    p.bg q.bg r.bg s.bg t.bg u.bg v.bg w.bg x.bg y.bg z.bg biz.bh cc.bh com.bh edu.bh info.bh
    net.bh org.bh adm.br adv.br agr.br am.br arq.br art.br ato.br bio.br blog.br bmd.br can.br
    cim.br cng.br cnt.br com.br dpn.br ecn.br edu.br eng.br esp.br etc.br eti.br far.br flog.br
    fm.br fnd.br fot.br fst.br g12.br ggf.br gov.br imb.br ind.br inf.br jor.br jus.br lel.br
    mat.br med.br mil.br mus.br net.br nom.br not.br ntr.br odo.br org.br ppg.br pro.br psc.br
    psi.br qsl.br rec.br slg.br srv.br tmp.br trd.br tur.br tv.br vet.br vlog.br wiki.br zlg.br
    com.bs edu.bs net.bs org.bs we.bs com.by mil.by na.by com.bz edu.bz net.bz org.bz ab.ca bc.ca
    gc.ca mb.ca nb.ca nf.ca nl.ca ns.ca nt.ca nu.ca on.ca pe.ca qc.ca sk.ca yk.ca cc.cc co.cc
    com.ch net.ch org.ch gob.cl ac.cn ah.cn bj.cn com.cn cq.cn edu.cn fj.cn gd.cn gov.cn gs.cn
    gx.cn gz.cn ha.cn hb.cn he.cn hi.cn hk.cn hl.cn hn.cn jl.cn js.cn jx.cn ln.cn mil.cn mo.cn
    net.cn nm.cn nx.cn org.cn qh.cn sc.cn sd.cn sh.cn sn.cn sx.cn tj.cn tw.cn xj.cn xz.cn yn.cn
    zj.cn arts.co com.co edu.co firm.co info.co int.co mil.co net.co nom.co org.co rec.co store.co
    web.co 110mb.com 150m.com 9k.com ac.com addr.com angelfire.com appspot.com ar.com au.com
    blog.com blog4ever.com blogspot.com br.com bravehost.com bravepages.com cn.com de.com eu.com
    freehostia.com freeservercity.com gb.com geocities.com getmyip.com go.com googlegroups.com
    googlepages.com gratishost.com hostevo.com hu.com ibelgique.com idoo.com ifrance.com
    iquebec.com isuisse.com iwebsource.com jimdo.com jpn.com kimsufi.com kr.com livejournal.com
    mooo.com myvnc.com netfirms.com netsolhost.com ning.com no.com notlong.com proboards.com qc.com
    ru.com sa.com se.com selfip.com servebbs.com serveftp.com sosblog.com szm.com t35.com
    tripod.com uk.com unlugar.com us.com uy.com webcindario.com webs.com weebly.com wordpress.com
    xanga.com za.com ath.cx ac.cy biz.cy com.cy ekloges.cy info.cy ltd.cy name.cy net.cy org.cy
    parliament.cy press.cy pro.cy tm.cy ic.cz wz.cz xf.cz 1blu.de funpic.de gmxhome.de co.dk com.ec
    edu.ec fin.ec info.ec k12.ec med.ec mil.ec net.ec org.ec pro.ec aip.ee co.ee com.ee edu.ee
    fie.ee lib.ee med.ee org.ee pri.ee riik.ee com.eg edu.eg eun.eg mil.eg net.eg org.eg sci.eg
    com.es edu.es gob.es iespana.es nom.es org.es helloweb.eu aland.fi aeroport.fr assedic.fr
    asso.fr avocat.fr avoues.fr barreau.fr cci.fr chambagri.fr chirurgiens-dentistes.fr com.fr
    experts-comptables.fr free.fr geometre-expert.fr gouv.fr greta.fr huissier-justice.fr
    medecin.fr nom.fr notaires.fr pharmacien.fr port.fr prd.fr presse.fr tm.fr unblog.fr
    veterinaire.fr com.ge edu.ge mil.ge net.ge org.ge pvt.ge ac.gg alderney.gg co.gg guernsey.gg
    ind.gg ltd.gg net.gg org.gg sark.gg sch.gg com.gr edu.gr net.gr org.gr com.hk edu.hk idv.hk
    net.hk org.hk com.hn edu.hn gob.hn mil.hn net.hn org.hn com.hr from.hr iz.hr name.hr 2000.hu
    ac.hu agrar.hu bolt.hu casino.hu city.hu co.hu edu.hu erotica.hu erotika.hu extra.hu film.hu
    forum.hu games.hu hotel.hu info.hu ingatlan.hu jogasz.hu konyvelo.hu lakas.hu media.hu news.hu
    nui.hu org.hu priv.hu reklam.hu sex.hu shop.hu sport.hu suli.hu szex.hu tm.hu tozsde.hu
    utazas.hu video.hu ac.id co.id go.id mil.id net.id or.id sch.id web.id ac.il co.il idf.il
    k12.il muni.il net.il org.il ac.im co.im eu.im il.im net.im nic.im org.im ro.im ac.in co.in
    edu.in ernet.in firm.in gen.in ind.in mil.in net.in nic.in org.in res.in com.io mil.io net.io
    org.io ac.ir co.ir id.ir net.ir org.ir sch.ir com.is edu.is int.is net.is org.is ag.it
    agrigento.it al.it alessandria.it alto-adige.it altoadige.it an.it ancona.it ao.it aosta.it
    aoste.it ap.it aq.it aquila.it ar.it arezzo.it ascoli-piceno.it ascolipiceno.it asti.it at.it
    av.it avellino.it ba.it balsan.it bari.it barletta-andria-trani.it barlettaandriatrani.it
    belluno.it benevento.it bergamo.it bg.it bi.it biella.it bl.it bn.it bo.it bologna.it
    bolzano.it bozen.it br.it brescia.it brindisi.it bs.it bz.it ca.it cagliari.it caltanissetta.it
    campobasso.it caserta.it catania.it catanzaro.it cb.it ce.it ch.it chieti.it cl.it cn.it co.it
    como.it cosenza.it cr.it cremona.it crotone.it cs.it ct.it cuneo.it cz.it edu.it en.it enna.it
    fc.it fe.it fermo.it ferrara.it fg.it fi.it firenze.it florence.it foggia.it forli-cesena.it
    forlicesena.it fr.it frosinone.it ge.it genoa.it genova.it go.it gorizia.it gr.it grosseto.it
    im.it imperia.it is.it isernia.it kr.it la-spezia.it laquila.it laspezia.it latina.it lc.it
    le.it lecce.it lecco.it li.it livorno.it lo.it lodi.it lt.it lu.it lucca.it macerata.it
    mantova.it massa-carrara.it massacarrara.it matera.it mc.it me.it messina.it mi.it milan.it
    milano.it mn.it mo.it modena.it monza.it ms.it mt.it na.it naples.it napoli.it no.it novara.it
    nu.it nuoro.it or.it oristano.it pa.it padova.it padua.it palermo.it parma.it pavia.it pc.it
    pd.it pe.it perugia.it pesaro-urbino.it pesarourbino.it pescara.it pg.it pi.it piacenza.it
    pisa.it pistoia.it pn.it po.it pordenone.it potenza.it pr.it prato.it pt.it pu.it pv.it pz.it
    ra.it ragusa.it ravenna.it rc.it re.it reggio-calabria.it reggio-emilia.it reggiocalabria.it
    reggioemilia.it rg.it ri.it rieti.it rimini.it rm.it rn.it ro.it roma.it rome.it rovigo.it
    sa.it salerno.it sassari.it savona.it si.it siena.it siracusa.it so.it sondrio.it sp.it sr.it
    ss.it suedtirol.it sv.it ta.it taranto.it te.it teramo.it terni.it tn.it to.it torino.it tp.it
    tr.it trapani.it trentino.it trento.it treviso.it trieste.it ts.it turin.it tv.it ud.it
    udine.it va.it varese.it vb.it vc.it ve.it venezia.it venice.it verbania.it vercelli.it
    verona.it vi.it vibo-valentia.it vibovalentia.it vicenza.it viterbo.it vr.it vt.it vv.it ac.je
    co.je ind.je jersey.je ltd.je net.je org.je sch.je ac.jp ad.jp aichi.jp akita.jp aomori.jp
    chiba.jp co.jp ed.jp ehime.jp fukui.jp fukuoka.jp fukushima.jp geocities.jp gifu.jp go.jp gr.jp
    gunma.jp hiroshima.jp hokkaido.jp hyogo.jp ibaraki.jp ishikawa.jp iwate.jp kagawa.jp
    kagoshima.jp kanagawa.jp kanazawa.jp kawasaki.jp kitakyushu.jp kobe.jp kochi.jp kumamoto.jp
    kyoto.jp lg.jp matsuyama.jp mie.jp miyagi.jp miyazaki.jp nagano.jp nagasaki.jp nagoya.jp
    nara.jp ne.jp net.jp niigata.jp oita.jp okayama.jp okinawa.jp or.jp org.jp osaka.jp saga.jp
    saitama.jp sapporo.jp sendai.jp shiga.jp shimane.jp shizuoka.jp takamatsu.jp tochigi.jp
    tokushima.jp tokyo.jp tottori.jp toyama.jp utsunomiya.jp wakayama.jp yamagata.jp yamaguchi.jp
    yamanashi.jp yokohama.jp ac.ke co.ke go.ke ne.ke new.ke or.ke sc.ke co.kg com.kg edu.kg mil.kg
    net.kg org.kg ac.kr busan.kr chungbuk.kr chungnam.kr co.kr daegu.kr daejeon.kr es.kr gangwon.kr
    go.kr gwangju.kr gyeongbuk.kr gyeonggi.kr gyeongnam.kr hs.kr incheon.kr jeju.kr jeonbuk.kr
    jeonnam.kr kg.kr kyonggi.kr mil.kr ms.kr ne.kr or.kr pe.kr re.kr sc.kr seoul.kr ulsan.kr com.kz
    edu.kz mil.kz net.kz org.kz c.la com.la edu.la info.la int.la net.la org.la per.la com.li
    net.li org.li mil.lt asn.lv at.lv ch.lv com.lv conf.lv de.lv edu.lv id.lv mil.lv net.lv org.lv
    ac.ma asso.ma co.ma net.ma org.ma press.ma ac.me co.me edu.me gbr.me its.me net.me org.me
    priv.me com.mk edu.mk inf.mk name.mk net.mk org.mk com.mn edu.mn museum.mn org.mn music.mobi
    weather.mobi au.ms br.ms cn.ms de.ms fr.ms hk.ms info.ms net.ms shop.ms us.ms ac.mu co.mu
    com.mu net.mu or.mu org.mu com.mx edu.mx gob.mx net.mx org.mx com.my edu.my mil.my name.my
    net.my org.my cjb.net clickbank.net de.net gb.net gigazu.net homeip.net ovh.net prserv.net
    redirectme.net se.net selfip.net t35.net uk.net whsites.net za.net aa.no aarborte.no aejrie.no
    afjord.no agdenes.no ah.no aknoluokta.no akrehamn.no al.no alaheadju.no alesund.no algard.no
    alstahaug.no alta.no alvdal.no amli.no amot.no andasuolo.no andebu.no andoy.no ardal.no
    aremark.no arendal.no arna.no aseral.no asker.no askim.no askoy.no askvoll.no asnes.no
    audnedaln.no aukra.no aure.no aurland.no aurskog-holand.no austevoll.no austrheim.no averoy.no
    badaddja.no bahcavuotna.no bahccavuotna.no baidar.no bajddar.no balat.no balestrand.no
    ballangen.no balsfjord.no bamble.no bardu.no barum.no batsfjord.no bearalvahki.no beardu.no
    beiarn.no berg.no bergen.no berlevag.no bievat.no bindal.no birkenes.no bjarkoy.no bjerkreim.no
    bjugn.no bodo.no bokn.no bomlo.no bremanger.no bronnoy.no bronnoysund.no brumunddal.no bryne.no
    bu.no budejju.no bygland.no bykle.no cahcesuolo.no davvenjarga.no davvesiida.no deatnu.no
    dep.no dielddanuorri.no divtasvuodna.no divttasvuotna.no donna.no dovre.no drammen.no
    drangedal.no drobak.no dyroy.no egersund.no eid.no eidfjord.no eidsberg.no eidskog.no
    eidsvoll.no eigersund.no elverum.no enebakk.no engerdal.no etne.no etnedal.no evenassi.no
    evenes.no evje-og-hornnes.no farsund.no fauske.no fedje.no fet.no fetsund.no fhs.no finnoy.no
    fitjar.no fjaler.no fjell.no fla.no flakstad.no flatanger.no flekkefjord.no flesberg.no
    flora.no floro.no fm.no folkebibl.no folldal.no forde.no forsand.no fosnes.no frana.no
    fredrikstad.no frei.no frogn.no froland.no frosta.no froya.no fuoisku.no fuossko.no fusa.no
    fylkesbibl.no fyresdal.no gaivuotna.no galsa.no gamvik.no gangaviika.no gaular.no gausdal.no
    giehtavuoatna.no gildeskal.no giske.no gjemnes.no gjerdrum.no gjerstad.no gjesdal.no gjovik.no
    gloppen.no gol.no gran.no grane.no granvin.no gratangen.no grimstad.no grong.no grue.no
    gulen.no guovdageaidnu.no ha.no habmer.no hadsel.no hagebostad.no halden.no halsa.no hamar.no
    hamaroy.no hammarfeasta.no hammerfest.no hapmir.no haram.no hareid.no harstad.no hasvik.no
    hattfjelldal.no haugesund.no hemne.no hemnes.no hemsedal.no herad.no hitra.no hjartdal.no
    hjelmeland.no hl.no hm.no hobol.no hof.no hokksund.no hol.no hole.no holmestrand.no holtalen.no
    honefoss.no hornindal.no horten.no hoyanger.no hoylandet.no hurdal.no hurum.no hvaler.no
    hyllestad.no ibestad.no idrett.no inderoy.no iveland.no ivgu.no jan-mayen.no jessheim.no
    jevnaker.no jolster.no jondal.no jorpeland.no kafjord.no karasjohka.no karasjok.no karlsoy.no
    karmoy.no kautokeino.no kirkenes.no klabu.no klepp.no kommune.no kongsberg.no kongsvinger.no
    kopervik.no kraanghke.no kragero.no kristiansand.no kristiansund.no krodsherad.no
    krokstadelva.no kvafjord.no kvalsund.no kvam.no kvanangen.no kvinesdal.no kvinnherad.no
    kviteseid.no kvitsoy.no laakesvuemie.no lahppi.no langevag.no lardal.no larvik.no lavagis.no
    lavangen.no leangaviika.no lebesby.no leikanger.no leirfjord.no leirvik.no leka.no leksvik.no
    lenvik.no lerdal.no lesja.no levanger.no lier.no lierne.no lillehammer.no lillesand.no
    lindas.no lindesnes.no loabat.no lodingen.no lom.no loppa.no lorenskog.no loten.no lund.no
    lunner.no luroy.no luster.no lyngdal.no lyngen.no malatvuopmi.no malselv.no malvik.no mandal.no
    marker.no marnardal.no masfjorden.no masoy.no matta-varjjat.no meland.no meldal.no melhus.no
    meloy.no meraker.no midsund.no midtre-gauldal.no mil.no mjondalen.no mo-i-rana.no moareke.no
    modalen.no modum.no molde.no mosjoen.no moskenes.no moss.no mosvik.no mr.no muosat.no museum.no
    naamesjevuemie.no namdalseid.no namsos.no namsskogan.no nannestad.no naroy.no narviika.no
    narvik.no naustdal.no navuotna.no nedre-eiker.no nesna.no nesodden.no nesoddtangen.no
    nesseby.no nesset.no nissedal.no nittedal.no nl.no nord-aurdal.no nord-fron.no nord-odal.no
    norddal.no nordkapp.no nordre-land.no nordreisa.no nore-og-uvdal.no notodden.no notteroy.no
    nt.no odda.no of.no oksnes.no ol.no omasvuotna.no oppdal.no oppegard.no orkanger.no orkdal.no
    orland.no orskog.no orsta.no osen.no oslo.no osoyro.no osteroy.no ostre-toten.no overhalla.no
    ovre-eiker.no oyer.no oygarden.no oystre-slidre.no porsanger.no porsangu.no porsgrunn.no
    priv.no rade.no radoy.no rahkkeravju.no raholt.no raisa.no rakkestad.no ralingen.no rana.no
    randaberg.no rauma.no rendalen.no rennebu.no rennesoy.no rindal.no ringebu.no ringerike.no
    ringsaker.no risor.no rissa.no rl.no roan.no rodoy.no rollag.no romsa.no romskog.no roros.no
    rost.no royken.no royrvik.no ruovat.no rygge.no salangen.no salat.no saltdal.no samnanger.no
    sandefjord.no sandnes.no sandnessjoen.no sandoy.no sarpsborg.no sauda.no sauherad.no sel.no
    selbu.no selje.no seljord.no sf.no siellak.no sigdal.no siljan.no sirdal.no skanit.no
    skanland.no skaun.no skedsmo.no skedsmokorset.no ski.no skien.no skierva.no skiptvet.no
    skjak.no skjervoy.no skodje.no slattum.no smola.no snaase.no snasa.no snillfjord.no snoasa.no
    sogndal.no sogne.no sokndal.no sola.no solund.no somna.no sondre-land.no songdalen.no
    sor-aurdal.no sor-fron.no sor-odal.no sor-varanger.no sorfold.no sorreisa.no sortland.no
    sorum.no spjelkavik.no spydeberg.no st.no stange.no stat.no stathelle.no stavanger.no
    stavern.no steigen.no steinkjer.no stjordal.no stjordalshalsen.no stokke.no stor-elvdal.no
    stord.no stordal.no storfjord.no strand.no stranda.no stryn.no sula.no suldal.no sund.no
    sunndal.no surnadal.no svalbard.no sveio.no svelvik.no sykkylven.no tana.no tananger.no time.no
    tingvoll.no tinn.no tjeldsund.no tjome.no tm.no tokke.no tolga.no tonsberg.no torsken.no tr.no
    trana.no tranby.no tranoy.no troandin.no trogstad.no tromsa.no tromso.no trondheim.no trysil.no
    tvedestrand.no tydal.no tynset.no tysfjord.no tysnes.no tysvar.no ullensaker.no ullensvang.no
    ulvik.no unjarga.no utsira.no va.no vaapste.no vadso.no vaga.no vagan.no vagsoy.no vaksdal.no
    valle.no vang.no vanylven.no vardo.no varggat.no varoy.no vefsn.no vega.no vegarshei.no
    vennesla.no verdal.no verran.no vestby.no vestnes.no vestre-slidre.no vestre-toten.no
    vestvagoy.no vevelstad.no vf.no vgs.no vik.no vikna.no vindafjord.no voagat.no volda.no voss.no
    vossevangen.no xn--drbak-wua.no xn--leagaviika-52b.no xn--ostery-fya.no xn--tysvr-vra.no
    xn--unjrga-rta.no xn--vegrshei-c0a.no com.np edu.np mil.np net.np org.np mine.nu ac.nz co.nz
    cri.nz geek.nz gen.nz govt.nz iwi.nz maori.nz mil.nz net.nz org.nz school.nz 6a.org ae.org
    altervista.org dk.org eu.org iblogger.org za.org com.pe edu.pe gob.pe mil.pe net.pe nom.pe
    org.pe com.ph edu.ph i.ph mil.ph net.ph ngo.ph org.ph biz.pk com.pk edu.pk fam.pk gob.pk gok.pk
    gon.pk gop.pk gos.pk info.pk net.pk org.pk web.pk 6bone.pl agro.pl aid.pl art.pl atm.pl
    augustow.pl auto.pl babia-gora.pl bedzin.pl beskidy.pl bialowieza.pl bialystok.pl bielawa.pl
    bieszczady.pl biz.pl boleslawiec.pl bydgoszcz.pl bytom.pl cieszyn.pl com.pl czeladz.pl czest.pl
    dlugoleka.pl edu.pl elblag.pl elk.pl gda.pl gdansk.pl gdynia.pl gliwice.pl glogow.pl gmina.pl
    gniezno.pl gorlice.pl grajewo.pl gsm.pl home.pl ilawa.pl info.pl interia.pl interii.pl irc.pl
    jaworzno.pl jelenia-gora.pl jgora.pl kalisz.pl karpacz.pl kartuzy.pl kaszuby.pl katowice.pl
    kazimierz-dolny.pl kepno.pl ketrzyn.pl klodzko.pl kobierzyce.pl kolobrzeg.pl konin.pl
    konskowola.pl krakow.pl kutno.pl lapy.pl lebork.pl legnica.pl lezajsk.pl limanowa.pl lodz.pl
    lomza.pl lowicz.pl lubin.pl lublin.pl lukow.pl mail.pl malbork.pl malopolska.pl mazowsze.pl
    mazury.pl mbone.pl med.pl media.pl miasta.pl mielec.pl mielno.pl mil.pl mragowo.pl naklo.pl
    net.pl ngo.pl nieruchomosci.pl nom.pl nowaruda.pl nysa.pl olawa.pl olecko.pl olkusz.pl
    olsztyn.pl one.pl opoczno.pl opole.pl org.pl ostroda.pl ostroleka.pl ostrowiec.pl ostrowwlkp.pl
    pc.pl pila.pl pisz.pl podhale.pl podlasie.pl polkowice.pl pomorskie.pl pomorze.pl powiat.pl
    poznan.pl priv.pl prochowice.pl pruszkow.pl przeworsk.pl pulawy.pl radom.pl rawa-maz.pl
    realestate.pl rel.pl republika.pl rybnik.pl rzeszow.pl sanok.pl sejny.pl sex.pl shop.pl
    sklep.pl skoczow.pl slask.pl slupsk.pl sopot.pl sos.pl sosnowiec.pl stalowa-wola.pl
    starachowice.pl stargard.pl suwalki.pl swidnica.pl swiebodzin.pl swinoujscie.pl szczecin.pl
    szczytno.pl szkola.pl targi.pl tarnobrzeg.pl tgory.pl tm.pl torun.pl tourism.pl travel.pl
    turek.pl turystyka.pl tychy.pl usenet.pl ustka.pl walbrzych.pl warmia.pl warszawa.pl waw.pl
    wegrow.pl wielun.pl wlocl.pl wloclawek.pl wodzislaw.pl wolomin.pl wroc.pl wroclaw.pl xorg.pl
    za.pl zachpomor.pl zagan.pl zakopane.pl zarow.pl zgora.pl zgorzelec.pl ac.pr biz.pr com.pr
    edu.pr est.pr info.pr isla.pr name.pr net.pr org.pr pro.pr prof.pr aca.pro bar.pro cpa.pro
    eng.pro jur.pro law.pro med.pro com.pt edu.pt int.pt net.pt nome.pt org.pt publ.pt sapo.pt
    com.py edu.py net.py org.py arts.ro com.ro firm.ro go.ro home.ro info.ro nom.ro nt.ro org.ro
    rec.ro store.ro tm.ro www.ro ac.rs co.rs edu.rs in.rs org.rs ac.ru adygeya.ru altai.ru amur.ru
    amursk.ru arkhangelsk.ru astrakhan.ru baikal.ru bashkiria.ru belgorod.ru bir.ru blog.ru
    bryansk.ru buryatia.ru by.ru cbg.ru chat.ru chel.ru chelyabinsk.ru chita.ru chukotka.ru
    chuvashia.ru cmw.ru com.ru dagestan.ru dudinka.ru e-burg.ru edu.ru fareast.ru front.ru
    grozny.ru hotbox.ru hotmail.ru hu2.ru hut2.ru id.ru int.ru irkutsk.ru ivanovo.ru izhevsk.ru
    jamal.ru jar.ru jino-net.ru joshkar-ola.ru k-uralsk.ru kalmykia.ru kaluga.ru kamchatka.ru
    karelia.ru kazan.ru kchr.ru kemerovo.ru khabarovsk.ru khakassia.ru khv.ru kirov.ru kms.ru
    koenig.ru komi.ru kostroma.ru krasnoyarsk.ru kuban.ru kurgan.ru kursk.ru kustanai.ru kuzbass.ru
    land.ru lipetsk.ru magadan.ru magnitka.ru mail.ru mail2k.ru mari.ru mari-el.ru marine.ru mil.ru
    mordovia.ru mosreg.ru msk.ru murmansk.ru mytis.ru nakhodka.ru nalchik.ru narod.ru net.ru
    newmail.ru nextmail.ru nightmail.ru nkz.ru nm.ru nnov.ru norilsk.ru nov.ru novosibirsk.ru
    nsk.ru omsk.ru orenburg.ru org.ru oryol.ru oskol.ru palana.ru penza.ru perm.ru pochta.ru
    pochtamt.ru pop3.ru pp.ru pskov.ru ptz.ru pyatigorsk.ru radikal.ru rbcmail.ru rm.ru rnd.ru
    rubtsovsk.ru ryazan.ru sakhalin.ru samara.ru saratov.ru simbirsk.ru smolensk.ru smtp.ru snz.ru
    sol.ru spb.ru stavropol.ru stv.ru surgut.ru syzran.ru tambov.ru tatarstan.ru test.ru tom.ru
    tomsk.ru tsaritsyn.ru tsk.ru tula.ru tuva.ru tver.ru tyumen.ru ucoz.ru udm.ru udmurtia.ru
    ulan-ude.ru vdonsk.ru vladikavkaz.ru vladimir.ru vladivostok.ru volgograd.ru vologda.ru
    voronezh.ru vrn.ru vyatka.ru web-soft.ru yakutia.ru yamal.ru yaroslavl.ru yekaterinburg.ru
    yuzhno-sakhalinsk.ru z8.ru zgrad.ru zmail.ru com.sa edu.sa med.sa net.sa org.sa pub.sa sch.sa
    com.sc edu.sc net.sc org.sc a.se ab.se ac.se b.se bd.se brand.se c.se d.se e.se f.se fh.se
    fhsk.se fhv.se g.se h.se i.se k.se komforb.se kommunalforbund.se komvux.se l.se lanarb.se
    lanbib.se m.se mil.se n.se naturbruksgymn.se o.se org.se p.se parti.se pp.se press.se r.se s.se
    sshn.se t.se tm.se u.se w.se x.se y.se z.se com.sg edu.sg idn.sg net.sg org.sg per.sg edu.sk
    host.sk mil.sk co.st com.st consulado.st corp.st edu.st embaixada.st mil.st net.st org.st
    principe.st saotome.st societe.st store.st url.st mail15.su mail333.su pisem.su at.tc au.tc
    be.tc bg.tc br.tc ch.tc cz.tc de.tc dk.tc edu.tc es.tc eu.tc fr.tc hk.tc hu.tc ie.tc int.tc
    it.tc kr.tc mx.tc net.tc no.tc perso.tc ph.tc pl.tc pro.tc ru.tc se.tc shop.tc site.tc th.tc
    ua.tc uk.tc us.tc ac.th co.th go.th in.th mi.th net.th or.th es.tl page.tl 163.to com.to edu.to
    fx.to kickme.to kwik.to mil.to net.to org.to t3.to uk.to us.to av.tr bbs.tr bel.tr biz.tr
    com.tr dr.tr edu.tr gen.tr gov.tr info.tr k12.tr mil.tr name.tr net.tr org.tr pol.tr web.tr
    aero.tt at.tt au.tt be.tt biz.tt ca.tt co.tt com.tt de.tt dk.tt edu.tt es.tt eu.tt fr.tt
    info.tt int.tt it.tt mobi.tt museum.tt name.tt net.tt nic.tt org.tt pro.tt se.tt travel.tt
    uk.tt us.tt co.tv com.tv net.tv org.tv club.tw com.tw ebiz.tw edu.tw game.tw idv.tw mil.tw
    net.tw org.tw ac.tz co.tz go.tz ne.tz or.tz biz.ua cherkassy.ua chernigov.ua chernovtsy.ua
    ck.ua cn.ua co.ua com.ua crimea.ua cv.ua dn.ua dnepropetrovsk.ua donetsk.ua dp.ua edu.ua if.ua
    in.ua ivano-frankivsk.ua kh.ua kharkov.ua kherson.ua khmelnitskiy.ua kiev.ua kirovograd.ua
    km.ua kr.ua ks.ua kv.ua lg.ua lugansk.ua lutsk.ua lviv.ua mk.ua net.ua nikolaev.ua od.ua
    odessa.ua org.ua pl.ua poltava.ua pp.ua rovno.ua rv.ua sebastopol.ua sumy.ua te.ua ternopil.ua
    uz.ua uzhgorod.ua vinnica.ua vn.ua zaporizhzhe.ua zhitomir.ua zp.ua zt.ua ac.uk bl.uk
    british-library.uk co.uk edu.uk icnet.uk jet.uk ltd.uk me.uk mil.uk mod.uk
    national-library-scotland.uk nel.uk net.uk nhs.uk nic.uk nls.uk org.uk parliament.uk plc.uk
    police.uk sch.uk com.uy edu.uy gub.uy mil.uy net.uy org.uy co.uz com.uz net.uz org.uz com.vc
    edu.vc mil.vc net.vc org.vc arts.ve bib.ve co.ve com.ve e12.ve edu.ve firm.ve gob.ve info.ve
    int.ve mil.ve net.ve nom.ve org.ve rec.ve store.ve tec.ve web.ve ac.vn biz.vn com.vn edu.vn
    health.vn info.vn int.vn name.vn net.vn org.vn pro.vn asso.ws com.ws edu.ws net.ws org.ws ac.za
    alt.za bourse.za city.za co.za edu.za law.za mil.za net.za ngo.za nom.za org.za school.za tm.za
    web.za );

our %surbl_thirdlevel_tlds = map { $_ => undef } # 135 entries
    qw( gs.aa.no gs.ah.no pref.aichi.jp nes.akershus.no pref.akita.jp pref.aomori.jp web.aplus.net
    co.at.lv or.at.lv co.at.tc com.au.ms com.au.tc gs.bu.no nes.buskerud.no city.chiba.jp
    pref.chiba.jp lkd.co.im ltd.co.im plc.co.im act.edu.au nsw.edu.au nt.edu.au qld.edu.au
    sa.edu.au tas.edu.au vic.edu.au wa.edu.au pref.ehime.jp web.fc2.com gs.fm.no
    blog.friendster.com pref.fukui.jp city.fukuoka.jp pref.fukuoka.jp pref.fukushima.jp
    pref.gifu.jp docs.google.com pref.gunma.jp os.hedmark.no valer.hedmark.no city.hiroshima.jp
    pref.hiroshima.jp gs.hl.no gs.hm.no pref.hokkaido.jp os.hordaland.no pref.hyogo.jp
    pref.ibaraki.jp pref.ishikawa.jp pref.iwate.jp gs.jan-mayen.no pref.kagawa.jp pref.kagoshima.jp
    pref.kanagawa.jp city.kawasaki.jp city.kitakyushu.jp city.kobe.jp pref.kochi.jp
    pref.kumamoto.jp city.kyoto.jp pref.kyoto.jp groups.live.com profile.live.com skydrive.live.com
    spaces.live.com bay.livefilestore.com blu.livefilestore.com pref.mie.jp pref.miyagi.jp
    pref.miyazaki.jp heroy.more-og-romsdal.no sande.more-og-romsdal.no gs.mr.no spaces.msn.com
    pref.nagano.jp pref.nagasaki.jp city.nagoya.jp pref.nara.jp sakura.ne.jp city.niigata.jp
    pref.niigata.jp gs.nl.no bo.nordland.no heroy.nordland.no gs.nt.no gs.of.no web.officelive.com
    pref.oita.jp city.okayama.jp pref.okayama.jp pref.okinawa.jp gs.ol.no city.osaka.jp
    pref.osaka.jp gs.oslo.no valer.ostfold.no cloud.prohosting.com gs.rl.no pref.saga.jp
    city.saitama.jp pref.saitama.jp com.sapo.pt do.sapo.pt home.sapo.pt no.sapo.pt paginas.sapo.pt
    city.sapporo.jp city.sendai.jp gs.sf.no pref.shiga.jp pref.shimane.jp city.shizuoka.jp
    pref.shizuoka.jp gs.st.no gs.svalbard.no homepage.t-online.de bo.telemark.no gs.tm.no
    pref.tochigi.jp pref.tokushima.jp metro.tokyo.jp pref.tottori.jp pref.toyama.jp gs.tr.no
    co.uk.tc me.uk.tc org.uk.tc gs.va.no sande.vestfold.no gs.vf.no pref.wakayama.jp
    pref.yamagata.jp pref.yamaguchi.jp pref.yamanashi.jp city.yokohama.jp );
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.