Re: lastest uribl plugin
[email protected] (Ed McLain)
| Newsgroups | perl.qpsmtpd |
|---|---|
| Message-ID | <C95C6D8F.30588%[email protected]> |
Steve's email just reminded me that I had made some updates to Jared's plugin to fix the same quoted-printable issue along with a few others that we ran into while testing and moving into production on our systems. It would be nice if there was an easy way to update the tld_list.pl and I'm planning on tackling that once I get a free minute. We run about 5M emails through this plugin daily and so far no issues. -- Ed McLain From: Jared Johnson <[email protected]<mailto:[email protected]>> Date: Sun, 8 Aug 2010 01:09:04 -0500 To: qpsmtpd <[email protected]<mailto:[email protected]>> Subject: lastest uribl plugin So I just finished testing my latest URIBL plugin iteration; this iteration is not yet in production and I'm still planning on getting git back in line, but it's significantly changed from the last version posted and anyway I'm really having fun so I thought I'd send an update :) I'm also hoping maybe someone will be interested not only in volunteering to test the functionality of the async changes I posted earlier, but also in putting together an async version of resolve_shortened_uris(). The version in the non-async plugin goes back to IO::Select to fire off all connection attempts + queries asynchronously, but still blocks until it gets results, so it's not suitable for the async daemon, I'm assuming that version will need to use Danga::Socket magic. I'm also very interested in the idea of immediately interrupting the wait for queries to come back in the event that we get a URIBL result that can result in rejection. I'm pretty sure this can't be done with the existing tools in the non-async plugin but it could be done in the async plugin, and after all we do hope to switch to that architecture someday :) I had to completely revamp find_uris(), inspired by the need to find shortening-paths in parameterized redirects, but also improved recognition of parameterized redirects and obfuscated IP addresses... while making RE performance 3x slower :/ Fortunately 3x slower is still something like 10,000 lines/sec. -Jared
tld_lists.pl
(text/x-perl-script, 44 KB)
# Lists of one, two, and three level TLDs
# generated Fri Jul 30 12:21:06 2010 by gen_tld_lists on dcrbl1
# First-level TLDs from http://data.iana.org/TLD/tlds-alpha-by-domain.txt
# Version 2010072901, Last Updated Fri Jul 30 14:07:01 2010 UTC
# Added from feed data: yu
# 289 entries
our %firstlevel_tlds = map { $_ => undef } qw(
ac ad ae aero af ag ai al am an ao aq ar arpa as asia at au aw ax az ba bb bd be bf bg bh bi
biz bj bm bn bo br bs bt bv bw by bz ca cat cc cd cf cg ch ci ck cl cm cn co com coop cr cu cv
cx cy cz de dj dk dm do dz ec edu ee eg er es et eu fi fj fk fm fo fr ga gb gd ge gf gg gh gi
gl gm gn gov gp gq gr gs gt gu gw gy hk hm hn hr ht hu id ie il im in info int io iq ir is it
je jm jo jobs jp ke kg kh ki km kn kp kr kw ky kz la lb lc li lk lr ls lt lu lv ly ma mc md me
mg mh mil mk ml mm mn mo mobi mp mq mr ms mt mu museum mv mw mx my mz na name nc ne net nf ng
ni nl no np nr nu nz om org pa pe pf pg ph pk pl pm pn pr pro ps pt pw py qa re ro rs ru rw sa
sb sc sd se sg sh si sj sk sl sm sn so sr st su sv sy sz tc td tel tf tg th tj tk tl tm tn to
tp tr travel tt tv tw tz ua ug uk us uy uz va vc ve vg vi vn vu wf ws xn--0zwm56d
xn--11b5bs3a9aj6g xn--80akhbyknj4f xn--9t4b11yi5a xn--deba0ad xn--fiqs8s xn--fiqz9s xn--g6w251d
xn--hgbk6aj7f53bba xn--hlcj6aya9esc7a xn--j6w193g xn--jxalpdlp xn--kgbechtv xn--kprw13d
xn--kpry57d xn--mgbaam7a8h xn--mgberp4a5d4ar xn--p1ai xn--wgbh1c xn--zckzah ye yt yu za zm zw
);
# URIBL Second and Third level TLDs
# 222 entries
our %uribl_secondlevel_tlds = map { $_ => undef } qw(
com.ar org.ar com.au net.au com.ba com.bd com.br etc.br eti.br gov.br ind.br inf.br nom.br
org.br na.by co.cc com.cn fj.cn gov.cn hk.cn net.cn org.cn qh.cn sd.cn sh.cn tw.cn com.co
110mb.com 150m.com angelfire.com blogspot.com br.com bravehost.com cn.com eu.com freehostia.com
gb.com geocities.com googlepages.com idoo.com ifrance.com jimdo.com kimsufi.com livejournal.com
ning.com no.com notlong.com proboards.com qc.com sa.com se.com selfip.com szm.com t35.com
tripod.com uk.com unlugar.com uribl.com us.com uy.com webs.com weebly.com wordpress.com
xanga.com za.com ath.cx com.cy wz.cz funpic.de gmxhome.de com.ec co.ee fie.ee pri.ee com.eg
com.es iespana.es nom.es org.ge pvt.ge com.gr com.hk org.hk com.hn com.hr extra.hu co.id web.id
co.il org.il co.im net.im org.im co.in firm.in ind.in org.in co.jp co.ke ac.kr co.kr go.kr
ne.kr or.kr com.mk net.mk com.mx edu.mx org.mx com.my net.my cjb.net galocalculter.net gb.net
homeip.net ovh.net selfip.net whsites.net za.net com.np co.nz altervista.org com.pe com.pk
net.pk com.pl home.pl info.pl interia.pl kalisz.pl net.pl poznan.pl republika.pl rzeszow.pl
waw.pl wroc.pl za.pl com.pt org.pt com.py com.ro go.ro home.ro co.rs in.rs by.ru com.ru
front.ru hotbox.ru hotmail.ru kaluga.ru land.ru mail.ru msk.ru net.ru newmail.ru nightmail.ru
nm.ru nov.ru nsk.ru org.ru oryol.ru pochta.ru pochtamt.ru pop3.ru rbcmail.ru smtp.ru spb.ru
tomsk.ru tyumen.ru ucoz.ru vrn.ru com.sa com.sc com.sg edu.sk mail15.su mail333.su pisem.su
net.tc ac.th co.th go.th in.th page.tl kickme.to bel.tr com.tr gen.tr gov.tr k12.tr org.tr
uk.tt co.tv com.tw edu.tw idv.tw net.tw go.tz co.ua com.ua dn.ua dp.ua if.ua in.ua kiev.ua
kr.ua ks.ua lg.ua mk.ua net.ua org.ua pl.ua co.uk me.uk org.uk com.uy com.vc com.ve gob.ve
com.vn co.za
);
# 16 entries
our %uribl_thirdlevel_tlds = map { $_ => undef } qw(
syearg.cjb.net hop.clickbank.net bkc.destructionrecline.com bkc.divisiongenerate.com
web.fc2.com blog.friendster.com profile.live.com spaces.live.com blu.livefilestore.com
sakura.ne.jp web.officelive.com bkc.planesvent.com cloud.prohosting.com com.sapo.pt no.sapo.pt
homepage.t-online.de
);
# SURBL Second and Third level TLDs
# From: http://www.surbl.org/tld/two-level-tlds
# http://www.surbl.org/tld/three-level-tlds
# 3801 entries
our %surbl_secondlevel_tlds = map { $_ => undef } qw(
com.ac edu.ac gov.ac mil.ac net.ac org.ac nom.ad ac.ae co.ae com.ae gov.ae mil.ae name.ae
net.ae org.ae pro.ae sch.ae accident-investigation.aero accident-prevention.aero aerobatic.aero
aeroclub.aero aerodrome.aero agents.aero air-surveillance.aero air-traffic-control.aero
aircraft.aero airline.aero airport.aero airtraffic.aero ambulance.aero amusement.aero
association.aero author.aero ballooning.aero broker.aero caa.aero cargo.aero catering.aero
certification.aero championship.aero charter.aero civilaviation.aero club.aero conference.aero
consultant.aero consulting.aero control.aero council.aero crew.aero design.aero dgca.aero
educator.aero emergency.aero engine.aero engineer.aero entertainment.aero equipment.aero
exchange.aero express.aero federation.aero flight.aero freight.aero fuel.aero gliding.aero
government.aero groundhandling.aero group.aero hanggliding.aero homebuilt.aero insurance.aero
journal.aero journalist.aero leasing.aero logistics.aero magazine.aero maintenance.aero
marketplace.aero media.aero microlight.aero modelling.aero navigation.aero parachuting.aero
paragliding.aero passenger-association.aero pilot.aero press.aero production.aero
recreation.aero repbody.aero res.aero research.aero rotorcraft.aero safety.aero scientist.aero
services.aero show.aero skydiving.aero software.aero student.aero taxi.aero trader.aero
trading.aero trainer.aero union.aero workinggroup.aero works.aero com.af edu.af gov.af net.af
org.af co.ag com.ag net.ag nom.ag org.ag com.ai edu.ai gov.ai net.ai off.ai org.ai com.al
edu.al gov.al inima.al net.al org.al soros.al tirana.al uniti.al upt.al ac.am co.am com.am
edu.am gov.am int.am mil.am net.am org.am tld.am com.an edu.an net.an org.an co.ao ed.ao gv.ao
it.ao og.ao pb.ao com.ar congresodelalengua3.ar edu.ar educ.ar gobiernoelectronico.ar gov.ar
int.ar mecon.ar mil.ar nacion.ar net.ar nic.ar org.ar promocion.ar retina.ar uba.ar e164.arpa
in-addr.arpa ip6.arpa iris.arpa uri.arpa urn.arpa gov.as ac.at co.at gv.at or.at priv.at act.au
asn.au com.au conf.au csiro.au edu.au gov.au id.au info.au net.au nsw.au nt.au org.au otc.au
oz.au qld.au sa.au tas.au telememo.au vic.au wa.au com.aw biz.az com.az edu.az gov.az info.az
int.az mil.az name.az net.az org.az pp.az pro.az co.ba com.ba edu.ba gov.ba mil.ba net.ba
org.ba rs.ba unbi.ba unsa.ba com.bb edu.bb gov.bb net.bb org.bb ac.bd com.bd edu.bd gov.bd
mil.bd net.bd org.bd ac.be belgie.be dns.be fgov.be gov.bf 0.bg 1.bg 2.bg 3.bg 4.bg 5.bg 6.bg
7.bg 8.bg 9.bg a.bg b.bg c.bg d.bg e.bg f.bg g.bg h.bg i.bg j.bg k.bg l.bg m.bg n.bg o.bg p.bg
q.bg r.bg s.bg t.bg u.bg v.bg w.bg x.bg y.bg z.bg biz.bh cc.bh com.bh edu.bh gov.bh info.bh
net.bh org.bh co.bi com.bi edu.bi or.bi org.bi com.bm edu.bm gov.bm net.bm org.bm com.bn edu.bn
net.bn org.bn com.bo edu.bo gob.bo gov.bo int.bo mil.bo net.bo org.bo tv.bo adm.br adv.br
agr.br am.br arq.br art.br ato.br bio.br blog.br bmd.br can.br cim.br cng.br cnt.br com.br
coop.br dpn.br ecn.br edu.br eng.br esp.br etc.br eti.br far.br flog.br fm.br fnd.br fot.br
fst.br g12.br ggf.br gov.br imb.br ind.br inf.br jor.br jus.br lel.br mat.br med.br mil.br
mus.br net.br nom.br not.br ntr.br odo.br org.br ppg.br pro.br psc.br psi.br qsl.br rec.br
slg.br srv.br tmp.br trd.br tur.br tv.br vet.br vlog.br wiki.br zlg.br com.bs edu.bs gov.bs
net.bs org.bs we.bs com.bt edu.bt gov.bt net.bt org.bt co.bw org.bw com.by gov.by mil.by na.by
com.bz edu.bz gov.bz net.bz org.bz ab.ca bc.ca gc.ca mb.ca nb.ca nf.ca nl.ca ns.ca nt.ca nu.ca
on.ca pe.ca qc.ca sk.ca yk.ca cc.cc co.cc com.cd gov.cd net.cd org.cd com.ch gov.ch net.ch
org.ch ac.ci asso.ci co.ci com.ci ed.ci edu.ci go.ci gouv.ci int.ci md.ci net.ci or.ci org.ci
presse.ci xn--aroport-bya.ci co.ck edu.ck gov.ck net.ck org.ck gob.cl gov.cl gov.cm ac.cn ah.cn
bj.cn com.cn cq.cn edu.cn fj.cn gd.cn gov.cn gs.cn gx.cn gz.cn ha.cn hb.cn he.cn hi.cn hk.cn
hl.cn hn.cn jl.cn js.cn jx.cn ln.cn mil.cn mo.cn net.cn nm.cn nx.cn org.cn qh.cn sc.cn sd.cn
sh.cn sn.cn sx.cn tj.cn tw.cn xj.cn xz.cn yn.cn zj.cn arts.co com.co edu.co firm.co gov.co
info.co int.co mil.co net.co nom.co org.co rec.co store.co web.co 110mb.com 150m.com 9k.com
ac.com addr.com angelfire.com appspot.com ar.com au.com blog.com blog4ever.com blogspot.com
br.com bravehost.com bravepages.com cn.com de.com eu.com freehostia.com freeservercity.com
gb.com geocities.com getmyip.com go.com googlegroups.com googlepages.com gratishost.com
hostevo.com hu.com ibelgique.com idoo.com ifrance.com iquebec.com isuisse.com iwebsource.com
jimdo.com jpn.com kimsufi.com kr.com livejournal.com mooo.com myvnc.com netfirms.com
netsolhost.com ning.com no.com notlong.com proboards.com qc.com ru.com sa.com se.com selfip.com
servebbs.com serveftp.com sosblog.com szm.com t35.com tripod.com uk.com unlugar.com us.com
uy.com webcindario.com webs.com weebly.com wordpress.com xanga.com za.com ac.cr co.cr ed.cr
fi.cr go.cr or.cr sa.cr com.cu edu.cu gov.cu inf.cu info.cu net.cu org.cu ath.cx gov.cx ac.cy
biz.cy com.cy ekloges.cy gov.cy info.cy ltd.cy name.cy net.cy org.cy parliament.cy press.cy
pro.cy tm.cy ic.cz wz.cz xf.cz 1blu.de funpic.de gmxhome.de co.dk com.dm edu.dm gov.dm net.dm
org.dm art.do com.do edu.do gob.do gov.do mil.do net.do org.do sld.do web.do art.dz ass.dz
asso.dz com.dz edu.dz gov.dz net.dz org.dz pol.dz com.ec edu.ec fin.ec gov.ec info.ec k12.ec
med.ec mil.ec net.ec org.ec pro.ec aip.ee co.ee com.ee edu.ee fie.ee gov.ee lib.ee med.ee
org.ee pri.ee riik.ee com.eg edu.eg eun.eg gov.eg mil.eg net.eg org.eg sci.eg com.er edu.er
gov.er ind.er mil.er net.er org.er com.es edu.es gob.es iespana.es nom.es org.es biz.et com.et
edu.et gov.et info.et name.et net.et org.et helloweb.eu aland.fi ac.fj biz.fj com.fj gov.fj
id.fj info.fj mil.fj name.fj net.fj org.fj pro.fj school.fj ac.fk co.fk com.fk gov.fk net.fk
nom.fk org.fk aeroport.fr assedic.fr asso.fr avocat.fr avoues.fr barreau.fr cci.fr chambagri.fr
chirurgiens-dentistes.fr com.fr experts-comptables.fr free.fr geometre-expert.fr gouv.fr
greta.fr huissier-justice.fr medecin.fr nom.fr notaires.fr pharmacien.fr port.fr prd.fr
presse.fr tm.fr unblog.fr veterinaire.fr com.ge edu.ge gov.ge mil.ge net.ge org.ge pvt.ge ac.gg
alderney.gg co.gg gov.gg guernsey.gg ind.gg ltd.gg net.gg org.gg sark.gg sch.gg com.gh edu.gh
gov.gh mil.gh org.gh com.gi edu.gi gov.gi ltd.gi mod.gi org.gi ac.gn com.gn edu.gn gov.gn
net.gn org.gn asso.gp com.gp edu.gp mobi.gp net.gp org.gp com.gr edu.gr gov.gr net.gr org.gr
com.gt edu.gt gob.gt ind.gt mil.gt net.gt org.gt com.gu edu.gu gov.gu mil.gu net.gu org.gu
co.gy com.gy net.gy com.hk edu.hk gov.hk idv.hk net.hk org.hk com.hn edu.hn gob.hn mil.hn
net.hn org.hn com.hr from.hr iz.hr name.hr adult.ht art.ht asso.ht com.ht coop.ht edu.ht
firm.ht gouv.ht info.ht med.ht net.ht org.ht perso.ht pol.ht pro.ht rel.ht shop.ht 2000.hu
ac.hu agrar.hu bolt.hu casino.hu city.hu co.hu edu.hu erotica.hu erotika.hu extra.hu film.hu
forum.hu games.hu gov.hu hotel.hu info.hu ingatlan.hu jogasz.hu konyvelo.hu lakas.hu media.hu
news.hu nui.hu org.hu priv.hu reklam.hu sex.hu shop.hu sport.hu suli.hu szex.hu tm.hu tozsde.hu
utazas.hu video.hu ac.id co.id go.id mil.id net.id or.id sch.id web.id gov.ie ac.il co.il
gov.il idf.il k12.il muni.il net.il org.il ac.im co.im eu.im gov.im il.im net.im nic.im org.im
ro.im ac.in co.in edu.in ernet.in firm.in gen.in gov.in ind.in mil.in net.in nic.in org.in
res.in eu.int com.io gov.io mil.io net.io org.io com.iq edu.iq gov.iq mil.iq net.iq org.iq
ac.ir co.ir gov.ir id.ir net.ir org.ir sch.ir com.is edu.is gov.is int.is net.is org.is ag.it
agrigento.it al.it alessandria.it alto-adige.it altoadige.it an.it ancona.it ao.it aosta.it
aoste.it ap.it aq.it aquila.it ar.it arezzo.it ascoli-piceno.it ascolipiceno.it asti.it at.it
av.it avellino.it ba.it balsan.it bari.it barletta-andria-trani.it barlettaandriatrani.it
belluno.it benevento.it bergamo.it bg.it bi.it biella.it bl.it bn.it bo.it bologna.it
bolzano.it bozen.it br.it brescia.it brindisi.it bs.it bz.it ca.it cagliari.it caltanissetta.it
campobasso.it caserta.it catania.it catanzaro.it cb.it ce.it ch.it chieti.it cl.it cn.it co.it
como.it cosenza.it cr.it cremona.it crotone.it cs.it ct.it cuneo.it cz.it edu.it en.it enna.it
fc.it fe.it fermo.it ferrara.it fg.it fi.it firenze.it florence.it foggia.it forli-cesena.it
forlicesena.it fr.it frosinone.it ge.it genoa.it genova.it go.it gorizia.it gov.it gr.it
grosseto.it im.it imperia.it is.it isernia.it kr.it la-spezia.it laquila.it laspezia.it
latina.it lc.it le.it lecce.it lecco.it li.it livorno.it lo.it lodi.it lt.it lu.it lucca.it
macerata.it mantova.it massa-carrara.it massacarrara.it matera.it mc.it me.it messina.it mi.it
milan.it milano.it mn.it mo.it modena.it monza.it ms.it mt.it na.it naples.it napoli.it no.it
novara.it nu.it nuoro.it or.it oristano.it pa.it padova.it padua.it palermo.it parma.it
pavia.it pc.it pd.it pe.it perugia.it pesaro-urbino.it pesarourbino.it pescara.it pg.it pi.it
piacenza.it pisa.it pistoia.it pn.it po.it pordenone.it potenza.it pr.it prato.it pt.it pu.it
pv.it pz.it ra.it ragusa.it ravenna.it rc.it re.it reggio-calabria.it reggio-emilia.it
reggiocalabria.it reggioemilia.it rg.it ri.it rieti.it rimini.it rm.it rn.it ro.it roma.it
rome.it rovigo.it sa.it salerno.it sassari.it savona.it si.it siena.it siracusa.it so.it
sondrio.it sp.it sr.it ss.it suedtirol.it sv.it ta.it taranto.it te.it teramo.it terni.it tn.it
to.it torino.it tp.it tr.it trapani.it trentino.it trento.it treviso.it trieste.it ts.it
turin.it tv.it ud.it udine.it va.it varese.it vb.it vc.it ve.it venezia.it venice.it
verbania.it vercelli.it verona.it vi.it vibo-valentia.it vibovalentia.it vicenza.it viterbo.it
vr.it vt.it vv.it ac.je co.je gov.je ind.je jersey.je ltd.je net.je org.je sch.je com.jm edu.jm
gov.jm net.jm org.jm com.jo edu.jo gov.jo mil.jo name.jo net.jo org.jo sch.jo ac.jp ad.jp
aichi.jp akita.jp aomori.jp chiba.jp co.jp ed.jp ehime.jp fukui.jp fukuoka.jp fukushima.jp
geocities.jp gifu.jp go.jp gov.jp gr.jp gunma.jp hiroshima.jp hokkaido.jp hyogo.jp ibaraki.jp
ishikawa.jp iwate.jp kagawa.jp kagoshima.jp kanagawa.jp kanazawa.jp kawasaki.jp kitakyushu.jp
kobe.jp kochi.jp kumamoto.jp kyoto.jp lg.jp matsuyama.jp mie.jp miyagi.jp miyazaki.jp nagano.jp
nagasaki.jp nagoya.jp nara.jp ne.jp net.jp niigata.jp oita.jp okayama.jp okinawa.jp or.jp
org.jp osaka.jp saga.jp saitama.jp sapporo.jp sendai.jp shiga.jp shimane.jp shizuoka.jp
takamatsu.jp tochigi.jp tokushima.jp tokyo.jp tottori.jp toyama.jp utsunomiya.jp wakayama.jp
yamagata.jp yamaguchi.jp yamanashi.jp yokohama.jp ac.ke co.ke go.ke ne.ke new.ke or.ke sc.ke
co.kg com.kg edu.kg gov.kg mil.kg net.kg org.kg com.kh edu.kh gov.kh mil.kh net.kh org.kh
per.kh biz.ki com.ki de.ki edu.ki gov.ki info.ki net.ki org.ki ass.km asso.km com.km coop.km
edu.km gouv.km gov.km medecin.km mil.km nom.km notaires.km org.km pharmaciens.km prd.km
presse.km tm.km veterinaire.km edu.kn gov.kn net.kn org.kn ac.kr busan.kr chungbuk.kr
chungnam.kr co.kr daegu.kr daejeon.kr es.kr gangwon.kr go.kr gwangju.kr gyeongbuk.kr
gyeonggi.kr gyeongnam.kr hs.kr incheon.kr jeju.kr jeonbuk.kr jeonnam.kr kg.kr kyonggi.kr mil.kr
ms.kr ne.kr or.kr pe.kr re.kr sc.kr seoul.kr ulsan.kr com.kw edu.kw gov.kw mil.kw net.kw org.kw
com.ky edu.ky gov.ky net.ky org.ky com.kz edu.kz gov.kz mil.kz net.kz org.kz c.la com.la edu.la
gov.la info.la int.la net.la org.la per.la com.lb edu.lb gov.lb mil.lb net.lb org.lb co.lc
com.lc edu.lc gov.lc net.lc org.lc com.li gov.li net.li org.li ac.lk assn.lk com.lk edu.lk
gov.lk grp.lk hotel.lk int.lk ltd.lk net.lk ngo.lk org.lk sch.lk soc.lk web.lk com.lr edu.lr
gov.lr net.lr org.lr co.ls org.ls gov.lt mil.lt gov.lu mil.lu net.lu org.lu asn.lv at.lv ch.lv
com.lv conf.lv de.lv edu.lv gov.lv id.lv mil.lv net.lv org.lv biz.ly com.ly edu.ly gov.ly id.ly
med.ly net.ly org.ly plc.ly sch.ly ac.ma asso.ma co.ma gov.ma net.ma org.ma press.ma asso.mc
tm.mc ac.me co.me edu.me gbr.me gov.me its.me net.me org.me priv.me com.mg edu.mg gov.mg mil.mg
nom.mg org.mg prd.mg tm.mg army.mil navy.mil com.mk edu.mk gov.mk inf.mk name.mk net.mk org.mk
com.ml edu.ml gouv.ml gov.ml net.ml org.ml com.mm edu.mm gov.mm net.mm org.mm com.mn edu.mn
gov.mn museum.mn org.mn com.mo edu.mo gov.mo net.mo org.mo music.mobi weather.mobi gov.mr au.ms
br.ms cn.ms de.ms fr.ms hk.ms info.ms net.ms shop.ms us.ms com.mt edu.mt gov.mt net.mt org.mt
tm.mt uu.mt ac.mu co.mu com.mu gov.mu net.mu or.mu org.mu academy.museum agriculture.museum
air.museum airguard.museum alabama.museum alaska.museum amber.museum ambulance.museum
american.museum americana.museum americanantiques.museum americanart.museum amsterdam.museum
and.museum annefrank.museum anthro.museum anthropology.museum antiques.museum aquarium.museum
arboretum.museum archaeological.museum archaeology.museum architecture.museum art.museum
artanddesign.museum artcenter.museum artdeco.museum arteducation.museum artgallery.museum
arts.museum artsandcrafts.museum asmatart.museum assassination.museum assisi.museum
association.museum astronomy.museum atlanta.museum austin.museum australia.museum
automotive.museum aviation.museum axis.museum badajoz.museum baghdad.museum bahn.museum
bale.museum baltimore.museum barcelona.museum baseball.museum basel.museum baths.museum
bauern.museum beauxarts.museum beeldengeluid.museum bellevue.museum bergbau.museum
berkeley.museum berlin.museum bern.museum bible.museum bilbao.museum bill.museum birdart.museum
birthplace.museum bonn.museum boston.museum botanical.museum botanicalgarden.museum
botanicgarden.museum botany.museum brandywinevalley.museum brasil.museum bristol.museum
british.museum britishcolumbia.museum broadcast.museum brunel.museum brussel.museum
brussels.museum bruxelles.museum building.museum burghof.museum bus.museum bushey.museum
cadaques.museum california.museum cambridge.museum can.museum canada.museum capebreton.museum
carrier.museum cartoonart.museum casadelamoneda.museum castle.museum castres.museum
celtic.museum center.museum chattanooga.museum cheltenham.museum chesapeakebay.museum
chicago.museum children.museum childrens.museum childrensgarden.museum chiropractic.museum
chocolate.museum christiansburg.museum cincinnati.museum cinema.museum circus.museum
civilisation.museum civilization.museum civilwar.museum clinton.museum clock.museum coal.museum
coastaldefence.museum cody.museum coldwar.museum collection.museum colonialwilliamsburg.museum
coloradoplateau.museum columbia.museum columbus.museum communication.museum
communications.museum community.museum computer.museum computerhistory.museum
contemporary.museum contemporaryart.museum convent.museum copenhagen.museum corporation.museum
corvette.museum costume.museum countryestate.museum county.museum crafts.museum
cranbrook.museum creation.museum cultural.museum culturalcenter.museum culture.museum
cyber.museum cymru.museum dali.museum dallas.museum database.museum ddr.museum
decorativearts.museum delaware.museum delmenhorst.museum denmark.museum depot.museum
design.museum detroit.museum dinosaur.museum discovery.museum dolls.museum donostia.museum
durham.museum eastafrica.museum eastcoast.museum education.museum educational.museum
egyptian.museum eisenbahn.museum elburg.museum elvendrell.museum embroidery.museum
encyclopedic.museum england.museum entomology.museum environment.museum
environmentalconservation.museum epilepsy.museum essex.museum estate.museum ethnology.museum
exeter.museum exhibition.museum family.museum farm.museum farmequipment.museum farmers.museum
farmstead.museum field.museum figueres.museum filatelia.museum film.museum fineart.museum
finearts.museum finland.museum flanders.museum florida.museum force.museum fortmissoula.museum
fortworth.museum foundation.museum francaise.museum frankfurt.museum franziskaner.museum
freemasonry.museum freiburg.museum fribourg.museum frog.museum fundacio.museum furniture.museum
gallery.museum garden.museum gateway.museum geelvinck.museum gemological.museum geology.museum
georgia.museum giessen.museum glas.museum glass.museum gorge.museum grandrapids.museum
graz.museum guernsey.museum halloffame.museum hamburg.museum handson.museum
harvestcelebration.museum hawaii.museum health.museum heimatunduhren.museum hellas.museum
helsinki.museum hembygdsforbund.museum heritage.museum histoire.museum historical.museum
historicalsociety.museum historichouses.museum historisch.museum historisches.museum
history.museum historyofscience.museum horology.museum house.museum humanities.museum
illustration.museum imageandsound.museum indian.museum indiana.museum indianapolis.museum
indianmarket.museum intelligence.museum interactive.museum iraq.museum iron.museum
isleofman.museum jamison.museum jefferson.museum jerusalem.museum jewelry.museum jewish.museum
jewishart.museum jfk.museum journalism.museum judaica.museum judygarland.museum
juedisches.museum juif.museum karate.museum karikatur.museum kids.museum koebenhavn.museum
koeln.museum kunst.museum kunstsammlung.museum kunstunddesign.museum labor.museum labour.museum
lajolla.museum lancashire.museum landes.museum lans.museum larsson.museum lewismiller.museum
lincoln.museum linz.museum living.museum livinghistory.museum localhistory.museum london.museum
losangeles.museum louvre.museum loyalist.museum lucerne.museum luxembourg.museum luzern.museum
mad.museum madrid.museum mallorca.museum manchester.museum mansion.museum mansions.museum
manx.museum marburg.museum maritime.museum maritimo.museum maryland.museum marylhurst.museum
media.museum medical.museum medizinhistorisches.museum meeres.museum memorial.museum
mesaverde.museum michigan.museum midatlantic.museum military.museum mill.museum miners.museum
mining.museum minnesota.museum missile.museum missoula.museum modern.museum moma.museum
money.museum monmouth.museum monticello.museum montreal.museum moscow.museum motorcycle.museum
muenchen.museum muenster.museum mulhouse.museum muncie.museum museet.museum museumcenter.museum
museumvereniging.museum music.museum national.museum nationalfirearms.museum
nationalheritage.museum nativeamerican.museum naturalhistory.museum naturalhistorymuseum.museum
naturalsciences.museum nature.museum naturhistorisches.museum natuurwetenschappen.museum
naumburg.museum naval.museum nebraska.museum neues.museum newhampshire.museum newjersey.museum
newmexico.museum newport.museum newspaper.museum newyork.museum niepce.museum norfolk.museum
north.museum nrw.museum nuernberg.museum nuremberg.museum nyc.museum nyny.museum
oceanographic.museum oceanographique.museum omaha.museum online.museum ontario.museum
openair.museum oregon.museum oregontrail.museum otago.museum oxford.museum pacific.museum
paderborn.museum palace.museum paleo.museum palmsprings.museum panama.museum paris.museum
pasadena.museum pharmacy.museum philadelphia.museum philadelphiaarea.museum philately.museum
phoenix.museum photography.museum pilots.museum pittsburgh.museum planetarium.museum
plantation.museum plants.museum plaza.museum portal.museum portland.museum portlligat.museum
posts-and-telecommunications.museum preservation.museum presidio.museum press.museum
project.museum public.museum pubol.museum quebec.museum railroad.museum railway.museum
research.museum resistance.museum riodejaneiro.museum rochester.museum rockart.museum
roma.museum russia.museum saintlouis.museum salem.museum salvadordali.museum salzburg.museum
sandiego.museum sanfrancisco.museum santabarbara.museum santacruz.museum santafe.museum
saskatchewan.museum satx.museum savannahga.museum schlesisches.museum schoenbrunn.museum
schokoladen.museum school.museum schweiz.museum science.museum science-fiction.museum
scienceandhistory.museum scienceandindustry.museum sciencecenter.museum sciencecenters.museum
sciencehistory.museum sciences.museum sciencesnaturelles.museum scotland.museum seaport.museum
settlement.museum settlers.museum shell.museum sherbrooke.museum sibenik.museum silk.museum
ski.museum skole.museum society.museum sologne.museum soundandvision.museum
southcarolina.museum southwest.museum space.museum spy.museum square.museum stadt.museum
stalbans.museum starnberg.museum state.museum stateofdelaware.museum station.museum
steam.museum steiermark.museum stjohn.museum stockholm.museum stpetersburg.museum
stuttgart.museum suisse.museum surgeonshall.museum surrey.museum svizzera.museum sweden.museum
sydney.museum tank.museum tcm.museum technology.museum telekommunikation.museum
television.museum texas.museum textile.museum theater.museum time.museum timekeeping.museum
topology.museum torino.museum touch.museum town.museum transport.museum tree.museum
trolley.museum trust.museum trustee.museum uhren.museum ulm.museum undersea.museum
university.museum usa.museum usantiques.museum usarts.museum uscountryestate.museum
usculture.museum usdecorativearts.museum usgarden.museum ushistory.museum ushuaia.museum
uslivinghistory.museum utah.museum uvic.museum valley.museum vantaa.museum versailles.museum
viking.museum village.museum virginia.museum virtual.museum virtuel.museum vlaanderen.museum
volkenkunde.museum wales.museum wallonie.museum war.museum washingtondc.museum
watch-and-clock.museum watchandclock.museum western.museum westfalen.museum whaling.museum
wildlife.museum williamsburg.museum windmill.museum workshop.museum york.museum
yorkshire.museum yosemite.museum youth.museum zoological.museum zoology.museum aero.mv biz.mv
com.mv coop.mv edu.mv gov.mv info.mv int.mv mil.mv museum.mv name.mv net.mv org.mv pro.mv ac.mw
biz.mw co.mw com.mw coop.mw edu.mw gov.mw int.mw museum.mw net.mw org.mw com.mx edu.mx gob.mx
net.mx org.mx com.my edu.my gov.my mil.my name.my net.my org.my co.mz alt.na ca.na cc.na co.na
com.na cul.na dr.na edu.na in.na info.na mobi.na mx.na name.na net.na or.na org.na pro.na
school.na telecom.na tv.na unam.na us.na ws.na com.nc net.nc org.nc cjb.net clickbank.net
de.net gb.net gigazu.net homeip.net ovh.net prserv.net redirectme.net se.net selfip.net t35.net
uk.net whsites.net za.net arts.nf com.nf firm.nf info.nf net.nf other.nf per.nf rec.nf store.nf
web.nf ac.ng com.ng edu.ng gov.ng net.ng org.ng sch.ng com.ni edu.ni gob.ni net.ni nom.ni
org.ni aa.no aarborte.no aejrie.no afjord.no agdenes.no ah.no aknoluokta.no akrehamn.no al.no
alaheadju.no alesund.no algard.no alstahaug.no alta.no alvdal.no amli.no amot.no andasuolo.no
andebu.no andoy.no ardal.no aremark.no arendal.no arna.no aseral.no asker.no askim.no askoy.no
askvoll.no asnes.no audnedaln.no aukra.no aure.no aurland.no aurskog-holand.no austevoll.no
austrheim.no averoy.no badaddja.no bahcavuotna.no bahccavuotna.no baidar.no bajddar.no balat.no
balestrand.no ballangen.no balsfjord.no bamble.no bardu.no barum.no batsfjord.no bearalvahki.no
beardu.no beiarn.no berg.no bergen.no berlevag.no bievat.no bindal.no birkenes.no bjarkoy.no
bjerkreim.no bjugn.no bodo.no bokn.no bomlo.no bremanger.no bronnoy.no bronnoysund.no
brumunddal.no bryne.no bu.no budejju.no bygland.no bykle.no cahcesuolo.no davvenjarga.no
davvesiida.no deatnu.no dep.no dielddanuorri.no divtasvuodna.no divttasvuotna.no donna.no
dovre.no drammen.no drangedal.no drobak.no dyroy.no egersund.no eid.no eidfjord.no eidsberg.no
eidskog.no eidsvoll.no eigersund.no elverum.no enebakk.no engerdal.no etne.no etnedal.no
evenassi.no evenes.no evje-og-hornnes.no farsund.no fauske.no fedje.no fet.no fetsund.no fhs.no
finnoy.no fitjar.no fjaler.no fjell.no fla.no flakstad.no flatanger.no flekkefjord.no
flesberg.no flora.no floro.no fm.no folkebibl.no folldal.no forde.no forsand.no fosnes.no
frana.no fredrikstad.no frei.no frogn.no froland.no frosta.no froya.no fuoisku.no fuossko.no
fusa.no fylkesbibl.no fyresdal.no gaivuotna.no galsa.no gamvik.no gangaviika.no gaular.no
gausdal.no giehtavuoatna.no gildeskal.no giske.no gjemnes.no gjerdrum.no gjerstad.no gjesdal.no
gjovik.no gloppen.no gol.no gran.no grane.no granvin.no gratangen.no grimstad.no grong.no
grue.no gulen.no guovdageaidnu.no ha.no habmer.no hadsel.no hagebostad.no halden.no halsa.no
hamar.no hamaroy.no hammarfeasta.no hammerfest.no hapmir.no haram.no hareid.no harstad.no
hasvik.no hattfjelldal.no haugesund.no hemne.no hemnes.no hemsedal.no herad.no hitra.no
hjartdal.no hjelmeland.no hl.no hm.no hobol.no hof.no hokksund.no hol.no hole.no holmestrand.no
holtalen.no honefoss.no hornindal.no horten.no hoyanger.no hoylandet.no hurdal.no hurum.no
hvaler.no hyllestad.no ibestad.no idrett.no inderoy.no iveland.no ivgu.no jan-mayen.no
jessheim.no jevnaker.no jolster.no jondal.no jorpeland.no kafjord.no karasjohka.no karasjok.no
karlsoy.no karmoy.no kautokeino.no kirkenes.no klabu.no klepp.no kommune.no kongsberg.no
kongsvinger.no kopervik.no kraanghke.no kragero.no kristiansand.no kristiansund.no
krodsherad.no krokstadelva.no kvafjord.no kvalsund.no kvam.no kvanangen.no kvinesdal.no
kvinnherad.no kviteseid.no kvitsoy.no laakesvuemie.no lahppi.no langevag.no lardal.no larvik.no
lavagis.no lavangen.no leangaviika.no lebesby.no leikanger.no leirfjord.no leirvik.no leka.no
leksvik.no lenvik.no lerdal.no lesja.no levanger.no lier.no lierne.no lillehammer.no
lillesand.no lindas.no lindesnes.no loabat.no lodingen.no lom.no loppa.no lorenskog.no loten.no
lund.no lunner.no luroy.no luster.no lyngdal.no lyngen.no malatvuopmi.no malselv.no malvik.no
mandal.no marker.no marnardal.no masfjorden.no masoy.no matta-varjjat.no meland.no meldal.no
melhus.no meloy.no meraker.no midsund.no midtre-gauldal.no mil.no mjondalen.no mo-i-rana.no
moareke.no modalen.no modum.no molde.no mosjoen.no moskenes.no moss.no mosvik.no mr.no
muosat.no museum.no naamesjevuemie.no namdalseid.no namsos.no namsskogan.no nannestad.no
naroy.no narviika.no narvik.no naustdal.no navuotna.no nedre-eiker.no nesna.no nesodden.no
nesoddtangen.no nesseby.no nesset.no nissedal.no nittedal.no nl.no nord-aurdal.no nord-fron.no
nord-odal.no norddal.no nordkapp.no nordre-land.no nordreisa.no nore-og-uvdal.no notodden.no
notteroy.no nt.no odda.no of.no oksnes.no ol.no omasvuotna.no oppdal.no oppegard.no orkanger.no
orkdal.no orland.no orskog.no orsta.no osen.no oslo.no osoyro.no osteroy.no ostre-toten.no
overhalla.no ovre-eiker.no oyer.no oygarden.no oystre-slidre.no porsanger.no porsangu.no
porsgrunn.no priv.no rade.no radoy.no rahkkeravju.no raholt.no raisa.no rakkestad.no
ralingen.no rana.no randaberg.no rauma.no rendalen.no rennebu.no rennesoy.no rindal.no
ringebu.no ringerike.no ringsaker.no risor.no rissa.no rl.no roan.no rodoy.no rollag.no
romsa.no romskog.no roros.no rost.no royken.no royrvik.no ruovat.no rygge.no salangen.no
salat.no saltdal.no samnanger.no sandefjord.no sandnes.no sandnessjoen.no sandoy.no
sarpsborg.no sauda.no sauherad.no sel.no selbu.no selje.no seljord.no sf.no siellak.no
sigdal.no siljan.no sirdal.no skanit.no skanland.no skaun.no skedsmo.no skedsmokorset.no ski.no
skien.no skierva.no skiptvet.no skjak.no skjervoy.no skodje.no slattum.no smola.no snaase.no
snasa.no snillfjord.no snoasa.no sogndal.no sogne.no sokndal.no sola.no solund.no somna.no
sondre-land.no songdalen.no sor-aurdal.no sor-fron.no sor-odal.no sor-varanger.no sorfold.no
sorreisa.no sortland.no sorum.no spjelkavik.no spydeberg.no st.no stange.no stat.no
stathelle.no stavanger.no stavern.no steigen.no steinkjer.no stjordal.no stjordalshalsen.no
stokke.no stor-elvdal.no stord.no stordal.no storfjord.no strand.no stranda.no stryn.no sula.no
suldal.no sund.no sunndal.no surnadal.no svalbard.no sveio.no svelvik.no sykkylven.no tana.no
tananger.no tel.no time.no tingvoll.no tinn.no tjeldsund.no tjome.no tm.no tokke.no tolga.no
tonsberg.no torsken.no tr.no trana.no tranby.no tranoy.no troandin.no trogstad.no tromsa.no
tromso.no trondheim.no trysil.no tvedestrand.no tydal.no tynset.no tysfjord.no tysnes.no
tysvar.no ullensaker.no ullensvang.no ulvik.no unjarga.no utsira.no va.no vaapste.no vadso.no
vaga.no vagan.no vagsoy.no vaksdal.no valle.no vang.no vanylven.no vardo.no varggat.no varoy.no
vefsn.no vega.no vegarshei.no vennesla.no verdal.no verran.no vestby.no vestnes.no
vestre-slidre.no vestre-toten.no vestvagoy.no vevelstad.no vf.no vgs.no vik.no vikna.no
vindafjord.no voagat.no volda.no voss.no vossevangen.no xn--drbak-wua.no xn--leagaviika-52b.no
xn--ostery-fya.no xn--tysvr-vra.no xn--unjrga-rta.no xn--vegrshei-c0a.no com.np edu.np gov.np
mil.np net.np org.np biz.nr com.nr edu.nr fax.nr gov.nr info.nr mob.nr mobil.nr mobile.nr
net.nr org.nr tel.nr tlf.nr mine.nu ac.nz co.nz cri.nz geek.nz gen.nz govt.nz iwi.nz maori.nz
mil.nz net.nz org.nz school.nz ac.om biz.om co.om com.om edu.om gov.om med.om mil.om mod.om
museum.om net.om org.om pro.om sch.om 6a.org ae.org altervista.org dk.org eu.org iblogger.org
za.org abo.pa ac.pa com.pa edu.pa gob.pa ing.pa med.pa net.pa nom.pa org.pa sld.pa com.pe
edu.pe gob.pe mil.pe net.pe nom.pe org.pe com.pf edu.pf org.pf ac.pg com.pg net.pg com.ph
edu.ph gov.ph i.ph mil.ph net.ph ngo.ph org.ph biz.pk com.pk edu.pk fam.pk gob.pk gok.pk gon.pk
gop.pk gos.pk gov.pk info.pk net.pk org.pk web.pk 6bone.pl agro.pl aid.pl art.pl atm.pl
augustow.pl auto.pl babia-gora.pl bedzin.pl beskidy.pl bialowieza.pl bialystok.pl bielawa.pl
bieszczady.pl biz.pl boleslawiec.pl bydgoszcz.pl bytom.pl cieszyn.pl com.pl czeladz.pl czest.pl
dlugoleka.pl edu.pl elblag.pl elk.pl gda.pl gdansk.pl gdynia.pl gliwice.pl glogow.pl gmina.pl
gniezno.pl gorlice.pl gov.pl grajewo.pl gsm.pl home.pl ilawa.pl info.pl interia.pl interii.pl
irc.pl jaworzno.pl jelenia-gora.pl jgora.pl kalisz.pl karpacz.pl kartuzy.pl kaszuby.pl
katowice.pl kazimierz-dolny.pl kepno.pl ketrzyn.pl klodzko.pl kobierzyce.pl kolobrzeg.pl
konin.pl konskowola.pl krakow.pl kutno.pl lapy.pl lebork.pl legnica.pl lezajsk.pl limanowa.pl
lodz.pl lomza.pl lowicz.pl lubin.pl lublin.pl lukow.pl mail.pl malbork.pl malopolska.pl
mazowsze.pl mazury.pl mbone.pl med.pl media.pl miasta.pl mielec.pl mielno.pl mil.pl mragowo.pl
naklo.pl net.pl ngo.pl nieruchomosci.pl nom.pl nowaruda.pl nysa.pl olawa.pl olecko.pl olkusz.pl
olsztyn.pl one.pl opoczno.pl opole.pl org.pl ostroda.pl ostroleka.pl ostrowiec.pl ostrowwlkp.pl
pc.pl pila.pl pisz.pl podhale.pl podlasie.pl polkowice.pl pomorskie.pl pomorze.pl powiat.pl
poznan.pl priv.pl prochowice.pl pruszkow.pl przeworsk.pl pulawy.pl radom.pl rawa-maz.pl
realestate.pl rel.pl republika.pl rybnik.pl rzeszow.pl sanok.pl sejny.pl sex.pl shop.pl
sklep.pl skoczow.pl slask.pl slupsk.pl sopot.pl sos.pl sosnowiec.pl stalowa-wola.pl
starachowice.pl stargard.pl suwalki.pl swidnica.pl swiebodzin.pl swinoujscie.pl szczecin.pl
szczytno.pl szkola.pl targi.pl tarnobrzeg.pl tgory.pl tm.pl torun.pl tourism.pl travel.pl
turek.pl turystyka.pl tychy.pl usenet.pl ustka.pl walbrzych.pl warmia.pl warszawa.pl waw.pl
wegrow.pl wielun.pl wlocl.pl wloclawek.pl wodzislaw.pl wolomin.pl wroc.pl wroclaw.pl xorg.pl
za.pl zachpomor.pl zagan.pl zakopane.pl zarow.pl zgora.pl zgorzelec.pl at.pn au.pn ca.pn ch.pn
cn.pn co.pn de.pn edu.pn es.pn fr.pn gov.pn government.pn it.pn jp.pn net.pn org.pn uk.pn ac.pr
biz.pr com.pr edu.pr est.pr gov.pr info.pr isla.pr name.pr net.pr org.pr pro.pr prof.pr aca.pro
bar.pro cpa.pro eng.pro jur.pro law.pro med.pro com.ps edu.ps gov.ps net.ps org.ps plo.ps
sec.ps com.pt edu.pt gov.pt int.pt net.pt nome.pt org.pt publ.pt sapo.pt belau.pw co.pw ed.pw
go.pw ne.pw or.pw com.py edu.py gov.py net.py org.py com.qa edu.qa gov.qa net.qa org.qa asso.re
com.re nom.re arts.ro com.ro firm.ro go.ro home.ro info.ro nom.ro nt.ro org.ro rec.ro store.ro
tm.ro www.ro ac.rs co.rs edu.rs gov.rs in.rs org.rs ac.ru adygeya.ru altai.ru amur.ru amursk.ru
arkhangelsk.ru astrakhan.ru baikal.ru bashkiria.ru belgorod.ru bir.ru blog.ru bryansk.ru
buryatia.ru by.ru cbg.ru chat.ru chel.ru chelyabinsk.ru chita.ru chukotka.ru chuvashia.ru
cmw.ru com.ru dagestan.ru dudinka.ru e-burg.ru edu.ru fareast.ru front.ru gov.ru grozny.ru
hotbox.ru hotmail.ru hu2.ru hut2.ru id.ru int.ru irkutsk.ru ivanovo.ru izhevsk.ru jamal.ru
jar.ru jino-net.ru joshkar-ola.ru k-uralsk.ru kalmykia.ru kaluga.ru kamchatka.ru karelia.ru
kazan.ru kchr.ru kemerovo.ru khabarovsk.ru khakassia.ru khv.ru kirov.ru kms.ru koenig.ru
komi.ru kostroma.ru krasnoyarsk.ru kuban.ru kurgan.ru kursk.ru kustanai.ru kuzbass.ru land.ru
lipetsk.ru magadan.ru magnitka.ru mail.ru mail2k.ru mari.ru mari-el.ru marine.ru mil.ru
mordovia.ru mosreg.ru msk.ru murmansk.ru mytis.ru nakhodka.ru nalchik.ru narod.ru net.ru
newmail.ru nextmail.ru nightmail.ru nkz.ru nm.ru nnov.ru norilsk.ru nov.ru novosibirsk.ru
nsk.ru omsk.ru orenburg.ru org.ru oryol.ru oskol.ru palana.ru penza.ru perm.ru pochta.ru
pochtamt.ru pop3.ru pp.ru pskov.ru ptz.ru pyatigorsk.ru radikal.ru rbcmail.ru rm.ru rnd.ru
rubtsovsk.ru ryazan.ru sakhalin.ru samara.ru saratov.ru simbirsk.ru smolensk.ru smtp.ru snz.ru
sol.ru spb.ru stavropol.ru stv.ru surgut.ru syzran.ru tambov.ru tatarstan.ru test.ru tom.ru
tomsk.ru tsaritsyn.ru tsk.ru tula.ru tuva.ru tver.ru tyumen.ru ucoz.ru udm.ru udmurtia.ru
ulan-ude.ru vdonsk.ru vladikavkaz.ru vladimir.ru vladivostok.ru volgograd.ru vologda.ru
voronezh.ru vrn.ru vyatka.ru web-soft.ru yakutia.ru yamal.ru yaroslavl.ru yekaterinburg.ru
yuzhno-sakhalinsk.ru z8.ru zgrad.ru zmail.ru ac.rw co.rw com.rw edu.rw gouv.rw gov.rw int.rw
mil.rw net.rw org.rw com.sa edu.sa gov.sa med.sa net.sa org.sa pub.sa sch.sa com.sb edu.sb
gov.sb net.sb org.sb com.sc edu.sc gov.sc net.sc org.sc com.sd edu.sd gov.sd info.sd med.sd
net.sd org.sd sch.sd tv.sd a.se ab.se ac.se b.se bd.se brand.se c.se d.se e.se f.se fh.se
fhsk.se fhv.se g.se h.se i.se k.se komforb.se kommunalforbund.se komvux.se l.se lanarb.se
lanbib.se m.se mil.se n.se naturbruksgymn.se o.se org.se p.se parti.se pp.se press.se r.se s.se
sshn.se t.se tm.se u.se w.se x.se y.se z.se com.sg edu.sg gov.sg idn.sg net.sg org.sg per.sg
co.sh com.sh edu.sh gov.sh mil.sh net.sh nom.sh org.sh edu.sk gov.sk host.sk mil.sk com.sl
edu.sl gov.sl net.sl org.sl co.st com.st consulado.st corp.st edu.st embaixada.st gov.st mil.st
net.st org.st principe.st saotome.st societe.st store.st url.st mail15.su mail333.su pisem.su
com.sv edu.sv gob.sv org.sv red.sv com.sy edu.sy gov.sy mil.sy net.sy org.sy ac.sz co.sz org.sz
at.tc au.tc be.tc bg.tc br.tc ch.tc cz.tc de.tc dk.tc edu.tc es.tc eu.tc fr.tc hk.tc hu.tc
ie.tc int.tc it.tc kr.tc mx.tc net.tc no.tc perso.tc ph.tc pl.tc pro.tc ru.tc se.tc shop.tc
site.tc th.tc ua.tc uk.tc us.tc at.tf bg.tf ca.tf ch.tf cz.tf de.tf edu.tf eu.tf int.tf net.tf
pl.tf ru.tf sg.tf us.tf ac.th co.th go.th in.th mi.th net.th or.th ac.tj biz.tj co.tj com.tj
edu.tj go.tj gov.tj int.tj mil.tj name.tj net.tj org.tj web.tj es.tl gov.tl page.tl biz.tm
co.tm com.tm euro.tm gov.tm mil.tm net.tm org.tm agrinet.tn com.tn defense.tn edunet.tn ens.tn
fin.tn gov.tn ind.tn info.tn intl.tn mincom.tn nat.tn net.tn org.tn perso.tn rnrt.tn rns.tn
rnu.tn tourism.tn turen.tn 163.to com.to edu.to fx.to gov.to kickme.to kwik.to mil.to net.to
org.to t3.to uk.to us.to gov.tp av.tr bbs.tr bel.tr biz.tr com.tr dr.tr edu.tr gen.tr gov.tr
info.tr k12.tr mil.tr name.tr net.tr org.tr pol.tr tel.tr web.tr aero.tt at.tt au.tt be.tt
biz.tt ca.tt co.tt com.tt coop.tt de.tt dk.tt edu.tt es.tt eu.tt fr.tt gov.tt info.tt int.tt
it.tt jobs.tt mobi.tt museum.tt name.tt net.tt nic.tt org.tt pro.tt se.tt travel.tt uk.tt us.tt
co.tv com.tv gov.tv net.tv org.tv club.tw com.tw ebiz.tw edu.tw game.tw gov.tw idv.tw mil.tw
net.tw org.tw ac.tz co.tz go.tz ne.tz or.tz biz.ua cherkassy.ua chernigov.ua chernovtsy.ua
ck.ua cn.ua co.ua com.ua crimea.ua cv.ua dn.ua dnepropetrovsk.ua donetsk.ua dp.ua edu.ua gov.ua
if.ua in.ua ivano-frankivsk.ua kh.ua kharkov.ua kherson.ua khmelnitskiy.ua kiev.ua
kirovograd.ua km.ua kr.ua ks.ua kv.ua lg.ua lugansk.ua lutsk.ua lviv.ua mk.ua net.ua
nikolaev.ua od.ua odessa.ua org.ua pl.ua poltava.ua pp.ua rovno.ua rv.ua sebastopol.ua sumy.ua
te.ua ternopil.ua uz.ua uzhgorod.ua vinnica.ua vn.ua zaporizhzhe.ua zhitomir.ua zp.ua zt.ua
ac.ug co.ug go.ug ne.ug or.ug sc.ug ac.uk bl.uk british-library.uk co.uk edu.uk gov.uk icnet.uk
jet.uk ltd.uk me.uk mil.uk mod.uk national-library-scotland.uk nel.uk net.uk nhs.uk nic.uk
nls.uk org.uk parliament.uk plc.uk police.uk sch.uk ak.us al.us ar.us as.us az.us ca.us co.us
ct.us dc.us de.us dni.us fed.us fl.us ga.us gu.us hi.us ia.us id.us il.us in.us isa.us kids.us
ks.us ky.us la.us ma.us md.us me.us mi.us mn.us mo.us ms.us mt.us nc.us nd.us ne.us nh.us nj.us
nm.us nsn.us nv.us ny.us oh.us ok.us or.us pa.us pr.us ri.us sc.us sd.us tn.us tx.us ut.us
va.us vi.us vt.us wa.us wi.us wv.us wy.us com.uy edu.uy gub.uy mil.uy net.uy org.uy co.uz
com.uz net.uz org.uz vatican.va com.vc edu.vc gov.vc mil.vc net.vc org.vc arts.ve bib.ve co.ve
com.ve e12.ve edu.ve firm.ve gob.ve gov.ve info.ve int.ve mil.ve net.ve nom.ve org.ve rec.ve
store.ve tec.ve web.ve co.vi com.vi edu.vi gov.vi k12.vi net.vi org.vi ac.vn biz.vn com.vn
edu.vn gov.vn health.vn info.vn int.vn name.vn net.vn org.vn pro.vn ch.vu com.vu de.vu edu.vu
fr.vu net.vu org.vu asso.ws com.ws edu.ws gov.ws net.ws org.ws com.ye edu.ye gov.ye mil.ye
net.ye org.ye ac.yu co.yu edu.yu org.yu ac.za alt.za bourse.za city.za co.za edu.za gov.za
law.za mil.za net.za ngo.za nom.za org.za school.za tm.za web.za ac.zm co.zm gov.zm org.zm
sch.zm ac.zw co.zw gov.zw org.zw
);
# 155 entries
our %surbl_thirdlevel_tlds = map { $_ => undef } qw(
gs.aa.no gs.ah.no pref.aichi.jp nes.akershus.no pref.akita.jp pref.aomori.jp web.aplus.net
co.at.lv or.at.lv co.at.pn co.at.tc com.au.ms com.au.pn com.au.tc gs.bu.no nes.buskerud.no
city.chiba.jp pref.chiba.jp lkd.co.im ltd.co.im plc.co.im act.edu.au nsw.edu.au nt.edu.au
qld.edu.au sa.edu.au tas.edu.au vic.edu.au wa.edu.au pref.ehime.jp web.fc2.com gs.fm.no
blog.friendster.com pref.fukui.jp city.fukuoka.jp pref.fukuoka.jp pref.fukushima.jp
pref.gifu.jp docs.google.com act.gov.au nsw.gov.au nt.gov.au qld.gov.au sa.gov.au tas.gov.au
vic.gov.au wa.gov.au pa.gov.pl po.gov.pl so.gov.pl sr.gov.pl starostwo.gov.pl ug.gov.pl
um.gov.pl upow.gov.pl uw.gov.pl pref.gunma.jp os.hedmark.no valer.hedmark.no city.hiroshima.jp
pref.hiroshima.jp gs.hl.no gs.hm.no pref.hokkaido.jp os.hordaland.no pref.hyogo.jp
pref.ibaraki.jp pref.ishikawa.jp pref.iwate.jp gs.jan-mayen.no pref.kagawa.jp pref.kagoshima.jp
pref.kanagawa.jp city.kawasaki.jp city.kitakyushu.jp city.kobe.jp pref.kochi.jp
pref.kumamoto.jp city.kyoto.jp pref.kyoto.jp groups.live.com profile.live.com skydrive.live.com
spaces.live.com bay.livefilestore.com blu.livefilestore.com pref.mie.jp pref.miyagi.jp
pref.miyazaki.jp heroy.more-og-romsdal.no sande.more-og-romsdal.no gs.mr.no spaces.msn.com
pref.nagano.jp pref.nagasaki.jp city.nagoya.jp pref.nara.jp sakura.ne.jp city.niigata.jp
pref.niigata.jp gs.nl.no bo.nordland.no heroy.nordland.no gs.nt.no gs.of.no web.officelive.com
pref.oita.jp city.okayama.jp pref.okayama.jp pref.okinawa.jp gs.ol.no city.osaka.jp
pref.osaka.jp gs.oslo.no valer.ostfold.no cloud.prohosting.com gs.rl.no pref.saga.jp
city.saitama.jp pref.saitama.jp com.sapo.pt do.sapo.pt home.sapo.pt no.sapo.pt paginas.sapo.pt
city.sapporo.jp city.sendai.jp gs.sf.no pref.shiga.jp pref.shimane.jp city.shizuoka.jp
pref.shizuoka.jp gs.st.no gs.svalbard.no homepage.t-online.de bo.telemark.no gs.tm.no
pref.tochigi.jp pref.tokushima.jp metro.tokyo.jp pref.tottori.jp pref.toyama.jp gs.tr.no
co.uk.pn co.uk.tc me.uk.tc org.uk.tc gs.va.no sande.vestfold.no gs.vf.no pref.wakayama.jp
pref.yamagata.jp pref.yamaguchi.jp pref.yamanashi.jp city.yokohama.jp
);
uribl
(text/plain, 31.3 KB)
#!/usr/bin/perl (Editor Hint)
=head1 NAME
uribl - URIBL blocking plugin for qpsmtpd
=head1 DESCRIPTION
This plugin implements DNSBL lookups for URIs found in spam, such as that
implemented by SURBL (see E<lt>http://surbl.org/E<gt>). Incoming messages are
scanned for URIs, which are then checked against one or more URIBLs in a
fashion similar to DNSBL systems.
=head1 CONFIGURATION
To enable the plugin, add it to I<~qpsmtpd/config/plugins>. The list of
URIBLs to check should be placed in I<uribl_zones> in the config directory
(typically I<~qpsmtpd/config>). If no zones appear in this file, a default
set of services is checked.
The format of the I<uribl_zones> file is a list of URIBL DNS zones, one per
line, consisting of one or more columns separated by whitespace. The first
column (the only mandatoy one) should consist of the URIBL zone.
The second column may optionally contain a comma-delimited list of integers selecting
mask values to be applied to the A record(s) returned from a URIBL. This
enables the use of composite DNSBLs, such as multi.surbl.org, where several
lists are combined so they may be accessed with a single query; any returns
are checked against the mask of lists you're interested in. If unspecified,
or if a negative number is given, all lists in a composite URIBL will be
checked. URIBL operators prefer that you use the composite lists to reduce
their own query load, and it's more efficient for qpsmtpd as well.
The next two columns may optionally contain a configuration directive and
corresponding value as described below, to be applied either to the entire
zone, or to the masks specified in the second column.
The third column specifies an action, which overrides the default action
configured with the I<action> setting discussed below.
Any masks and parameters not listed will inherit default parameters recommended
by the zone in question, if known.
For example:
=over 4
dbl.spamhaus.org
multi.uribl.com action denysoft
multi.uribl.com 2,4 action deny
multi.surbl.org
multi.surbl.org 2,8 action deny
multi.surbl.org 4,16
=back
You may specify the following config option(s) in the I<uribl_zones> file:
=over 4
=item action
Overrides the default action configured with the I<action> setting discussed
below. This can be set either for the entire zone or for a specific set of
masks.
=item check_ip
Submit IP addresses found in URIs to the URIBL service. Some services
discourage sending IP addresses and some warn that they will result in false
positives. SpamHaus DBL, for instance, strongly discourages it and will in fact
always give a positive result when sent an IP address. Defaults to enabled
unless service-specific defaults exists.
=item check_a_record
Look up A records on hostnames found in URIs and check the results against the
zone in question. SBL-XBL is one service that specifically recommends doing
this; others discourage it and may generate false positives if you try it.
Defaults to disabled unless service-specific defaults exist.
=item check_ns_record
Look up NS records on hostnames and IPs found in URIs and check the results
against the zone in question. Like check_a_record, this is recommended by
SBL-XBL but could generate false positives on other services. Defaults to
disabled unless service-specific defaults exist.
=item check_host
Submit hostnames found in URIs to the URIBL service. Some services, especially
'RBL' services, only list IP addresses. SBL-XBL is an example of this; SpamHaus
recommends checking IP addresses from URIs to this list. Defaults to enabled
unless service-specific defaults exist.
=item host_wildcards_supported
The zone supports wildcards -- e.g. checking a.b.c.com will generate a hit if
a.b.c.com, b.c.com, or c.com is listed. Spamhaus DBL supports wildcard; most
services do not. Defaults to disabled unless service-specific defaults exist.
=item txt_records_supported
The zone supports TXT lookups. Turning this off will increase efficiency, but
prevent extra information that could be useful for troubleshooting (especially
explanation URLs) provided by the zone.
=back
You may specify the following config option(s) in the I<qpsmtpd/config> file:
=over 4
=item action
Specifies what to do when a URI is matched in a URIBL. Available options are
I<add-header> (the default) I<deny> and I<denysoft>. If set to add-header, an
X-URIBL-Match: header will be added explaining the URIBL entry found. If set
to 'deny,' the delivery will be declined with a hard failure. If set to
denysoft, the delivery will be soft failed (this is probably not a good idea.)
=item timeout
Timeout for DNS requests, in seconds. The default is 30 seconds. DNS
requests are issued asynchronously and in parallel for all hosts found
in URIs in the mail; the same timeout will apply to each; see the
Net::DNS documentation for details.
=item check-headers
If set 'all', any URIs found in any headers will be checked as well. If set to
one or more header names separated by commas, any URIs found in the specified
headers will be checked. Disabled by default.
=back
=head1 CAUTIONS
When used in I<deny> or I<denysoft> mode, a URIBL check can block not
only the original spam containing a listed URI, but mail unintentionally
carrying that URI, such as forwarded complaints. The uribl checks should
only be used in these modes if you know what you're doing.
The URI scanner used by the uribl plugin is quite aggressive, and attempts to
detect all forms of URIs supported by typical MUAs (even those that lack a
protocol specification, for example.) However, it does not attempt to detect
URIs that have been mangled beyond programmatic reconstruction. Even so, it
may issue spurious lookups on unintended URIs.
=head1 COPYRIGHT
uribl is copyright 2004-2007 by Devin Carraway E<lt>[email protected]<gt>. It
may be used and redistributed under the same terms as qpsmtpd itself.
=cut
use strict;
use warnings;
use List::Util qw( min max );
use IO::Select;
use IO::Socket::INET;
use Net::DNS::Async;
use Time::HiRes qw(time);
# use Data::Dumper;
require '/home/smtpd/qpsmtpd/plugins/lib/tld_lists.pl';
$Data::Dumper::Terse = 1; # don't output names where feasible
$Data::Dumper::Indent = 0;
our ( %firstlevel_tlds, %uribl_secondlevel_tlds, %uribl_thirdlevel_tlds,
%surbl_secondlevel_tlds, %surbl_thirdlevel_tlds );
# These won't work unless we go to a lot of trouble to make them work; let's
# log when we encounter them so we can find out whether it would be worthwhile
my %broken_shorteners = map { $_ => undef }
qw( ad.ag cli.gs shorl.com mobiletinyurl.com
urlshorteningservicefortwitter.com );
my %shorteners = ( %broken_shorteners, map { $_ => undef } qw(
bit.ly doiop.com go.qb.by goo.gl ho.io idek.net is.gd migre.me moourl.com
myturl.com ow.ly pnt.me qurl.com redire.ru retwt.me short.to sn.im snipr.com
snipurl.com su.pr tiny.cc tinyurl.com to.ly tr.im u.nu url.ie vb.ly
) );
my $max_host_levels = 4; # Never send more levels than a.b.c.d
my %zone_defaults = (
action => 'add-header',
check_ip => 1,
check_a_record => 0,
check_ns_record => 0,
check_host => 1,
host_wildcards_supported => 0,
txt_records_supported => 1,
);
my $zones = {
'dbl.spamhaus.org' => {
label => 'Spamhaus DBL',
# action => 'log',
check_ip => 0,
host_wildcards_supported => 1,
},
'sbl-xbl.spamhaus.org' => {
label => 'Spamhaus SBL/XBL',
check_a_record => 1,
check_ns_record => 1,
check_host => 0,
},
'multi.surbl.org' => {
label => 'SURBL',
masks => { 2 => { label => 'SpamCop' }, # , action => 'log' },
4 => { label => 'sa-blacklist' },
8 => { label => 'Phishing and Malware' },
16 => { label => 'Outblaze' },
32 => { label => 'AbuseButler' },
64 => { label => 'jwSpamSpy' } },
host_levels => { 2 => \%surbl_secondlevel_tlds,
3 => \%surbl_thirdlevel_tlds },
},
'multi.uribl.com' => {
label => 'URIBL',
masks => { 2 => { label => 'Black' },
4 => { label => 'Grey' },
8 => { label => 'Red' } },
host_levels => { 2 => \%uribl_secondlevel_tlds,
3 => \%uribl_thirdlevel_tlds },
},
};
# Derive individual list defaults from their 'multi' counterparts
my $surbl = $zones->{'multi.surbl.org'};
for ( [ 2 => 'sc' ], [ 4 => 'ws' ], [ 8 => 'ph' ],
[ 16 => 'ob' ], [ 32 => 'ab' ], [ 64 => 'jp' ] ) {
my $z = $zones->{ "$_->[1].surbl.org" } = { %$surbl };
my $descr = $z->{masks}->{ $_->[0] }->{label};
$z->{label} = "$z->{label} ($descr)";
delete $z->{masks};
}
#print "Dump #2";
#print Dumper($zones);
my $uribl = $zones->{'multi.uribl.com'};
for ( [ 2 => 'black' ], [ 4 => 'grey' ], [ 8 => 'red' ] ) {
my $z = $zones->{ "$_->[1].uribl.com" } = { %$uribl };
my $descr = $z->{masks}->{ $_->[0] }->{label};
$z->{label} = "$z->{label} ($descr)";
delete $z->{masks};
}
sub init { # used by async plugin
my ( $self, $qp, %args ) = @_;
$self->{timeout} = $args{timeout} || 30;
$self->{whitelist_zones} = { %{ $self->{whitelist_zones} || {} },
map { $_ => undef }
# W3C shows up in a bunch of doctype declarations
'w3c.org', 'w3.org',
$qp->config('uribl_whitelist_domains') };
# Includes support for legacy arg 'scan-headers'
( $self->{check_headers} ) = grep { defined } ( @args{qw( check-headers scan-headers )}, 0 );
# Legacy support for boolean check_headers value
$self->{check_headers} = 'all' if $self->{check_headers} == 1;
# Allow inheriting plugins to define extra zone defaults
$zones = { %$zones, %{ $self->{extra_zones} } } if $self->{extra_zones};
my @zones = $self->qp->config('uribl_zones');
@zones = @{ $self->{default_zones} || [qw( dbl.spamhaus.org sbl-xbl.spamhaus.org
multi.surbl.org multi.uribl.com )] }
unless @zones;
my $localzones;
for ( @zones ) {
my ( $zone, @arg ) = split /\s+/;
my $z = $localzones->{$zone} ||= {};
next unless @arg;
my @c = qw( action check_ip check_a_record check_ns_record
check_host host_wildcards_supported );
if ( grep { $arg[0] eq $_ } @c ) {
# Config directive for entire zone
my ( $key, $value ) = @arg;
die "Could not parse config line: $_" unless defined $value;
$z->{$key} = $value if defined $value;
next;
} elsif ( $arg[0] !~ /[\d,]/ ) {
die "Could not parse config line: $_";
}
# mask definitions / settings
my ( $masks, $key, $value ) = @arg;
$z->{masks}->{$_} ||= {} for split /,/, $masks;
next unless defined $key; # Mask definition with no additional prefs
if ( grep { $key eq $_ } qw( log add-header deny denysoft ) ) { # legacy definition
( $key, $value ) = ( 'action', $key );
$z->{action} ||= $value;
}
die "Could not parse config line: $_" if ! defined $value or ! grep { $key eq $_ } @c;
for my $m ( split /,/, $masks ) {
$z->{masks}->{$m}->{$key} = $value;
next if ! exists $zones->{$zone}
or ! exists $zones->{$zone}->{masks}
or ! exists $zones->{$zone}->{masks}->{$m};
$z->{masks}->{$m}->{label} ||= $zones->{$zone}->{masks}->{$m}->{label} || $m;
$z->{masks}->{$m}->{action} ||= $zones->{$zone}->{masks}->{$m}->{action};
}
}
for my $zone ( keys %$localzones ) {
$self->{zones}->{$zone} = {
# Defaults
label => $zone,
%zone_defaults,
( $args{action} ? ( action => $args{action} ) : () ),
# Defaults specific to URIBL services we know about
%{ $zones->{$zone} || {} },
# Config file settings
%{ $localzones->{$zone} },
};
}
$self->init_masks;
}
sub register { shift->register_hook( data_post => 'data_handler' ) }
sub init_masks {
for my $zone ( values %{ shift->{zones} } ) {
my $mask = 0;
for ( keys %{ $zone->{masks} || {} } ) {
$mask |= $_ >= 0 ? $_ : 0x00ffffff;
}
$zone->{mask} = $mask || 0x00ffffff;
}
}
sub send_queries { # used async plugin
# Determine which queries to send to which zones and ship them off to the appropriate sub
my ( $self, $addr, $arg ) = @_;
return unless $addr;
if ( $arg->{forward_ip} ) {
return if exists $self->{whitelist_zones}->{$addr};
} else { # hostname
return if $self->host_whitelisted( $addr );
}
# Allow the async version to use ParaDNS rather than Net::DNS::Async
my $sendq = $self->{async_version} ? 'queue_pdns_query' : 'send_query';
for my $z ( $arg->{zone} or keys %{ $self->{zones} } ) {
my $q = { zone => $z, name => $arg->{forward_ip} || $addr };
$self->{start_time} ||= time();
if ( $self->{zones}->{$z}->{check_ns_record} and ! $arg->{from_cb} ) {
$self->$sendq( $q, 'ns_record', $addr, 'NS' );
}
my $hostname;
if ( ! $arg->{forward_ip} ) { # hostname
$self->$sendq( $q, 'a_record', $addr, 'A' )
if $self->{zones}->{$z}->{check_a_record};
next unless $self->{zones}->{$z}->{check_host};
$hostname = $self->truncate_host( $addr, $z );
}
next if $arg->{forward_ip} and ! $self->{zones}->{$z}->{check_ip};
my $querystring = $hostname ? "$hostname.$z" : "$addr.$z";
$self->$sendq( $q, 'uri', $querystring, 'A' );
$self->$sendq( $q, 'uri', $querystring, 'TXT' )
if $self->{zones}->{$z}->{txt_records_supported};
}
}
sub host_whitelisted {
my ( $self, $hostname ) = @_;
my @h = split /\./, $hostname;
for ( 2 .. $#h + 1 ) {
my $subhost = join( '.', @h[ -$_ .. -1 ] );
next unless exists $self->{whitelist_zones}->{$subhost};
$self->log(LOGINFO, "uribl: Skipping whitelist URI domain '$hostname' ('$subhost' whitelisted)");
return 1;
}
return 0;
}
sub truncate_host {
my ( $self, $hostname, $zone ) = @_;
# truncate to max level
my @h = reverse split /\./, $hostname;
@h = reverse @h > $max_host_levels ? @h[ 0 .. $max_host_levels - 1 ] : @h;
# send the max allowed levels if wildcards are supported
return join '.',@h if $self->{zones}->{$zone}->{host_wildcards_supported};
my $host_levels = $self->{zones}->{$zone}->{host_levels} || {};
return join '.',@h[-4..-1] if @h > 3 and exists $host_levels->{3}->{ join '.', @h[-3..-1] };
return join '.',@h[-3..-1] if @h > 2 and exists $host_levels->{2}->{ join '.', @h[-2..-1] };
return join '.',@h[-2..-1];
}
sub send_query { # NOT used by async plugin
my ( $self, $q, $cbtype, $querystring, $type ) = @_;
$cbtype = $cbtype . '_cb';
my $cb = sub { $self->$cbtype( $q, @_ ) };
$self->{dns} ||= Net::DNS::Async->new( Timeout => $self->{timeout} ) or return;
my $key = "$q->{zone}|$querystring|$type";
return if exists $self->{queries}->{$key};
$self->{queries}->{$key} = undef;
$self->log(LOGDEBUG,"uribl: Looking up $type record for '$querystring'");
eval { $self->{dns}->add( $cb, $querystring, $type );
$self->{started_queries}++ };
return 1 unless $@;
$self->log(LOGERROR, "uribl: Couldn't open socket for $type record '$querystring': $@");
return 0;
}
sub uri_cb {
my ( $self, $q, $packet ) = @_;
return unless $packet;
$self->{queries_completed}->{uri}++;
my $z = $q->{zone};
for my $answer ( grep { $_->type eq 'A' } $packet->answer ) {
my $address = $answer->address or next;
my @lists = $self->evaluate( $z, $address ) or next;
$self->log(LOGDEBUG, "uribl: match in $z");
$q->{lists} = \@lists;
$q->{a} ||= $address;
push @{ $self->{matches} }, $q;
return;
}
$q->{txt} ||= $_->txtdata and return for grep { $_->type eq 'TXT' } $packet->answer;
}
sub a_record_cb {
my ( $self, $q, $packet ) = @_;
return unless $packet;
$self->{queries_completed}->{a}++;
for my $answer ( grep { $_->type eq 'A' } $packet->answer ) {
my @fwd = split /\./, $answer->address or next;
my $fwd = join '.', @fwd;
my $rev = join '.', reverse @fwd;
$self->log(LOGDEBUG,"uribl: looking up $fwd (A record of $q->{name})");
$self->send_queries( $rev, { forward_ip => $fwd, zone => $q->{zone}, from_cb => 1 } );
}
}
sub ns_record_cb {
my ( $self, $q, $packet ) = @_;
return unless $packet;
$self->{queries_completed}->{ns}++;
for my $answer ( grep { $_->type eq 'NS' } $packet->answer ) {
my $address = $answer->address or next;
if ( $address =~ /[^\d.]/ ) { # hostname
$self->log(LOGDEBUG,"uribl: looking up $address (NS record of $q->{name})");
$self->send_queries( $address, { zone => $q->{zone}, from_cb => 1 } );
} else { # IP address
my @fwd = split /\./, $address or next;
my $fwd = join '.', @fwd;
my $rev = join '.', reverse @fwd;
$self->log(LOGDEBUG,"uribl: looking up $fwd (NS record of $q->{name})");
$self->send_queries( $rev, { forward_ip => $fwd, zone => $q->{zone}, from_cb => 1 } );
}
}
}
sub cleanup { # used by async plugin
my ( $self ) = @_;
delete @$self{qw( dns matched start_time queries_completed queries shortened
started_queries a_lookups ns_lookups pdns_lookups )};
}
sub evaluate { # used by async plugin
my ( $self, $zone, $a ) = @_;
return unless $zone and $a;
my $z = $self->{zones}->{$zone};
my $mask = $z->{mask} or return;
$a =~ /^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/ or return;
my $v = (($1 & 0xff) << 24) | (($2 & 0xff) << 16) | (($3 & 0xff) << 8) | ($4 & 0xff);
my $match = $v & $mask or return;
my $list = { label => $z->{label}, action => $z->{action} };
my @lists;
for ( keys %{ $z->{masks} || {} } ) {
next unless $match & $_;
my $extra = $z->{masks}->{$_}->{label};
push @lists, { label => "$list->{label}",
action => $z->{masks}->{$_}->{action} || $z->{action} };
}
@lists = ( $list ) unless @lists;
return @lists;
}
sub scan_mime_body {
my ( $self, $part ) = @_;
$self->log(LOGDEBUG, "uribl: Scanning Mime Body");
if ( $part->parts ) {
$self->find_uris( $_ ) for $part->parts;
return;
}
return unless $part->effective_type =~ [qw( text/plain text/html )];
$self->log(LOGDEBUG, "uribl: Scanning Mime Body - text-html");
my $fh = $part->bodyhandle->open('r');
$self->find_uris($_) while ( <$fh> );
}
sub scan_body {
my ( $self, $txn ) = @_;
$self->log(LOGDEBUG, "uribl: Scanning Plain Text Body");
$txn->body_resetpos;
my $line;
my @qp_continuations;
while ( $line = $txn->body_getline ) {
chomp $line;
if ( $line =~ /(.*)=$/ ) {
push @qp_continuations, $1;
next;
} elsif ( @qp_continuations ) {
$line = join('', @qp_continuations, $1);
@qp_continuations = ();
}
$self->find_uris($line);
}
if ( @qp_continuations ) {
$self->log(LOGINFO, "uribl: WARNING: scan_body exiting with line continuations left. Bad Email?");
$line = join('', @qp_continuations, $line);
@qp_continuations = ();
$self->find_uris($line);
}
}
my $tld = join '|', keys %firstlevel_tlds;
$tld =~ s/([a-zA-Z])/[\l$1\u$1]/g;
$tld = "(?:$tld)";
my $minchars = min( map { length } keys %firstlevel_tlds );
my $maxchars = max( map { length } keys %firstlevel_tlds ) * 3;
my $tld_chars = qr/[a-zA-Z0-9%-]{$minchars,$maxchars}/;
my $encoded_dot = qr/[=%]2[eE]/;
my $encoded_octet = qr/(?: (?: (?: (?<= \.) | (?<= $encoded_dot ) ) (?: (?:%3)?[0-9] )+ )
| (?:0[xX][[:xdigit:]]+)
| (?: (?: (?:%3)?[0-9] ){7,} ) )/x;
my $octet = qr/(?: (?: (?<= \.) [0-9]+ )
| (?:0[xX][[:xdigit:]]+)
| [0-9]{7,} )/x;
sub find_uris { # used by async plugin
my $self = shift;
$_ = shift;
# Early-exit if there's no possible URI content on the entire line
return unless m{[a-zA-Z0-9](?:\.|$encoded_dot|://)[a-zA-Z0-9%]}o;
chomp;
while ( m{ ( (?: [a-zA-Z0-9:./_@?%&#+-]+ )?
(?: [a-zA-Z0-9.=%&\#;_-]+ (?:\.|$encoded_dot) $tld_chars # Possible Hostname
| [xX[:xdigit:].=%&\#;-]* $encoded_octet ) # Possible IP
(?: [a-zA-Z0-9:./_@=?%&#+-]+ )? )
(?! \.?\w ) }gxo ) {
my $addr = $1;
my ( $path, $ip_obfu );
if ( $addr =~ s/.*?[hH][tT][tT][pP][sS]?:\/\///o ) {
$ip_obfu = 1;
# 'http://realsite.com/cgi/[email protected]'
# 'http://user:[email protected]'
$path = $1 if $addr =~ s/[\/\?](.*)//o;
$addr =~ s/.*(?:@|[=%]40)//o;
} elsif ( $addr =~ /(.*):./ ) {
# 'user:[email protected]'
# 'mailto:[email protected]'
next if $1 =~ /[mM][aA][iI][lL][tT][oO]/o;
$addr =~ s/.*(?:@|[=%]40)//o;
$path = $1 if $addr =~ s/[\/\?](.*)//o;
} else {
# 'realsite.com/cgi/[email protected]'
$path = $1 if $addr =~ s/[\/\?](.*)//o;
next if $addr =~ /@|[=%]40/o;
}
next unless $addr;
$addr = unmunge( $addr );
my $rev;
if ( my $host = hostname( $addr ) ) {
$addr = $host;
} else {
( $addr, $rev ) = ip( $addr, $ip_obfu );
}
next unless $addr;
# if the host is a shortener, we never check the lists against the host
# itself, but we'll try to resolve URL it points to
if ( exists $shorteners{$addr} ) {
if ( $path and ! exists $self->{shortened}->{"$addr|$path"} ) {
$self->log(LOGDEBUG, "uribl: Found shortened URI $addr/$path");
$self->{shortened}->{"$addr|$path"} = { host => $addr, path => $path };
}
} elsif ( ! exists $self->{pending}->{$addr} ) {
$self->{pending}->{$addr} = undef;
if ( $rev ) {
$self->log(LOGDEBUG, "uribl: matched IP address $addr");
$self->send_queries( $rev, { forward_ip => $addr } );
} else {
$self->log(LOGDEBUG, "uribl: matched hostname $addr");
$self->send_queries( $addr );
}
}
next unless $path;
for my $p ( split /\//, $path ) {
$p = unmunge($p);
while ( $p =~ s/ ( @?
(?:
(?: [a-zA-Z0-9._-]+ \. (?:$tld) # Possible Hostname
(?![a-zA-Z0-9._-] ) )
| (?: [xX[:xdigit:].-]* $octet ) # Possible IP
)
)
//xo ) {
# parameterized redirect
$addr = $1;
next if $addr =~ /@/;
if ( my $host = hostname( $addr ) ) {
$addr = $host;
# dash and underscore are allowed anywhere in subdomains, but the dash seems to
# also be commonly used as a delimiter in parameterized redirects
my $extra_host = hostname( $host ) if $host =~ s/^-.+-//;
} else {
( $addr, $rev ) = ip( $addr );
}
next unless $addr;
my ($path) = $p =~ /^\/(.+)([hH][tT][tT][pP])?/;
my $extra_host;
for ( $addr, $extra_host ? $extra_host : () ) {
if ( exists $shorteners{$addr} ) {
if ( $path and ! exists $self->{shortened}->{"$_|$path"} ) {
$self->log(LOGDEBUG, "uribl: Found shortend URI $_/$path");
$self->{shortened}->{"$_|$path"} = { host => $_, path => $path };
}
} elsif ( ! exists $self->{pending}->{$_} ) {
$self->{pending}->{$_} = undef;
if ( $rev ) {
$self->log(LOGDEBUG, "uribl: matched IP address $_");
$self->send_queries( $rev, { forward_ip => $_ } );
} else {
$self->log(LOGDEBUG, "uribl: matched hostname $_");
$self->send_queries( $_ );
}
}
}
}
}
}
}
sub unmunge {
my ( $addr ) = @_;
# URI escape munging
$addr =~ s/[=%]([[:xdigit:]]{2})/chr(hex($1))/geo;
# HTML entity munging (e.g. in parameterized redirects)
$addr =~ s/&#(\d{2,3});?/chr($1)/geo;
# more entity munging
$addr =~ s/&#[xX]0*([[:xdigit:]]{2});?/chr(hex($1))/geo;
return $addr;
}
sub hostname {
$_[0] =~ /^ (?: .*[^0-9a-zA-Z._-]\.? )?
(?: .*\.\. )?
(?: [wW]{3}\. )? # strip www
(?: ( [a-zA-Z0-9._-]+ ) \. )? # all subdomains
( [a-zA-Z0-9][a-zA-Z0-9-]* # domain name - cannot begin with a dash
(?: (?<=[a-zA-Z0-9]) | [a-zA-Z0-9] ) ) # ...or end with a dash
\.($tld)
(?:[^0-9a-zA-Z_].*)?
$/x or return;
return defined $1 ? lc "$1.$2.$3" : lc "$2.$3";
}
sub ip {
my ( $addr, $ip_obfu ) = @_;
if ( $ip_obfu ) {
$addr =~ s/(^|\.)(0*[xX][[:xdigit:]]{1,8})/$1 . hex($2)/geo;
$addr =~ s/(^|\.)(0[0-7]+)/$1 . oct($2)/geo;
}
$addr =~ s/^(?:[^0-9]|\.)*\.//;
$addr =~ s/[^0-9.].*//;
my @ip = split /\./, $addr or return;
return if @ip > 4;
if ( $ip_obfu ) {
my $last = $ip[-1];
$ip[$_] = ( $last >> ( 24, 16, 8, 0 )[$_] ) & 0xff for $#ip .. 3;
}
return if @ip != 4;
return if grep { ! /^[0-9]+$/ } @ip;
return if grep { $_ > 255 } @ip;
return if grep { $ip[0] == $_ } ( 0, 255, 127, 10 );
return if $ip[0] == 172 and ( $ip[1] & 0xf0 ) == 16; # 172.16.0.0/12
return if $ip[0] == 192 and $ip[1] == 168;
return if $ip[0] == 169 and $ip[1] == 254; # link-local
return if ( $ip[0] & 0xf0 ) == 224; # 224.0.0.0/4
map { s/^0+([0-9])/$1/ } @ip;
return join('.', @ip), join('.', reverse @ip);
}
sub scan_message { # used by async plugin
my ( $self, $txn ) = @_;
map { $self->find_uris( $txn->header->get($_) ) }
$self->{check_headers} = 'all' ? $txn->header->tags
: split /,/, $self->{check_headers}
if $self->{check_headers};
$txn->notes('mime_body') ? $self->scan_mime_body( $txn->notes('mime_body') )
: $self->scan_body( $txn );
if ( ! $self->{started_queries} and ! $self->{shortened} ) {
$self->log(LOGINFO, "uribl: No URIs found in mail");
return 0;
}
return 1;
}
sub data_handler { # NOT used by async plugin
my ( $self, $txn ) = @_;
$self->cleanup();
# Check if we should run or not
my $options = $txn->notes('plug_options');
return DECLINED if $self->qp->connection->notes('whitelisthost');
return DECLINED if $txn->notes('whitelistsender');
return DECLINED unless $self->scan_message( $txn );
$self->{dns}->await() if $self->{dns};
my @r = $self->check_results;
# Resolving shortned URIs is more expensive than regular queries;
# don't bother if we already found a reason to reject
if ( $r[0] == DECLINED and $self->resolve_shortened_uris ) {
$self->{dns}->await() if $self->{dns};
@r = $self->check_results;
}
# $self->log(LOGINFO, sprintf( "lookups: %2d zone, %2d A, %2d NS, %2d shortener"
# . " matches: %2d time: %.6f",
# ( map { $self->{queries_completed}->{$_} }
# qw( uri a ns shortener ) ),
# scalar @{ $self->{matches} || [] },
# time() - $self->{start_time} ));
return @r;
}
sub resolve_shortened_uris {
my ( $self ) = @_;
my $lookup = $self->{shortened};
for ( grep { exists $broken_shorteners{$_} }
map { $lookup->{$_}->{host} }
keys %$lookup ) {
$self->log(LOGNOTICE, "uribl: Can't look up unsupported URI shortener $lookup->{$_}->{host}");
delete $lookup->{$_};
}
return unless %$lookup;
my $sockets;
my $send = IO::Select->new;
for ( keys %$lookup ) {
my $L = $lookup->{$_};
my $sock = IO::Socket::INET->new( PeerAddr => $L->{host}, PeerPort => 80, Blocking => 0 );
$sockets->{$sock} = $L;
$send->add($sock);
delete $lookup->{$_};
}
my $sent_queries = 0;
my $started = time();
my $receive = IO::Select->new;
while ( my @send = $send->can_write( $self->{timeout} - ( time() - $started ) ) ) {
for my $sock ( @send ) {
my $L = $sockets->{$sock};
print $sock "HEAD /$L->{path} HTTP/1.1\r\n"
. "Host: $L->{host}\r\n"
. "Connection: close\r\n\r\n";
$receive->add($sock);
$send->remove($sock);
}
while ( my @receive = $receive->can_read( $self->{timeout} - ( time() - $started ) ) ) {
for my $sock ( @receive ) {
my $L = $sockets->{$sock};
my $found = 0;
while (<$sock>) {
next unless m{^Location: http://(?:[wW]{3}\.)?([a-zA-Z0-9.-]+)};
my $address = $1;
$found = $sent_queries = 1;
$self->log(LOGDEBUG, "uribl: Sending query from shortened URI (via $L->{host}): $1");
if ( $address =~ /[^\d.]/ ) { # hostname
$self->send_queries( $address );
} else { # IP address
my @fwd = split /\./, $address or next;
my $fwd = join '.', @fwd;
my $rev = join '.', reverse @fwd;
$self->send_queries( $rev, { forward_ip => $fwd } );
}
last;
}
$self->log(LOGNOTICE, "uribl: Couldn't resolve shortened URI $L->{host}/$L->{path}")
unless $found;
$receive->remove($sock);
delete $sockets->{$sock};
$self->{queries_completed}->{shortener}++;
}
}
}
return $sent_queries;
}
sub check_results { # used by async plugin
my ( $self ) = @_;
my $txn = $self->transaction;
while ( my $q = shift @{ $self->{matches} || [] } ) {
for ( @{ $q->{lists} } ) {
my $desc = "$q->{name} found in $_->{label}";
$desc .= ": $q->{txt}" if $q->{txt};
$self->log(LOGWARN, "uribl: ".$q->{txt} ? $desc : "$desc: $q->{a}");
if ( $_->{action} eq 'add-header' ) {
$txn->header->add('X-URIBL-Match', $desc);
} elsif ( $_->{action} eq 'deny' ) {
return DENY, $desc;
} elsif ( $_->{action} eq 'denysoft' ) {
return DENYSOFT, $desc;
}
}
}
return DECLINED;
}