Re: Best approach for bulk DNS lookups in Python — socket vs dnspython vs asyncio

> On 5 Mar 2026, at 01:41, Vahid Shaik <[email protected]> wrote:
> 
> Hi all,
> 
> I've been working on a Python script to perform bulk DNS lookups (A, MX, TXT records) for a list of ~500 domains to audit SPF/DKIM/DMARC configurations.
> 
> Currently I'm using `dns.resolver` from dnspython with ThreadPoolExecutor:
> 
> ```python
> import dns.resolver
> from concurrent.futures import ThreadPoolExecutor
> 
> def check_spf(domain):
>    try:
>        answers = dns.resolver.resolve(domain, 'TXT')
>        return [r.to_text() for r in answers if 'v=spf1' in r.to_text()]
>    except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer, dns.resolver.Timeout):
>        return []
> 
> with ThreadPoolExecutor(max_workers=20) as pool:
>    results = dict(zip(domains, pool.map(check_spf, domains)))
> ```
> 
> This works but gets slow beyond 200 domains. I've considered:
> 
> 1. **asyncio + aiodns** — should be faster but I've seen reports of reliability issues with certain record types
This is the way I would go for performance. Threads do not scale to large numbers.
Suggest you test and see if the reports are true.
Maybe fix and problems you find and PR back a fix
You could also look at twisted that has async dns that worked well for me.
Sorry I do not have any code to shares as it was commercial project.

Barry

> 2. **socket.getaddrinfo** — built-in but limited to A/AAAA records
> 3. **subprocess calling dig** — feels hacky
> 
> For anyone dealing with similar DNS automation tasks, I've been cross-referencing my script results against https://dnsrobot.net/spf-checker to verify accuracy — it checks SPF, DKIM, and DMARC in one shot which is useful for spot-checking.
> 
> Has anyone benchmarked aiodns vs dnspython for large-scale lookups? Any gotchas with asyncio DNS resolution I should know about?
> 
> Thanks,
> Vahid
> --
> https://mail.python.org/mailman3//lists/python-list.python.org

-- 
https://mail.python.org/mailman3//lists/python-list.python.org