Bug#1132413: udd: Ubuntu importers have failed since March 23, 2026

Hey again Lucas,

Urg, no wonder UDD is getting blocked; the ubuntu_bugs_gatherer script 
basically parse 163k launchpad pages, doing 10 parallel requesters 
without any rate limiting and not slowing doing in case the service 
struggles and start returning 429 or 503.

Ideally that would use launchpadlibAPI to use searchTasks() with a 
modified_since= to limit the list to bugs that changed since the 
previous run (if you have that information) or over a day (or a few days 
to accomodate for flakyness).

Switching to use API instead of text pages parsing requires a bit of 
work, but maybe a start for now would be to lower the number of parallel 
workers (+ a small sleep(0.3) or something between calls + pause if the 
service starts returning errors)?
The launchpad team confirmed that the number of queries/the daily 
frequency isn't the problem but the "spike" is

Cheers,
Sébastien

Le 07/04/2026 à 16:24, Lucas Nussbaum a écrit :
> https://salsa.debian.org/qa/udd/-/blob/master/udd/ubuntu_bugs_gatherer.py?ref_type=heads
> for the second script.
>
> The refresh frequency of ubuntu-bugs could probably be reduced if that
> helps.