Bug#1132413: udd: Ubuntu importers have failed since March 23, 2026

Hi,

On 07/04/26 at 17:08 +0200, Sebastien Bacher wrote:
> Hey again Lucas,
> 
> Urg, no wonder UDD is getting blocked; the ubuntu_bugs_gatherer script
> basically parse 163k launchpad pages, doing 10 parallel requesters without
> any rate limiting and not slowing doing in case the service struggles and
> start returning 429 or 503.
> 
> Ideally that would use launchpadlibAPI to use searchTasks() with a
> modified_since= to limit the list to bugs that changed since the previous
> run (if you have that information) or over a day (or a few days to
> accomodate for flakyness).
> 
> Switching to use API instead of text pages parsing requires a bit of work,
> but maybe a start for now would be to lower the number of parallel workers
> (+ a small sleep(0.3) or something between calls + pause if the service
> starts returning errors)?
> The launchpad team confirmed that the number of queries/the daily frequency
> isn't the problem but the "spike" is

For now I reduced the parallelism (num_fetchers) from 10 to 2, which
should clearly help spread the load over time. Maybe let's see if that's
better? I could reduce it to 1 and add a delay, but long-running
importers are a bit painful for other reasons, so I'd rather avoid that.

Lucas