Bug#1132413: udd: Ubuntu importers have failed since March 23, 2026

Hey Lucas,

Thanks for flagging that, it had been indeed blocked again while they 
were trying to cut some load issue due to IA bots/scapers.

Do you have an idea of the number of API requests/pages load UDD is 
usually doing on a $period (day?), I wonder if the load is really that 
high and if there is some client optimization that could be done in UDD...

Cheers,
Sébastien

Le 06/04/2026 à 21:26, Lucas Nussbaum a écrit :
> On 03/04/26 at 11:16 -0400, Jeremy Bícha wrote:
>> On Tue, Mar 31, 2026 at 5:17 PM Lucas Nussbaum <[email protected]> wrote:
>>> On 31/03/26 at 08:32 -0700, Jeremy Bícha wrote:
>>>> https://udd.debian.org/udd-status.cgi is reporting that these Ubuntu
>>>> importers haven't run successfully since March 23, 2026:
>>>> - archive-ubuntu
>>>> - ubuntu-bugs
>>>> - ubuntu-upload-history
>>>>
>>>> The ubuntu-upload-history log excerpt includes:
>>>>
>>>>    urllib.error.HTTPError: HTTP Error 503: Service Unavailable
>>>>    ERROR:launchpad-ubuntu-changes:Error: Something broke badly.
>>>>
>>>> I know that Launchpad has been having difficulty staying fully
>>>> available and fast in the past few weeks.
>>>>
>>>> I'd expect for udd to retry imports later after failure but the "next
>>>> time" column also says March 23 so maybe these importers aren't
>>>> retrying any more?
>>> Hi,
>>>
>>> Thanks for the report.
>>>
>>> I think that the reason is that ubuntu-upload-history got stuck, and as
>>> a result, blocked the other importers because they are in the same
>>> "queue" in the UDD orchestrator.
>>>
>>> I added a timeout. Let's see if this fixes the issue.
>> ubuntu_upload_history is still lagging behind (and Launchpad is
>> struggling today!), but the other Ubuntu jobs are up-to-date or close
>> enough to it. It's those other jobs that are powering the dashboards
>> that I use so the situation is much better for me now.
> Hi,
>
> It looks like ullmann.debian.org (the machine behind udd.debian.org) is
> backlisted by launchpad again.
>
> ullmann:~$ curl -6 -kv https://launchpad.net/ubuntu/+archive/primary/+sourcepub/18280857/+files/changelog
> * Host launchpad.net:443 was resolved.
> * IPv6: 2620:2d:4000:1009::f3, 2620:2d:4000:1009::3ba
> * IPv4: (none)
> *   Trying [2620:2d:4000:1009::f3]:443...
>
> ^C
>
> ullmann:~$ mtr -c 5 -r -6 -T -P 443 2620:2d:4000:1009::f3
> Start: 2026-04-06T19:24:21+0000
> HOST: ullmann                     Loss%   Snt   Last   Avg  Best  Wrst StDev
>    1.|-- 2607:f8f0:614:1::2         0.0%     5    0.4   0.4   0.3   0.8   0.2
>    2.|-- 2607:f8f0:610:4::302       0.0%     5    0.4   0.4   0.3   0.4   0.0
>    3.|-- 2607:f8f0:610:4::305       0.0%     5    1.1   1.0   0.8   1.1   0.1
>          2607:f8f0:610:4::304
>    4.|-- 381-oran-cr1-ubc.vncv1.bc  0.0%     5    2.0   1.6   1.0   2.0   0.5
>          2607:f8f0:610:4::1f1
>    5.|-- 381-oran-cr1-ubc.vncv1.bc  0.0%     5  5105. 1634.   1.7 5105. 1990.1
>    6.|-- 2607:f8f0:1400::16         0.0%     5  1029. 1025.   2.2 3062. 1249.4
>    7.|-- 2607:f8f0:1400::16         0.0%     5  2050. 616.7   2.0 2050. 916.0
>    8.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
>    9.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
>   10.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
>   11.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
>   12.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
>   13.|-- he-as6939.torix.ca         0.0%     5  3119. 1895.  52.1 4154. 1832.7
>   14.|-- he-as6939.torix.ca         0.0%     5  2082. 1068.  51.3 2102. 1020.5
>   15.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
>   16.|-- port-channel1.core1.lon6. 40.0%     5  127.7 809.4 127.7 1154. 590.4
>   17.|-- swp9.il3-core2.canonical.  0.0%     5  1157. 741.4 126.6 2168. 913.9
>          port-channel1.core1.lon9.he.net
>          port-channel1.core1.lon6.he.net
>          swp9.il3-core1.canonical.com
>   18.|-- swp9.il3-core2.canonical.  0.0%     5  126.9 127.2 126.9 128.0   0.4
>          swp9.il3-core1.canonical.com
>          port-channel1.core1.lon6.he.net
>   19.|-- port-channel1.core1.lon6.  0.0%     5  128.2 534.9 127.0 1157. 557.6
>          vrrp.bond-core.il3-fw.canonical.com
>   20.|-- swp9.il3-core2.canonical.  0.0%     5  3197. 1965. 127.4 4219. 1829.8
>          swp9.il3-core1.canonical.com
>   21.|-- vrrp.bond-core.il3-fw.can  0.0%     5  7249. 2167. 127.2 7249. 2965.1
>   22.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
>
> ullmann.d.o is 2607:f8f0:614:1::1274:38
>
> If some could reach out to the Canonical team in charge of this, it
> would be great.
>
> Lucas