new in UDD: duck importer (URL checker)

Lucas Nussbaum <[email protected]>
Newsgroups gmane.linux.debian.devel.quality-assurance
Message-ID <[email protected]>
Hi,

I added a duck importer[0] and a duck dashboard[1] to UDD.
Previously the duck.debian.net service was providing regular checks
about URLs in Debian package, but it broke at some point around 2020.

# importer internals

The UDD importer[0] works as follows:
- URLs are collected out of all source packages using the 'duck' package
  (a vendorized version[2], because I needed to add a dump mode, which
  is also submitted as #1107642)
  This is tracked in the duck_raw (raw output from duck) and duck_urls
  (easier to work with, one row per (source/version, URL))
- On a regular basis, UDD checks all known URLs (with a retry policy
  depending on whether it's possibly a transient failure).
  duck is _not_ used for this part (the importer has its own check
  logic), because the check happens URL per URL, not package per package.
  This is tracked in the duck_url_status.

# dashboards

- per maintainer view: https://udd.debian.org/duck/?email1=pkg-perl-maintainers%40lists.alioth.debian.org#results
- The DMD 'duck' column works again: https://udd.debian.org/dmd/?email1=pkg-perl-maintainers%40lists.alioth.debian.org&email2=&email3=&packages=&ignpackages=&format=html#details

# statistics

* UDD knows about 207055 URLs
* 15207 URLs (7.34%) are failing
* 937 (2.37%) source packages failed to be processed by duck (I need to
  look into that)

[0] https://salsa.debian.org/qa/udd/-/blob/master/rimporters/duck.rb?ref_type=heads
[1] https://udd.debian.org/duck/
[2] https://salsa.debian.org/qa/udd/-/tree/master/vendor/duck?ref_type=heads

Best,

Lucas
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.