Re: [bitbake-devel] [PATCH v2] fetch2/crate: support configurable registry and index URLs

Hi Mathieu,

Thanks for testing this and digging into the failure.

I reproduced it locally with master + Chen Qi's filter_regex series + this
patch. The KeyError comes from the conflict resolution in
latest_versionstring(), not from either patch on its own. The merged tree
kept the old prefix check (ud.versionsurl.startswith('
https://index.crates.io/')), but this patch intentionally drops that in
favour of an explicit ud.crate_index_format == 'sparse' check, since the
whole point is to support custom sparse registries whose index URL isn't on
index.crates.io. The local sparse registry test serves its index from
127.0.0.1, so the prefix check sends NDJSON to the JSON API parser.

The right merge keeps the format check and threads filter_regex through
both helpers:

def latest_versionstring(self, ud, d, filter_regex=None):
if getattr(ud, 'crate_index_format', None) == 'sparse':
return self._latest_versionstring_from_index(ud, d, filter_regex)
return self._latest_versionstring_from_api(ud, d, filter_regex)

With that, CrateTest passes here (12 tests, 5 skipped), including
test_crate_fetches_from_local_sparse_registry.

I've got a v3 ready with exactly that resolution, rebased on the
filter_regex series. Agreed it's cleaner to wait until that series lands or
gets dropped, then I'll send the refreshed v3. I'll watch the thread.

Thanks,
Minsung

On Fri, May 15, 2026 04:12 AM, Mathieu Dubois-Briand <
[email protected]> wrote:

> On Thu May 14, 2026 at 10:05 PM CEST, minsung.cho via
> lists.openembedded.org wrote:
> > The crate fetcher previously hardcoded the crates.io download URL and
> > only supported a fixed /crate/versions API shape for custom registries.
> > This prevented recipes from using private registries or mirrors with
> > different download paths or Cargo sparse indexes.
> >
> > Add BB_CRATE_REGISTRY_URL[host] and BB_CRATE_INDEX_URL[host] templates
> > with {crate}, {version}, and {index_path} placeholders. Keep the
> > existing crates.io defaults, allow custom API-style version endpoints,
> > and mark sparse indexes explicitly so latest-version checks parse Cargo
> > index NDJSON instead of JSON API responses.
> >
> > Add non-network tests for default crates.io URLs, custom registry
> > templates, sparse index paths, trailing slash handling, and both
> > latest-version parser paths. Also make the fetch test cleanup chmod
> > invocation portable by placing -R before the mode.
> >
> > [YOCTO #16276]
> >
> > Signed-off-by: minsung.cho <[email protected]>
> > ---
>
> Hi,
>
> Thanks for your patch.
>
> I note it does conflict with some other patches currently in testing,
> and once these conflicts solved, I get some bitbake-selftest errors:
>
> ERROR: test_crate_fetches_from_local_sparse_registry
> (bb.tests.fetch.CrateTest.test_crate_fetches_from_local_sparse_registry)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/bitbake/lib/bb/tests/fetch.py",
> line 2829, in test_crate_fetches_from_local_sparse_registry
>     self.assertEqual(ud.method.latest_versionstring(ud, self.d),
>                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/bitbake/lib/bb/fetch2/crate.py",
> line 197, in latest_versionstring
>     return self._latest_versionstring_from_api(ud, d)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/bitbake/lib/bb/fetch2/crate.py",
> line 204, in _latest_versionstring_from_api
>     versions = [(0, i["num"], "") for i in json_data["versions"]]
>                                            ~~~~~~~~~^^^^^^^^^^^^
> KeyError: 'versions'
>
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3864
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3727
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3960
>
> Maybe the easiest solution is to wait a few days until said patches are
> either merged or rejected, then send a refreshed version.
>
> If you want to have a look at the current situation, here is the branch
> that was used for above builds:
>
> https://git.yoctoproject.org/poky-ci-archive/log/?h=bitbake/autobuilder.yoctoproject.org/valkyrie/a-full-3832
>
> Thanks,
> Mathieu
>
> --
> Mathieu Dubois-Briand, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>
>