Re: [bitbake-devel] [yocto] Documentation around adding custom fetchers to layer

On Wed, 2025-09-17 at 11:19 +0200, Alexander Kanavin via lists.openembedded.org wrote:
> On Wed, 17 Sept 2025 at 11:15, Ross Burton via lists.yoctoproject.org
> <[email protected]> wrote:
> > > I'm not seeing any documentation on how to add a custom fetcher to a layer, and the fetchers in poky don't provide enough information on how to register these fetchers. Is this functionality supported, or will we have to fork poky and keep it up-to-date to add a new fetcher?
> > 
> > This is a bitbake issue, so I’ve copied in the bitbake list.
> > 
> > There’s no standard way to add out-of-bitbake fetchers, as you say you need to add the fetcher instance to bb.fetch2.methods.
> > 
> > I guess you might be able to write a do_fetch prefunc that creates an instance of your fetcher and adds it to the list, but I would not be surprised if there were more places where that would be needed.
> > 
> > python use_my_fetcher() {
> >   bb.fetch2.methods.append(MyFetcher())
> > }
> > do_fetch[prefuncs] += “use_my_fetcher”
> 
> To be honest, I was going to say that custom fetchers are not
> supported. Doing that would fragment the ecosystem and make people
> invent all kinds of private magic in their builds, and subvert the
> promises that standard fetchers make.
> 
> Perhaps it would help to step back and look at what this customer
> fecher does. Can we see its source? Or an extended description of how
> the password manager is used, what data it takes and produces, and how
> that data is used in actual fetching? Then we'd think of how to do
> this with regular fetchers.

I'm struggling with time/bandwidth so I'll just mention what I can
remember without digging too much.

There have been cases of successfully injecting extra fetchers from the
metadata, particularly if you do it from the global class code,
probably in an event handler. I think some of the early npm/node/crate
code did this (I don't remember which) before we merged things into
bitbake itself.

There was an open bug for making it easier to define new fetchers from
the metadata and we did get most things working. Sadly I can't spot the
bug. The one area that was problematic was AUTOREV since that happened
before you could inject the fetcher in the class code in some cases. We
never did work out an API which would allow AUTOREV to work and allow a
general API.

Cheers,

Richard