Re: [bitbake-devel] [PATCH] fetch/git: add reference option
Bruce Ashfield <[email protected]> Fri, 26 Jun 2026 15:56:02 -0400
| Newsgroups | org.openembedded.lists.bitbake-devel |
|---|---|
| Message-ID | <CADkTA4MVj9Ja806ja_0y7zo2J4sFdOvvUbm2_aR3tcQig4brvA@mail.gmail.com> |
--0000000000004fa34f06552d806c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Jun 26, 2026 at 3:10=E2=80=AFPM Daniel Walker via lists.openembedde= d.org <[email protected]> wrote: > Adds a option to the git fetcher which can be used to specify a local > reference for a given SRC_URI. Which is then fed into the clone command > as "--reference-if-able" which make the option fault tolerant if the > directory doesn't exist. > We tried this in the past (maybe 10 years ago now) to chunk the kernel into different sets of blobs and do a set of references for optimal sharing, repacking, etc. In the end, it was discarded as unnecessary complexity. I'm still of the opinion that is the case. What problem is this solving that justifies the complexity ? the patch doesn't show a real world use case, what the SRC_URI would look like, where the referenced clone would be, what would happen if the referenced repository was somehow cleaned, or any number of different interactions (what happens if shallow is specified, etc, etc). Bruce > > AI-Generated: OpenAI Codex > Signed-off-by: Daniel Walker <[email protected]> > Change-Id: Ib5eb685ddfb48fa19f1c65cfc468894806f99e83 > --- > .../bitbake-user-manual-fetching.rst | 7 +++++++ > lib/bb/fetch2/git.py | 12 +++++++++++- > 2 files changed, 18 insertions(+), 1 deletion(-) > > diff --git a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst > b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst > index b96018a4e..6e4fc0800 100644 > --- a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst > +++ b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst > @@ -398,6 +398,13 @@ This fetcher supports the following parameters: > preserve the local cache carefully for future use. The default value > for this parameter is "0". > > +- *"reference":* Specifies a local directory to use as a reference > + repository. When set, the fetcher passes the directory to ``git clone= `` > + with ``--reference-if-able``. Git uses the reference repository's > + objects when possible, reducing the amount of object data copied or > + fetched. If the reference directory cannot be used, Git continues > + cloning without it. > + > - *"nobranch":* Tells the fetcher to not check the SHA validation for > the branch when set to "1". The default is "0". Set this option for > the recipe that refers to the commit that is valid for any namespace > diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py > index 0fbd85441..df1c4415f 100644 > --- a/lib/bb/fetch2/git.py > +++ b/lib/bb/fetch2/git.py > @@ -22,6 +22,10 @@ Supported SRC_URI options are: > remind fetcher to preserve local cache carefully for future use. > The default value is "0", set rebaseable=3D1 for rebaseable git repo. > > +- reference > + Use the specified local directory as a reference repository by passin= g > it > + to git clone with --reference-if-able. > + > - nocheckout > Don't checkout source code when unpacking. set this option for the > recipe > who has its own routine to checkout code. > @@ -195,6 +199,12 @@ class Git(FetchMethod): > if ud.bareclone: > ud.cloneflags.append("--mirror") > > + ud.reference_args =3D [] > + ud.reference =3D ud.parm.get("reference") > + if ud.reference: > + ud.reference_args =3D ["--reference-if-able", ud.reference] > + ud.cloneflags.extend(ud.reference_args) > + > ud.shallow_skip_fast =3D False > ud.shallow =3D d.getVar("BB_GIT_SHALLOW") =3D=3D "1" > ud.shallow_extra_refs =3D (d.getVar("BB_GIT_SHALLOW_EXTRA_REFS")= or > "").split() > @@ -436,7 +446,7 @@ class Git(FetchMethod): > objects =3D os.path.join(repourl_path, 'objects') > if os.path.isdir(objects) and not os.path.islink(objects= ): > repourl =3D repourl_path > - clone_cmd =3D ud.basecmd + ['clone', '--bare', '--mirror', > repourl, ud.clonedir, '--progress'] > + clone_cmd =3D ud.basecmd + ['clone', '--bare', '--mirror'] + > ud.reference_args + [repourl, ud.clonedir, '--progress'] > if ud.proto.lower() !=3D 'file': > bb.fetch2.check_network_access(d, clone_cmd, ud.url) > progresshandler =3D GitProgressHandler(d) > -- > 2.44.0 > > > -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- > Links: You receive all messages sent to this group. > View/Reply Online (#19794): > https://lists.openembedded.org/g/bitbake-devel/message/19794 > Mute This Topic: https://lists.openembedded.org/mt/119994628/1050810 > Group Owner: [email protected] > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ > [email protected]] > -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- > > --=20 - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II --0000000000004fa34f06552d806c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"fon= t-size:small"><br></div></div><br><div class=3D"gmail_quote gmail_quote_con= tainer"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, Jun 26, 2026 at 3:10= =E2=80=AFPM Daniel Walker via <a href=3D"http://lists.openembedded.org">lis= ts.openembedded.org</a> <danielwa=3D<a href=3D"mailto:[email protected]= enembedded.org">[email protected]</a>> wrote:<br></div><b= lockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-le= ft:1px solid rgb(204,204,204);padding-left:1ex">Adds a option to the git fe= tcher which can be used to specify a local<br> reference for a given SRC_URI. Which is then fed into the clone command<br> as "--reference-if-able" which make the option fault tolerant if = the<br> directory doesn't exist.<br></blockquote><div><br></div><div><div class= =3D"gmail_default" style=3D"font-size:small">We tried this in the past (may= be 10 years ago now) to chunk the</div><div class=3D"gmail_default" style= =3D"font-size:small">kernel into different sets=C2=A0<span style=3D"backgro= und-color:transparent">of blobs and do a set of references for=C2=A0</span>= </div><div class=3D"gmail_default" style=3D"font-size:small"><span style=3D= "background-color:transparent">optimal=C2=A0sharing, repacking,=C2=A0</span= ><span style=3D"background-color:transparent">etc.</span></div><div class= =3D"gmail_default" style=3D"font-size:small"><br></div><div class=3D"gmail_= default" style=3D"font-size:small">In the end, it was discarded=C2=A0as unn= ecessary=C2=A0complexity.<br><br>I'm still of the opinion that is the c= ase. What=C2=A0problem is this solving</div><div class=3D"gmail_default" st= yle=3D"font-size:small">that justifies the complexity ?</div><div class=3D"= gmail_default" style=3D"font-size:small"><br></div><div class=3D"gmail_defa= ult" style=3D"font-size:small">the patch doesn't show a real world use = case, what the SRC_URI</div><div class=3D"gmail_default" style=3D"font-size= :small">would look like, where the referenced clone would be, what would</d= iv><div class=3D"gmail_default" style=3D"font-size:small">happen if the ref= erenced repository was somehow=C2=A0cleaned, or</div><div class=3D"gmail_de= fault" style=3D"font-size:small">any number of different interactions (what= happens if shallow is</div><div class=3D"gmail_default" style=3D"font-size= :small">specified, etc, etc).</div><div class=3D"gmail_default" style=3D"fo= nt-size:small"><br></div><div class=3D"gmail_default" style=3D"font-size:sm= all">Bruce</div></div><div>=C2=A0</div><blockquote class=3D"gmail_quote" st= yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padd= ing-left:1ex"> <br> AI-Generated: OpenAI Codex<br> Signed-off-by: Daniel Walker <<a href=3D"mailto:[email protected]" targ= et=3D"_blank">[email protected]</a>><br> Change-Id: Ib5eb685ddfb48fa19f1c65cfc468894806f99e83<br> ---<br> =C2=A0.../bitbake-user-manual-fetching.rst=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 7 +++++++<br> =C2=A0lib/bb/fetch2/git.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| 12 += ++++++++++-<br> =C2=A02 files changed, 18 insertions(+), 1 deletion(-)<br> <br> diff --git a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/doc= /bitbake-user-manual/bitbake-user-manual-fetching.rst<br> index b96018a4e..6e4fc0800 100644<br> --- a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst<br> +++ b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst<br> @@ -398,6 +398,13 @@ This fetcher supports the following parameters:<br> =C2=A0 =C2=A0 preserve the local cache carefully for future use. The defaul= t value<br> =C2=A0 =C2=A0 for this parameter is "0".<br> <br> +-=C2=A0 *"reference":* Specifies a local directory to use as a r= eference<br> +=C2=A0 =C2=A0repository. When set, the fetcher passes the directory to ``g= it clone``<br> +=C2=A0 =C2=A0with ``--reference-if-able``. Git uses the reference reposito= ry's<br> +=C2=A0 =C2=A0objects when possible, reducing the amount of object data cop= ied or<br> +=C2=A0 =C2=A0fetched. If the reference directory cannot be used, Git conti= nues<br> +=C2=A0 =C2=A0cloning without it.<br> +<br> =C2=A0-=C2=A0 *"nobranch":* Tells the fetcher to not check the SH= A validation for<br> =C2=A0 =C2=A0 the branch when set to "1". The default is "0&= quot;. Set this option for<br> =C2=A0 =C2=A0 the recipe that refers to the commit that is valid for any na= mespace<br> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py<br> index 0fbd85441..df1c4415f 100644<br> --- a/lib/bb/fetch2/git.py<br> +++ b/lib/bb/fetch2/git.py<br> @@ -22,6 +22,10 @@ Supported SRC_URI options are:<br> =C2=A0 =C2=A0 remind fetcher to preserve local cache carefully for future u= se.<br> =C2=A0 =C2=A0 The default value is "0", set rebaseable=3D1 for re= baseable git repo.<br> <br> +- reference<br> +=C2=A0 =C2=A0Use the specified local directory as a reference repository b= y passing it<br> +=C2=A0 =C2=A0to git clone with --reference-if-able.<br> +<br> =C2=A0- nocheckout<br> =C2=A0 =C2=A0 Don't checkout source code when unpacking. set this optio= n for the recipe<br> =C2=A0 =C2=A0 who has its own routine to checkout code.<br> @@ -195,6 +199,12 @@ class Git(FetchMethod):<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if ud.bareclone:<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ud.cloneflags.append("= --mirror")<br> <br> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 ud.reference_args =3D []<br> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 ud.reference =3D ud.parm.get("reference&q= uot;)<br> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 if ud.reference:<br> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ud.reference_args =3D ["--r= eference-if-able", ud.reference]<br> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ud.cloneflags.extend(ud.referenc= e_args)<br> +<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ud.shallow_skip_fast =3D False<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ud.shallow =3D d.getVar("BB_GIT_SHAL= LOW") =3D=3D "1"<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ud.shallow_extra_refs =3D (d.getVar("= ;BB_GIT_SHALLOW_EXTRA_REFS") or "").split()<br> @@ -436,7 +446,7 @@ class Git(FetchMethod):<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0objects =3D o= s.path.join(repourl_path, 'objects')<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if os.path.is= dir(objects) and not os.path.islink(objects):<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0repourl =3D repourl_path<br> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 clone_cmd =3D ud.basecmd + ['= ;clone', '--bare', '--mirror', repourl, ud.clonedir, &#= 39;--progress']<br> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 clone_cmd =3D ud.basecmd + ['= ;clone', '--bare', '--mirror'] + ud.reference_args + [r= epourl, ud.clonedir, '--progress']<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if ud.proto.lower() !=3D &#= 39;file':<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bb.fetch2.che= ck_network_access(d, clone_cmd, ud.url)<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0progresshandler =3D GitProg= ressHandler(d)<br> -- <br> 2.44.0<br> <br> <br> -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-<br> Links: You receive all messages sent to this group.<br> View/Reply Online (#19794): <a href=3D"https://lists.openembedded.org/g/bit= bake-devel/message/19794" rel=3D"noreferrer" target=3D"_blank">https://list= s.openembedded.org/g/bitbake-devel/message/19794</a><br> Mute This Topic: <a href=3D"https://lists.openembedded.org/mt/119994628/105= 0810" rel=3D"noreferrer" target=3D"_blank">https://lists.openembedded.org/m= t/119994628/1050810</a><br> Group Owner: <a href=3D"mailto:bitbake-devel%[email protected]= " target=3D"_blank">[email protected]</a><br> Unsubscribe: <a href=3D"https://lists.openembedded.org/g/bitbake-devel/unsu= b" rel=3D"noreferrer" target=3D"_blank">https://lists.openembedded.org/g/bi= tbake-devel/unsub</a> [<a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a>]<br> -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-<br> <br> </blockquote></div><div><br clear=3D"all"></div><div><br></div><span class= =3D"gmail_signature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_s= ignature">- Thou shalt not follow the NULL pointer, for chaos and madness a= wait thee at its end<br>- "Use the force Harry" - Gandalf, Star T= rek II<br><br></div></div> --0000000000004fa34f06552d806c--