Re: [bitbake-devel] [2.18][PATCH] bitbake-worker: Evaluate exported_vars() before emptying the environment
Yoann Congal <[email protected]> Sat, 25 Jul 2026 20:35:50 +0200
| Newsgroups | org.openembedded.lists.bitbake-devel |
|---|---|
| Message-ID | <CAMSfU+72wmY=nzD41=_=eg7sKDLLyXJcFsBTCw_wLx8G3Gb3ZA@mail.gmail.com> |
--000000000000cd9005065773c23b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Le sam. 25 juil. 2026, 20:01, Alejandro Mery via lists.openembedded.org <[email protected]> a =C3=A9crit : > > On 23/07/2026 07:06, Richard Purdie wrote: > > On Mon, 2026-07-20 at 19:58 +0000, Alejandro Mery via > lists.openembedded.org wrote: > >> exported_vars() returns a lazy generator. It was bound before > >> empty_environment() cleared the process environment and only iterated > >> afterwards, so any datastore expansion deferred until iteration ran wi= th > >> PATH already wiped. > >> > >> Recipes whose exported variables expand a command during that loop hit > >> this. A gitver-style PV such as "${@get_git_pv(d, ...)}" runs git whil= e > >> PATH is empty, so the git wrapper on PATH is bypassed and the real git > >> runs directly. Under pseudo this fakes uid 0 against a repository owne= d > >> by the real user, and git aborts with "detected dubious ownership", > >> failing do_package intermittently (only on reparse, when the value is > >> re-expanded rather than served from cache). > >> > >> Materialise the generator into a list before emptying the environment = so > >> every expansion happens while PATH is still intact. > >> > >> Signed-off-by: Alejandro Mery <[email protected]> > >> --- > >> bin/bitbake-worker | 2 +- > >> 1 file changed, 1 insertion(+), 1 deletion(-) > >> > >> diff --git a/bin/bitbake-worker b/bin/bitbake-worker > >> index aa14ef191..0db1531e0 100755 > >> --- a/bin/bitbake-worker > >> +++ b/bin/bitbake-worker > >> @@ -290,7 +290,7 @@ def fork_off_task(cfg, data, databuilder, > workerdata, extraconfigdata, runtask): > >> > >> # exported_vars() returns a generator which *cannot* > be passed to os.environ.update() > >> # successfully. We also need to unset anything from > the environment which shouldn't be there > >> - exports =3D bb.data.exported_vars(the_data) > >> + exports =3D list(bb.data.exported_vars(the_data)) > >> > >> bb.utils.empty_environment() > >> for e, v in exports: > > Thanks for this. Nice work in tracking it down! > > > > I think in this case we need a couple of things, firstly will be a bit > > more explination in the code with a comment about why we need a list > > there. > > > > The failure case is so unusual we should also probably write a test > > case to make sure we test for and don't regress that issue which we'd > > add to the existing tests in bitbake-selftest. > > > > Help with that would be welcome or we'll get to it when we can... > > > I'll send a v2 soon > Is this for master? In that case, please remove the [2.18] tag. Thanks! --=20 Yoann Congal > --000000000000cd9005065773c23b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"auto"><div dir=3D"auto"><br></div><br><div class=3D"gmail_quote= gmail_quote_container" dir=3D"auto"><div dir=3D"ltr" class=3D"gmail_attr">= Le sam. 25 juil. 2026, 20:01, Alejandro Mery via <a href=3D"http://lists.op= enembedded.org">lists.openembedded.org</a> <amery=3D<a href=3D"mailto:ap= [email protected]">[email protected]</a>> a = =C3=A9crit=C2=A0:<br></div><blockquote class=3D"gmail_quote" style=3D"margi= n:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex= "><br> On 23/07/2026 07:06, Richard Purdie wrote:<br> > On Mon, 2026-07-20 at 19:58 +0000, Alejandro Mery via <a href=3D"http:= //lists.openembedded.org" rel=3D"noreferrer noreferrer" target=3D"_blank">l= ists.openembedded.org</a> wrote:<br> >> exported_vars() returns a lazy generator. It was bound before<br> >> empty_environment() cleared the process environment and only itera= ted<br> >> afterwards, so any datastore expansion deferred until iteration ra= n with<br> >> PATH already wiped.<br> >><br> >> Recipes whose exported variables expand a command during that loop= hit<br> >> this. A gitver-style PV such as "${@get_git_pv(d, ...)}"= runs git while<br> >> PATH is empty, so the git wrapper on PATH is bypassed and the real= git<br> >> runs directly. Under pseudo this fakes uid 0 against a repository = owned<br> >> by the real user, and git aborts with "detected dubious owner= ship",<br> >> failing do_package intermittently (only on reparse, when the value= is<br> >> re-expanded rather than served from cache).<br> >><br> >> Materialise the generator into a list before emptying the environm= ent so<br> >> every expansion happens while PATH is still intact.<br> >><br> >> Signed-off-by: Alejandro Mery <<a href=3D"mailto:[email protected]= o" target=3D"_blank" rel=3D"noreferrer">[email protected]</a>><br> >> ---<br> >>=C2=A0 =C2=A0bin/bitbake-worker | 2 +-<br> >>=C2=A0 =C2=A01 file changed, 1 insertion(+), 1 deletion(-)<br> >><br> >> diff --git a/bin/bitbake-worker b/bin/bitbake-worker<br> >> index aa14ef191..0db1531e0 100755<br> >> --- a/bin/bitbake-worker<br> >> +++ b/bin/bitbake-worker<br> >> @@ -290,7 +290,7 @@ def fork_off_task(cfg, data, databuilder, work= erdata, extraconfigdata, runtask):<br> >>=C2=A0 =C2=A0<br> >>=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 # exported_vars() returns a generator = which *cannot* be passed to os.environ.update()<br> >>=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 # successfully. We also need to unset = anything from the environment which shouldn't be there<br> >> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 exports =3D bb.data.exported_vars(the_data)<br> >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 exports =3D list(bb.data.exported_vars(the_data= ))<br> >>=C2=A0 =C2=A0<br> >>=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 bb.utils.empty_environment()<br> >>=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for e, v in exports:<br> > Thanks for this. Nice work in tracking it down!<br> ><br> > I think in this case we need a couple of things, firstly will be a bit= <br> > more explination in the code with a comment about why we need a list<b= r> > there.<br> ><br> > The failure case is so unusual we should also probably write a test<br= > > case to make sure we test for and don't regress that issue which w= e'd<br> > add to the existing tests in bitbake-selftest.<br> ><br> > Help with that would be welcome or we'll get to it when we can...<= br> <br> <br> I'll send a v2 soon<br></blockquote></div><div dir=3D"auto"><br></div><= div dir=3D"auto">Is this for master? In that case, please remove the [2.18]= tag.</div><div dir=3D"auto"><br></div><div dir=3D"auto">Thanks!</div><div = dir=3D"auto">--=C2=A0</div><div dir=3D"auto">Yoann Congal</div><div class= =3D"gmail_quote gmail_quote_container" dir=3D"auto"><blockquote class=3D"gm= ail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,= 204,204);padding-left:1ex"> </blockquote></div></div> --000000000000cd9005065773c23b--