Re: [meta-virtualization] [RFC PATCH 4/4] vcontainer-tarball: fix SDK environment script for CI

On Tue, Apr 28, 2026 at 4:56 AM Bruce Ashfield <[email protected]>
wrote:

>
>
> On Mon, Apr 27, 2026 at 9:13 PM Tim Orling via lists.yoctoproject.org
> <[email protected]> wrote:
>
>> When running in an AutoBuilder context, the variables are stripped
>> since the script is not actually "sourced", but rather parsed.
>>
>> This resulted in VCONTAINER_DIR being empty and therefore no
>> 'vdkr', 'vpdmn' and friends available to run in CI builder steps.
>>
>
>  Tim,
>
> I agree the autobuilder's Python parser can't handle our bash-oriented
> environment script, but I'd rather not add this much complexity to make
> one script serve both environments.
>
> The inline if [ -n "${BASH_SOURCE[0]:-}" ] guards and dual-path logic
> make the script harder to reason about  and maintain.
>

Agreed. That was an ugly hack and I did not like it either.


>
> Instead, I think we should generate two scripts:
>
>   1. environment-setup-* (existing) — stays as-is, pure bash, BASH_SOURCE
> based, for interactive/devshell use
>   2. environment-setup-ci — flat export FOO="/absolute/path" lines with
> baked-in paths, no shell variable references, no unset, specifically for
> the autobuilder's enable_tools_tarball() parser
>
>
The current enable_tools_tarball uses a "greedy" glob of
"environment-setup-*" [1], which could be modified to be passed in
as a keyword arg -- with a default.

[1]
https://git.yoctoproject.org/yocto-autobuilder-helper/tree/scripts/utils.py?id=7ba82651cae130890e2120dccca939153ec7042e#n445

The CI script would be simple:
>
>   export VCONTAINER_DIR="/opt/poky/sysroots/x86_64-pokysdk-linux"
>   export OECORE_NATIVE_SYSROOT="/opt/poky/sysroots/x86_64-pokysdk-linux"
>   export
> PATH="/opt/poky/sysroots/x86_64-pokysdk-linux:/opt/poky/sysroots/x86_64-pokysdk-linux/usr/bin:/usr/bin:/bin"
>
>
Currently, since the vcontainer-tarball is not yet published to
downloads.yoctoproject.org, the CI path is passed in as:
VCONTAINER_SDK=/srv/autobuilder/
autobuilder.yocto.io/pub/vcontainer-tarball-latest/vcontainer-standalone.sh

When 'vcontainer' is enabled as a worker feature, a pre-build step is added
which currently installs the SDK to:
/home/pokybuild/yocto-worker/vcontainer-tests/build/build/vcontainer-test-extracted

Where "vcontainer-tests" is a "builder" (job) which needs the SDK
installed. This was also partially done this way so
that I can test a "just built" vcontainer-tarball with test jobs like
vcontainer-tests, vdkr-tests and vpdmn-tests (before
even trying to build and push containers).



> SDK relocation rewrites the absolute paths at install time. The autobuilder
> helper would reference this file instead of the bash one. No conditional
> logic, no dual-purpose complexity.
>

I agree we should avoid that brittle path of conditional logic and
dual-purpose complexity.


>
> Would this work with the autobuilder helper's enable_tools_tarball()?
>

I think it will work. Given the reality of how these things interact in the
Yocto AutoBuilder context, I will play
around with it. I already know what works and doesn't and how to test it ;)

Thank you for the feedback!

--Tim


>
> If so I can implement it.
>
> Bruce
>
>
>
>>
>> AI-Generated: Claude Cowork Opus 4.7
>> Signed-off-by: Tim Orling <[email protected]>
>> ---
>>  .../vcontainer/vcontainer-tarball.bb          | 69 +++++++++++++++----
>>  1 file changed, 57 insertions(+), 12 deletions(-)
>>
>> diff --git a/recipes-containers/vcontainer/vcontainer-tarball.bb
>> b/recipes-containers/vcontainer/vcontainer-tarball.bb
>> index ed9b8e13..9564164c 100644
>> --- a/recipes-containers/vcontainer/vcontainer-tarball.bb
>> +++ b/recipes-containers/vcontainer/vcontainer-tarball.bb
>> @@ -353,20 +353,57 @@ EOF
>>
>>  script=${SDK_OUTPUT}/${SDKPATH}/environment-setup-${REAL_MULTIMACH_TARGET_SYS}
>>
>>      # Create environment script
>> -    # Set OECORE_NATIVE_SYSROOT temporarily for SDK relocation, then
>> unset it
>> -    # (like buildtools-tarball does to avoid confusing other Yocto tools)
>> +    #
>> +    # The script is written so that it works both when sourced by bash
>> (the
>> +    # normal interactive / devshell case) AND when "installed" by
>> +    # yocto-autobuilder-helper's enable_tools_tarball() in CI, which
>> does NOT
>> +    # source the file with bash -- it parses line by line in Python and
>> only
>> +    # honours lines that start with "export " or "unset " at column 0,
>> and
>> +    # only substitutes $PATH (see
>> yocto-autobuilder-helper/scripts/utils.py).
>> +    #
>> +    # Consequences:
>> +    #  - The primary VCONTAINER_DIR / OECORE_NATIVE_SYSROOT / PATH
>> values must
>> +    #    be emitted as plain `export FOO="<absolute-path>"` lines with
>> the
>> +    #    absolute paths baked in at build time via Yocto variables
>> +    #    (${SDKPATH}, ${SDKPATHNATIVE}). Yocto SDK relocation rewrites
>> these
>> +    #    paths at install time (via the installer's -d <dir> argument).
>> +    #  - PATH must not reference $VCONTAINER_DIR /
>> $OECORE_NATIVE_SYSROOT via
>> +    #    shell indirection (the Python parser won't expand them); inline
>> the
>> +    #    absolute paths there too.
>> +    #  - Any bash-only refinement (BASH_SOURCE-based relocation, final
>> +    #    unset of OECORE_NATIVE_SYSROOT) lives inside an `if` block so
>> the
>> +    #    Python parser ignores it, while real bash still executes it.
>>      cat > $script <<'HEADER'
>>  #!/bin/bash
>>  # vcontainer environment setup script
>>  # Source this file: source environment-setup-none
>>  # Or use the symlink: source init-env.sh
>> -
>> -VCONTAINER_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
>>  HEADER
>> -    # Yocto variables (${SDK_SYS}, ${SDKPATHNATIVE}) expand at parse time
>> -    # Shell variables use $VAR to avoid Yocto expansion
>> +    # Primary env vars: emit as plain `export FOO="<abs-path>"` so the
>> +    # yocto-autobuilder-helper Python parser picks them up. Yocto
>> variables
>> +    # expand at bitbake time; the SDK installer's relocation pass
>> rewrites
>> +    # the baked-in paths at install time.
>> +    echo '' >> $script
>> +    echo '# Primary values -- literal absolute paths that survive
>> parsing by' >> $script
>> +    echo '# yocto-autobuilder-helper enable_tools_tarball() (not sourced
>> by bash).' >> $script
>> +    echo '# SDK relocation at install time rewrites the absolute paths
>> below.' >> $script
>> +    echo 'export VCONTAINER_DIR="'"${SDKPATH}"'"' >> $script
>>      echo 'export OECORE_NATIVE_SYSROOT="'"${SDKPATHNATIVE}"'"' >> $script
>> -    echo 'export
>> PATH="$VCONTAINER_DIR:$OECORE_NATIVE_SYSROOT/usr/bin:/usr/bin:/bin:$PATH"'
>> >> $script
>> +    echo 'export
>> PATH="'"${SDKPATH}"':'"${SDKPATHNATIVE}"'/usr/bin:/usr/bin:/bin:$PATH"' >>
>> $script
>> +
>> +    # Bash-only refinement: if the file is actually being sourced by
>> bash,
>> +    # re-derive VCONTAINER_DIR from the real on-disk location so a
>> manually
>> +    # moved/copied tarball still works. Invisible to the autobuilder
>> Python
>> +    # parser (doesn't start with export/unset at column 0).
>> +    cat >> $script <<'BASHREFINE'
>> +
>> +# When sourced by bash (interactive / devshell), prefer the real on-disk
>> +# location so a manually-moved tarball still works.
>> +if [ -n "${BASH_SOURCE[0]:-}" ]; then
>> +    VCONTAINER_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
>> +    export VCONTAINER_DIR
>> +fi
>> +BASHREFINE
>>      cat >> $script <<'FOOTER'
>>
>>  echo "vcontainer environment configured."
>> @@ -395,11 +432,19 @@ echo ""
>>  echo "Architectures: ${VCONTAINER_ARCHITECTURES}"
>>  ENVEOF
>>
>> -    # Unset OECORE_NATIVE_SYSROOT to avoid confusing other Yocto tools
>> -    # (same pattern as buildtools-tarball)
>> -    echo '' >> $script
>> -    echo '# Clean up - unset to avoid confusing other Yocto tools' >>
>> $script
>> -    echo 'unset OECORE_NATIVE_SYSROOT' >> $script
>> +    # Gated unset: same pattern as buildtools-tarball (unset
>> +    # OECORE_NATIVE_SYSROOT after interactive sourcing so it doesn't
>> confuse
>> +    # other Yocto tools), but guarded behind an `if` so the autobuilder's
>> +    # line-based parser (which matches "unset " only at column 0) leaves
>> +    # OECORE_NATIVE_SYSROOT set in the CI environment.
>> +    cat >> $script <<'UNSET_BLOCK'
>> +
>> +# Avoid confusing other Yocto tools post-source (matches
>> buildtools-tarball).
>> +# Gated so yocto-autobuilder-helper's parser leaves
>> OECORE_NATIVE_SYSROOT set.
>> +if [ -n "${BASH_SOURCE[0]:-}" ]; then
>> +    unset OECORE_NATIVE_SYSROOT
>> +fi
>> +UNSET_BLOCK
>>
>>      # Replace placeholder with actual SDK_SYS
>>      sed -i -e "s:@SDK_SYS@:${SDK_SYS}:g" $script
>> --
>> 2.50.1 (Apple Git-155)
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#9753):
>> https://lists.yoctoproject.org/g/meta-virtualization/message/9753
>> Mute This Topic: https://lists.yoctoproject.org/mt/119042096/1050810
>> Group Owner: [email protected]
>> Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub [
>> [email protected]]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await thee
> at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
>