Re: [yocto] Default to isolated Python virtual environments (venv)

while not specific to yocto i have hit simular problems in multiple different ways with python.

and python venv does not fix the problem for me.

example: rhel has python 3.6, if i run a python script from within xilinix fpga tools i get the xilinix supplied python, and if i use microchip/microsemi tools i get their version of python. same with vxworks and with stm cube suite.

why is this?  many of these include a version of gdb with python support.. which has to lock down to the absolute python version they link against and that is both great and causes these problems

its not like i can call xilinx and ask for a fix… (even though we buy north of $10million a year from them…) i have to deal with all of the other vendors too… and redhat and ubunto it sucks. i require a solution so i can ship my product.

so what we do is create a shell variable COMPANY_PYTHON3_EXE=/ABS/PATH/TO/PYTHON

on linux this often (or can) points to a shell script not the python executable

our buld setup script detects if this var is set or not.  in our case we die if it is not set.  you in yocto might choose to pick a default version.  

i would create a choose_python_exe.sh script to do that. all build scripts scripts should (must/shall) source this…
i would use the variable named YOCTO_PYTHON3_EXE

THIS SCRIPT MUST: lessons in have learned)
  remove all python directories from the PATH
   it is not as simple as putting yours first.. shit gets done in evil ways

  remove (unset) various PYTHONLIB, AND PYTHON PATH VARIABLES (there are many!)
  and undo any python virtual env varaibles too.


why do i take that extreme approach.. because these damn chip vendors often incorrectly and in a hostile way setup (read FORCE) their python version in bad ways.. it is the only approach that has worked

why?  often the young engineer at xilinix microsemi or other places has no clue about how something should be engineered to be nice. they are operating as a new college grad or are some low wage new collgege grad fresh out of school and they have an assignment from their professor (boss) they must get to work.  

sadly their leadership does not give a shit.. they only want their stuff to to work f-the customers version of python they can use ours i stead.. you remember the shit you turned in at 11:59 at night because you had 1 minute remaining before the cut off.  that is the bastard intelligence you are dealing against

so when we invoke python within our scripts we 

a) can always find the right version of python via our env variable 

b) if a venv directory does not exist in some project specific place we create one

c) we then activate that venv. and run our script

i sucks but its the only way we have gotren things to work.



our 

Sent from my iPad

> On Aug 4, 2026, at 12:50, Richard Purdie via lists.yoctoproject.org <[email protected]> wrote:
> 
> On Tue, 2026-08-04 at 20:47 +0100, Richard Purdie via lists.yoctoproject.org wrote:
>>> On Tue, 2026-08-04 at 11:49 -0700, Michael Halstead wrote:
>>> I tracked down the persistent issue on the autobuilder setup where
>>> worker builds were failing after maintenance specifically on systems
>>> like RHEL8-based hosts where we don't run the system python3.
>>> 
>>> We had `virtualenv_site_packages: true` configured across the cluster
>>> to better test distro-provided Python packages and stay aligned with
>>> current docs. Setting `virtualenv_site_packages: false` for RHEL8 and
>>> OpenSUSE 15.6 based workers resolved the issue.
>> 
>> Changing the config "virtualenv_site_packages: false" on those older
>> systems should be fine.
>> 
>>> I propose we stop using site packages on the autobuilder in general
>>> to avoid similar issues in the future. I already have the changes
>>> ready for the next maintenance window.
>>> 
>>> If we do so our documentation should probably be updated to match so
>>> developers avoid these same environment edge cases. We could
>>> update ref-manual/system-requirements.html to default to standard,
>>> isolated virtual environments and pull host Python dependencies via
>>> PyPI rather than assuming distro packages.
>>> 
>>> Let me know what you think or if there are any edge cases I haven't
>>> considered. 
>> 
>> If we go this route, we may as well drop several of the distros from
>> testing and mandate buildtools tarballs :/. I really don't want to go
>> down the route of prescribing a venv when we've gone to relative levels
>> of trouble to make bitbake easier to run on most systems.
>> 
>> I am a bit puzzled where we're running into issues as our buildtools-
>> tarball does include python3-websockets so on those older systems, we
>> should be using buildtools and that should have websockets in there?
>> 
>> What are we running outside of a buildtools env?
> 
> Just to clarify, we need a newer python on those systems for buildbot
> itself, which isn't run inside buildtools. That is where the
> requirement for the venv comes from.
> 
> I'm still puzzled about where/why we can't just use websockets from
> buildtools in our build though...
> 
> Cheers,
> 
> Richard
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#66679): https://lists.yoctoproject.org/g/yocto/message/66679
> Mute This Topic: https://lists.yoctoproject.org/mt/120599099/8932996
> Group Owner: [email protected]
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]]
> -=-=-=-=-=-=-=-=-=-=-=-
>