Re: [Fuego] [PATCH 2/4] linaro: update to python3

Hi Tim,

Thanks for your review.
See my comments inline.

> -----Original Message-----
> From: [email protected] <[email protected]>
> Sent: Friday, August 27, 2021 2:54 AM
> To: sangorrin daniel(サンゴリン ダニエル □ＳＷＣ◯ＡＣＴ) <[email protected]>
> Cc: [email protected]; nguyen dat tho(ＴＳＤＶ Ｅｎｇ　１) <[email protected]>
> Subject: RE: [PATCH 2/4] linaro: update to python3
> 
> Ok - I went to apply this today, and I realized I still have some questions...
> 
> See inline below.
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin <[email protected]>
> >
> > From: Nguyen Dat Tho <[email protected]>
> >
> > The upstream linaro repository[1] now uses python3 so use pip3 to
> > install the requirements
> >
> > [1] https://github.com/Linaro/test-definitions
> >
> > Signed-off-by: Nguyen Dat Tho <[email protected]>
> > Signed-off-by: Daniel Sangorrin <[email protected]>
> > ---
> >  tests/Functional.linaro/fuego_test.sh | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/tests/Functional.linaro/fuego_test.sh
> > b/tests/Functional.linaro/fuego_test.sh
> > index 677ab49..c775baa 100755
> > --- a/tests/Functional.linaro/fuego_test.sh
> > +++ b/tests/Functional.linaro/fuego_test.sh
> > @@ -26,8 +26,10 @@ function test_pre_check {  }
> >
> >  function test_build {
> > +    apt-get install python3-pip
> >      source ./automated/bin/setenv.sh
> > -    pip install -r $REPO_PATH/automated/utils/requirements.txt --user
> > +    pip3 install setuptools --user
> > +    pip3 install -r $REPO_PATH/automated/utils/requirements.txt
> > + --user
> >  }
> 
> What is the effective user account when this is run in Fuego non-container mode?

We normally install fuego as root and without jenkins. So it would be running with root permissions.
 
> For non-container execution, do you execute build steps as user 'fuego'
> or 'jenkins'?  

We use root. However, you can also install jenkins even when you don't use containers.

https://bitbucket.org/fuegotest/fuego/src/caf37be917f3756980879f32ad044b8f240fbd25/install-native.sh#lines-97

So in that case, it would jenkins.

> For a container environment, the build steps are performed as user 'jenkins'.  Will these steps even work, in that
> environment?

I believe they should work, but we will double check and let you know.
 
> Does this need to be 'sudo apt-get install', to handle the container case?

Probably we can use the Debian packages, we haven't tested that. Also the requirements.txt from Linaro unfortunately does not specify versions.
 
> Do the pip3 operations work at the system level, or at the level of an individual account.

I think that if you run them from jenkins it will install them using jenkins (although i am not sure what happens if jenkins does not have a $HOME folder). We run it with root so i guess they go to /root/.cache/pip or something like that.

> If a better python3-pip is needed for building, I think this should go into the installation scripts, unless this is intended to be doing
> something on the device under test, and not the host.

This was intended to run on the host for host-target configurations, and on the DUT for native configurations (eg. LAVA).

We can put them on the installation scripts, I also thought about this.
The only bad thing is that if the upstream repository changes the requirements.txt file, we might not notice.

Also, it is easier to understand the dependencies of each test if we put them there. Actually this should be handled better in general.

Should I move this to the install scripts then?

In that case, we will install the requirements without --user (or perhaps using Debian packages) .
Regarding the upstream requirements.txt file, we can ignore until the test doesnt work or we can pin the current commit id.
 
> This will have a permanent effect on the build environment, for either the container, or the host machine where Fuego is running.

Correct.

> In your test environment, is the host machine the same as the device under test, so that these changes are thrown away when the
> image is thrown away?  Or is this persistent?

We want to use the LAVA model of always testing on a pristine image. So after running all tests, the image is discarded.

> It seems like there should be a check to avoid re-installing these, if they are already present.  But maybe "apt-get install " or "pip3
> install" will just issue a warning, and there is not harm to executing these if the packages are already there.

Yes, they would only issue a warning. But if we remove test_build and move the dependencies to the install scripts, then we will not need checks at all.

> 
> Anyway, as  you can tell, I am confused by this patch.
> 
> Overall, I don't object to it.  I just want to understand what's happening with pip3 installation and setuptools installation at the user
> level and system level, for the case where:
>  - the test is executing in a Fuego container, with a remote DUT
>  - the test is executing in a Fuego container, with a local DUT (DUT=self)
>  - the text is executing natively, with a remote DUT
>  - the test is executing natively, with a local DUT (DUT=self)

Sorry for the confusion.

We will move the run dependencies to the install scripts. Then, we will test that those combinations work and what user/location is used to install the pip packages.

What do you think?

Thanks,
Daniel