Re: [Fuego] binary packages (was RE: fuego install.sh script failed)

Hi Tim,

> -----Original Message-----
> From: Bird, Tim <[email protected]>
> Sent: Tuesday, July 28, 2020 4:34 AM
> > Apart from the technical requirements, another thing to solve is license compliance.
> > When you release binaries (eg to dockerhub) you still need to comply with the licenses.
> Good point.  This reminder comes at a good time.  I've recently been working on an option for Fuego to be used without having to install
> toolchains.  It involves "test binary packages".
> 
> That is, I have support now for building "test binary packages" (with some limitations), and for using those from local cache, if a flag in the
> board file says to do so.
> I've got some prototype support for storing the packages in fserver.  The next step is to add support to ftc to populate the local cache from
> fserver, either on install or on demand as packages are used.  Then I should be able to build up several caches (one per toolchain) on the
> main fserver site, that can be used by any Fuego user worldwide.

What are the limitations? Do the binaries have some dependency on the libc implementation/version on the target?
Also, will you provide signatures for the binaries? It sounds a bit dangerous from the security point of view.

> This feature would allow a user to skip all the package building, and just use pre-built binary packages instead.  This would mean someone
> would NOT have to install a toolchain for their board in order to use Fuego (for boards with "normal" architectures and distro layouts).

It sounds useful for those scenarios where the user
* is not able to build the tests into the target OS image 
* is not able to install a toolchain for Fuego
* just wants to test that Fuego works quickly

> But I believe it raises similar license compliance issues.
> 
> >
> > Microsoft image:
> > https://hub.docker.com/_/microsoft-dotnet-core (click "Discover
> > licensing for Linux image contents") They explain how to list
> > packages, licenses, copyrights from the image, and also get the source code from the original repositories (or snapshots).
> > We should be able to reuse that information.
> 
> >
> > On the other hand, I don't know how to comply with the license of Jenkins and its plugins.
> > Perhaps, we could start with the Docker image without Jenkins.
> 
> Most Jenkins plugins are licensed MIT, and are obtained from the official Jenkins plugin repository (https://plugins.jenkins.io/description-
> setter/) via Jenkins itself.
> 
> Jenkins itself is installed from a Debian package, for which (presumably) there is a source package available.  However, I checked and I
> couldn't find a Debian source package on pkg.jenkins.io.  Unfortunately, it's not clear how I would obtain one.

It seems that Jenkins is also MIT. Since we are using the binaries as they are, I think we don't need to do anything.
https://tldrlegal.com/license/mit-license
https://choosealicense.com/licenses/mit/

> > Maybe you want to consult the LF experts to setup a LLC (limited liability company) as well.
> What would the LLC be used for?

Well, maybe this is an exaggeration for Fuego but I am not an expert. Some LF expert may say you need an LLC to protect you against license compliance mistakes. But again I'm not an expert.

Thanks,
Daniel

> 
>  -- Tim
> 
> > > -----Original Message-----
> > > From: Fuego <[email protected]> On Behalf Of
> > > [email protected]
> > > Sent: Monday, July 27, 2020 11:14 AM
> > > To: [email protected]; [email protected];
> > > [email protected]
> > > Subject: Re: [Fuego] fuego install.sh script failed
> > >
> > > Hi Tim,
> > >
> > > > -----Original Message-----
> > > > From: Bird, Tim <[email protected]>
> > > [...]
> > > > > In that case, I think we should remove the proxy ENV variables
> > > > > those can be configured in ".docker/config.json" (or via docker
> > > > > run
> > > > > --env|e, or via docker run --env-file), and only use them as
> > > > > --env|ARGs
> > > > > when
> > > > building the image.
> > > >
> > > > This is one of the issues.  The other is the set of bind mounts that might be needed for the system.
> > > > In my experience, I always use the "privileged" containers, with
> > > > some extra holes punched in them to access things like /dev/usb*
> > > > and
> > > > /dev/acm* on the host machine.  A lot of the control hardware in my lab is controlled over usb-serial.
> > >
> > > Bind mounts can be set by the user (or the fuego script) when they run or build the containers. They are not built into the docker image.
> > >
> > > > Also, I found when I made a container with bind mounts pointing to
> > > > directories in my host (/fuego-ro pointing to
> > > > /home/tbird/work/fuego/fuego-ro), that I couldn't change the
> > > > location of the host directory without the container getting
> > > > messed up.  I'm not sure how you'd do a prebuilt container with
> > > > the requisite bind mounts.  You could possibly just put everything
> > > > inside the container
> > (but
> > > then the buildzone data gets too big, and you can't access it from
> > > the host), or maybe just put things into a well-known location on
> > > the
> > host
> > > (maybe under, say, /opt/fuego instead of allowing Fuego to be installed anywhere.
> > >
> > > Again, I think that you are confusing docker images with docker containers.
> > >
> > > Thanks,
> > > Daniel
> > >
> > > >
> > > > If we could solve these problems, then I think it would be great
> > > > if we could make prebuilt containers that people could download so they didn't have to build them themselves.
> > > >  -- Tim
> > > >
> > > >
> > > > >
> > > > > ________________________________________
> > > > > From: Fuego <[email protected]> on behalf
> > > > > of Bird, Tim <[email protected]>
> > > > > Sent: Tuesday, July 21, 2020 5:03 AM
> > > > > To: Bird, Tim; seigo t; [email protected]
> > > > > Subject: Re: [Fuego] fuego install.sh script failed
> > > > >
> > > > > One more thing - this might be the result of proxies, or it
> > > > > might be a real "man-in-the-middle" attack.  If you suspect it
> > > > > is due to your proxies, and not something malicious, it is OK to
> > > > > use the workaround described below.  But please recognize you
> > > > > are reducing the security of the container creation.
> > > > >  -- Tim
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From:  Bird, Tim
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: seigo t
> > > > > > >
> > > > > > > Hi fuego members.
> > > > > > >
> > > > > > > I'm Seigo Takada work as embedded system engineer, and new to Fuego.
> > > > > > >
> > > > > > > I tried to install fuego, bu install.sh script failed.
> > > > > > >
> > > > > > > Are there any problems or something to be changed?
> > > > > > >
> > > > > > > I couldn't find any information on Gitlab so that I mailed this.
> > > > > > > https://gitlab.com/fuegotest/fuego/issues
> > > > > > >
> > > > > > > <https://gitlab.com/fuegotest/fuego/issues> here are logs.
> > > > > > >
> > > > > > > ---
> > > > > > >
> > > > > > > ubuntu@ubuntu-ZBOX-EN72080V-EN72070V-EN52060V-EN51660T:~/fue
> > > > > > > go$
> > > > > > > ./install.sh
> > > > > > >
> > > > > > > Using Port 8090
> > > > > > >
> > > > > > > Sending build context to Docker daemon  569.3MB
> > > > > > >
> > > > > > > Step 1/73 : FROM debian:stretch-slim
> > > > > > >
> > > > > > >  ---> 3718ed702ef5
> > > > > > >
> > > > > > > Step 2/73 : MAINTAINER [email protected]
> > > > > > > <mailto:[email protected]>
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> ec4f5f784030
> > > > > > >
> > > > > > > Step 3/73 : ARG HTTP_PROXY
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 0fef569b7c05
> > > > > > >
> > > > > > > Step 4/73 : ENV http_proxy ${HTTP_PROXY}
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 737350a6fd26
> > > > > > >
> > > > > > > Step 5/73 : ENV https_proxy ${HTTP_PROXY}
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> edb1de602717
> > > > > > >
> > > > > > > Step 6/73 : ARG DEBIAN_FRONTEND=noninteractive
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 88ec79603c75
> > > > > > >
> > > > > > > Step 7/73 : WORKDIR /
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> e965db580abb
> > > > > > >
> > > > > > > Step 8/73 : RUN echo deb http://deb.debian.org/debian
> > > > > > > stretch main non-free > /etc/apt/sources.list
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> ac317d5f1d0b
> > > > > > >
> > > > > > > Step 9/73 : RUN echo deb
> > > > > > > http://security.debian.org/debian-security stretch/updates
> > > > > > > main
> > > > > > > >> /etc/apt/sources.list
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> c801ab52cd0c
> > > > > > >
> > > > > > > Step 10/73 : RUN if [ -n "$HTTP_PROXY" ]; then echo
> > > > > > > 'Acquire::http::proxy "'$HTTP_PROXY'";' >
> > > > > > > /etc/apt/apt.conf.d/80proxy; fi
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 921383896f02
> > > > > > >
> > > > > > > Step 11/73 : RUN mkdir -p /usr/share/man/man1
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> da1790de599f
> > > > > > >
> > > > > > > Step 12/73 : RUN apt-get update -q=2 && apt-get -q=2 -V
> > > > > > > --no-install-recommends install python-lxml
> > > > > > > python-simplejson python-yaml python-openpyxl
> > > > > > > python-requests python-reportlab python-parsedatetime
> > > > > > > python-pexpect python-pip python-setuptools
> > > > > > > python-
> > > > > wheel
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 1efe073da203
> > > > > > >
> > > > > > > Step 13/73 : RUN pip install filelock
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 042f2b38970b
> > > > > > >
> > > > > > > Step 14/73 : RUN apt-get -q=2 -V --no-install-recommends
> > > > > > > install git sshpass openssh-client sudo net-tools wget curl
> > > > > > > lava-tool
> > > > > > > bash- completion iproute2
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 905946765c0f
> > > > > > >
> > > > > > > Step 15/73 : RUN apt-get -q=2 -V --no-install-recommends
> > > > > > > install build-essential cmake bison flex automake libtool
> > > > > > > libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev
> > > > > > > libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync
> > > > > > > u-boot-tools
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> d50391129368
> > > > > > >
> > > > > > > Step 16/73 : RUN apt-get -q=2 -V --no-install-recommends
> > > > > > > install iperf iperf3 netperf bzip2 bc python-matplotlib
> > > > > > > python-xmltodict netpipe-tcp iputils-ping
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 85de295ba83e
> > > > > > >
> > > > > > > Step 17/73 : RUN pip install flake8
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> d06145e6d6e1
> > > > > > >
> > > > > > > Step 18/73 : RUN apt-get -q=2 -V --no-install-recommends install     python-serial     diffstat     vim     time
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 9a2efc657e33
> > > > > > >
> > > > > > > Step 19/73 : RUN apt-get -q=2 -V --no-install-recommends install     genromfs
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 27cc5f6384cd
> > > > > > >
> > > > > > > Step 20/73 : RUN /bin/bash -c 'echo "dash dash/sh boolean false" | debconf-set-selections ; dpkg-reconfigure dash'
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 70716880fca5
> > > > > > >
> > > > > > > Step 21/73 : RUN if [ -n "$HTTP_PROXY" ]; then echo
> > > > > > > "use_proxy = on" >> /etc/wgetrc; fi
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 4024073b0421
> > > > > > >
> > > > > > > Step 22/73 : RUN if [ -n "$HTTP_PROXY" ]; then echo -e
> > > > > > > "http_proxy=$HTTP_PROXY\nhttps_proxy=$HTTP_PROXY" >>
> > > > > /etc/environment;
> > > > > > fi
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 50ef3ed94126
> > > > > > >
> > > > > > > Step 23/73 : ARG user=jenkins
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> b0eeac245788
> > > > > > >
> > > > > > > Step 24/73 : ARG group=jenkins
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 2c173e8c5d8b
> > > > > > >
> > > > > > > Step 25/73 : ARG uid=1000
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> c342cf370184
> > > > > > >
> > > > > > > Step 26/73 : ARG gid=${uid}
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> af64dc4d22ee
> > > > > > >
> > > > > > > Step 27/73 : ARG JENKINS_PORT=8090
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 26be13d551d4
> > > > > > >
> > > > > > > Step 28/73 : ARG JENKINS_VERSION=2.164.2
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> b71e06c76efe
> > > > > > >
> > > > > > > Step 29/73 : ARG
> > > > > > > JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 31390254def1
> > > > > > >
> > > > > > > Step 30/73 : ARG
> > > > > > > JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenk
> > > > > > > ins_
> > > > > > > ${
> > > > > > > JENKINS_VERSION}_all.deb
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 6f64d9baa5e3
> > > > > > >
> > > > > > > Step 31/73 : ARG JENKINS_UC=https://updates.jenkins.io
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 99c9ee9790f2
> > > > > > >
> > > > > > > Step 32/73 : ARG REF=/var/lib/jenkins/plugins
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 77b70527889b
> > > > > > >
> > > > > > > Step 33/73 : ENV JENKINS_HOME=/var/lib/jenkins
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 0165b5fd4726
> > > > > > >
> > > > > > > Step 34/73 : ENV JENKINS_PORT=$JENKINS_PORT
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> dd7bca1baa28
> > > > > > >
> > > > > > > Step 35/73 : RUN apt-get -q=2 -V --no-install-recommends
> > > > > > > install default-jdk daemon psmisc adduser procps unzip
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> dffc7862fc82
> > > > > > >
> > > > > > > Step 36/73 : RUN pip install python-jenkins==1.4.0
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> cc0bd5cc453d
> > > > > > >
> > > > > > > Step 37/73 : RUN echo -e "JENKINS_PORT=$JENKINS_PORT" >>
> > > > > > > /etc/environment
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 2bcaafc3c37b
> > > > > > >
> > > > > > > Step 38/73 : RUN getent group ${gid} >/dev/null || groupadd
> > > > > > > -g ${gid} ${group}
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> 7220f09e21b0
> > > > > > >
> > > > > > > Step 39/73 : RUN useradd -l -m -d "${JENKINS_HOME}" -u
> > > > > > > ${uid} -g ${gid} -G sudo -s /bin/bash ${user}
> > > > > > >
> > > > > > >  ---> Using cache
> > > > > > >
> > > > > > >  ---> e5d54809a63b
> > > > > > >
> > > > > > > Step 40/73 : RUN wget -nv ${JENKINS_URL}
> > > > > > >
> > > > > > >  ---> Running in e15776c15d7a
> > > > > > >
> > > > > > > https://pkg.jenkins.io/debian-stable/binary/jenkins_2.164.2_all.deb:
> > > > > > >
> > > > > > > 2020-07-18 17:38:37 ERROR 503: certificate has expired.
> > > > > > >
> > > > > > > The command '/bin/sh -c wget -nv ${JENKINS_URL}' returned a
> > > > > > > non-zero code: 8
> > > > > > >
> > > > > > > ubuntu@ubuntu-ZBOX-EN72080V-EN72070V-EN52060V-EN51660T:~/fue
> > > > > > > go$
> > > > > >
> > > > > > I just tried a new install here, and didn't see the problem.
> > > > > > Here is an excerpt from the output from "./install.sh --no-cache --priv"
> > > > > >
> > > > > > ----
> > > > > > Step 39/73 : RUN useradd -l -m -d "${JENKINS_HOME}" -u ${uid}
> > > > > > -g ${gid} -G sudo -s /bin/bash ${user}  ---> Running in
> > > > > > 90305799e925 Removing intermediate container 90305799e925
> > > > > > ---> 0d62007a592a Step
> > > > > > 40/73 : RUN wget -nv ${JENKINS_URL}  ---> Running in
> > > > > > eb39cc8589c5
> > > > > > 2020-07-20 19:40:44
> > > > > > URL:https://prodjenkinsreleases.blob.core.windows.net/debian-s
> > > > > > tabl e/ jenkins_2.164.2_all.deb [76722062/76722062] ->
> > > > > > "jenkins_2.164.2_all.deb" [1] Removing intermediate container
> > > > > > eb39cc8589c5  ---> c1e4485e2b0c Step 41/73 : RUN echo
> > > > > > "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum
> > > > > > -c -
> > > > > > ---> Running in b5a2e14aa4a7
> > > > > > jenkins_2.164.2_all.deb: OK
> > > > > > Removing intermediate container b5a2e14aa4a7  --->
> > > > > > 74724603692d
> > > > > >
> > > > > > I'm not having a problem with it here.  My host is "Ubuntu 16.04.6"
> > > > > > and I'm not using proxies.
> > > > > >
> > > > > > I would think that the wget in step 40 would use certificates
> > > > > > from inside the container (which should be the same for anyone
> > > > > > doing a container build), but maybe I'm wrong about that.
> > > > > > There is some difference in your host setup compared to mine
> > > > > > that is causing the difference in behavior.
> > > > > >
> > > > > > I see from some web research that it's possible to have wget
> > > > > > skip certificate checks.  See this page:
> > > > > > https://stackoverflow.com/questions/9224298/how-do-i-fix-certi
> > > > > > fica
> > > > > > te
> > > > > > -errors-when-running-wget-on-an-https-url-in-cygwin#14218279
> > > > > >
> > > > > > You might try changing this line in the Dockerfile, as a workaround:
> > > > > >
> > > > > > from
> > > > > > RUN wget -nv ${JENKINS_URL}
> > > > > > to
> > > > > > RUN wget --no-check-certificate ${JENKINS_URL}
> > > > > >
> > > > > > and try ./install.sh again.
> > > > > >
> > > > > > Let me know what happens.
> > > > > >  -- Tim
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Fuego mailing list
> > > > > > [email protected]
> > > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > > > > _______________________________________________
> > > > > Fuego mailing list
> > > > > [email protected]
> > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > > _______________________________________________
> > > Fuego mailing list
> > > [email protected]
> > > https://lists.linuxfoundation.org/mailman/listinfo/fuego