[Fuego] binary packages (was RE: fuego install.sh script failed)

> -----Original Message-----
> From: [email protected] 
> Hi Tim,
> 
> Apart from the technical requirements, another thing to solve is license compliance.
> When you release binaries (eg to dockerhub) you still need to comply with the licenses.
Good point.  This reminder comes at a good time.  I've recently been working
on an option for Fuego to be used without having to install toolchains.  It involves
"test binary packages".

That is, I have support now for building "test binary packages" (with some limitations),
and for using those from local cache, if a flag in the board file says to do so.

I've got some prototype support for storing the packages in fserver.  The next
step is to add support to ftc to populate the local cache from fserver, either on install
or on demand as packages are used.  Then I should be able to build up several
caches (one per toolchain) on the main fserver site, that can be used by any
Fuego user worldwide.

This feature would allow a user to skip all the package building, and just use
pre-built binary packages instead.  This would mean someone would NOT have
to install a toolchain for their board in order to use Fuego
(for boards with "normal" architectures and distro layouts).

But I believe it raises similar license compliance issues.

> 
> Microsoft image:
> https://hub.docker.com/_/microsoft-dotnet-core (click "Discover licensing for Linux image contents")
> They explain how to list packages, licenses, copyrights from the image, and also get the source code from the original repositories (or
> snapshots).
> We should be able to reuse that information.

> 
> On the other hand, I don't know how to comply with the license of Jenkins and its plugins.
> Perhaps, we could start with the Docker image without Jenkins.

Most Jenkins plugins are licensed MIT, and are obtained from the official Jenkins
plugin repository (https://plugins.jenkins.io/description-setter/) via Jenkins itself.

Jenkins itself is installed from a Debian package, for which (presumably) there is
a source package available.  However, I checked and I couldn't find a Debian
source package on pkg.jenkins.io.  Unfortunately, it's not clear how I would
obtain one.

> 
> Maybe you want to consult the LF experts to setup a LLC (limited liability company) as well.
What would the LLC be used for?

 -- Tim

> > -----Original Message-----
> > From: Fuego <[email protected]> On Behalf Of [email protected]
> > Sent: Monday, July 27, 2020 11:14 AM
> > To: [email protected]; [email protected]; [email protected]
> > Subject: Re: [Fuego] fuego install.sh script failed
> >
> > Hi Tim,
> >
> > > -----Original Message-----
> > > From: Bird, Tim <[email protected]>
> > [...]
> > > > In that case, I think we should remove the proxy ENV variables those
> > > > can be configured in ".docker/config.json" (or via docker run
> > > > --env|e, or via docker run --env-file), and only use them as ARGs
> > > > when
> > > building the image.
> > >
> > > This is one of the issues.  The other is the set of bind mounts that might be needed for the system.
> > > In my experience, I always use the "privileged" containers, with some
> > > extra holes punched in them to access things like /dev/usb* and
> > > /dev/acm* on the host machine.  A lot of the control hardware in my lab is controlled over usb-serial.
> >
> > Bind mounts can be set by the user (or the fuego script) when they run or build the containers. They are not built into the docker image.
> >
> > > Also, I found when I made a container with bind mounts pointing to
> > > directories in my host (/fuego-ro pointing to
> > > /home/tbird/work/fuego/fuego-ro), that I couldn't change the location
> > > of the host directory without the container getting messed up.  I'm
> > > not sure how you'd do a prebuilt container with the requisite bind mounts.  You could possibly just put everything inside the container
> (but
> > then the buildzone data gets too big, and you can't access it from the host), or maybe just put things into a well-known location on the
> host
> > (maybe under, say, /opt/fuego instead of allowing Fuego to be installed anywhere.
> >
> > Again, I think that you are confusing docker images with docker containers.
> >
> > Thanks,
> > Daniel
> >
> > >
> > > If we could solve these problems, then I think it would be great if we
> > > could make prebuilt containers that people could download so they didn't have to build them themselves.
> > >  -- Tim
> > >
> > >
> > > >
> > > > ________________________________________
> > > > From: Fuego <[email protected]> on behalf of
> > > > Bird, Tim <[email protected]>
> > > > Sent: Tuesday, July 21, 2020 5:03 AM
> > > > To: Bird, Tim; seigo t; [email protected]
> > > > Subject: Re: [Fuego] fuego install.sh script failed
> > > >
> > > > One more thing - this might be the result of proxies, or it might be
> > > > a real "man-in-the-middle" attack.  If you suspect it is due to your
> > > > proxies, and not something malicious, it is OK to use the workaround
> > > > described below.  But please recognize you are reducing the security
> > > > of the container creation.
> > > >  -- Tim
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From:  Bird, Tim
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: seigo t
> > > > > >
> > > > > > Hi fuego members.
> > > > > >
> > > > > > I'm Seigo Takada work as embedded system engineer, and new to Fuego.
> > > > > >
> > > > > > I tried to install fuego, bu install.sh script failed.
> > > > > >
> > > > > > Are there any problems or something to be changed?
> > > > > >
> > > > > > I couldn't find any information on Gitlab so that I mailed this.
> > > > > > https://gitlab.com/fuegotest/fuego/issues
> > > > > >
> > > > > > <https://gitlab.com/fuegotest/fuego/issues> here are logs.
> > > > > >
> > > > > > ---
> > > > > >
> > > > > > ubuntu@ubuntu-ZBOX-EN72080V-EN72070V-EN52060V-EN51660T:~/fuego$
> > > > > > ./install.sh
> > > > > >
> > > > > > Using Port 8090
> > > > > >
> > > > > > Sending build context to Docker daemon  569.3MB
> > > > > >
> > > > > > Step 1/73 : FROM debian:stretch-slim
> > > > > >
> > > > > >  ---> 3718ed702ef5
> > > > > >
> > > > > > Step 2/73 : MAINTAINER [email protected]
> > > > > > <mailto:[email protected]>
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> ec4f5f784030
> > > > > >
> > > > > > Step 3/73 : ARG HTTP_PROXY
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 0fef569b7c05
> > > > > >
> > > > > > Step 4/73 : ENV http_proxy ${HTTP_PROXY}
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 737350a6fd26
> > > > > >
> > > > > > Step 5/73 : ENV https_proxy ${HTTP_PROXY}
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> edb1de602717
> > > > > >
> > > > > > Step 6/73 : ARG DEBIAN_FRONTEND=noninteractive
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 88ec79603c75
> > > > > >
> > > > > > Step 7/73 : WORKDIR /
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> e965db580abb
> > > > > >
> > > > > > Step 8/73 : RUN echo deb http://deb.debian.org/debian stretch
> > > > > > main non-free > /etc/apt/sources.list
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> ac317d5f1d0b
> > > > > >
> > > > > > Step 9/73 : RUN echo deb
> > > > > > http://security.debian.org/debian-security stretch/updates main
> > > > > > >> /etc/apt/sources.list
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> c801ab52cd0c
> > > > > >
> > > > > > Step 10/73 : RUN if [ -n "$HTTP_PROXY" ]; then echo
> > > > > > 'Acquire::http::proxy "'$HTTP_PROXY'";' >
> > > > > > /etc/apt/apt.conf.d/80proxy; fi
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 921383896f02
> > > > > >
> > > > > > Step 11/73 : RUN mkdir -p /usr/share/man/man1
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> da1790de599f
> > > > > >
> > > > > > Step 12/73 : RUN apt-get update -q=2 && apt-get -q=2 -V
> > > > > > --no-install-recommends install python-lxml python-simplejson
> > > > > > python-yaml python-openpyxl python-requests python-reportlab
> > > > > > python-parsedatetime python-pexpect python-pip python-setuptools
> > > > > > python-
> > > > wheel
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 1efe073da203
> > > > > >
> > > > > > Step 13/73 : RUN pip install filelock
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 042f2b38970b
> > > > > >
> > > > > > Step 14/73 : RUN apt-get -q=2 -V --no-install-recommends install
> > > > > > git sshpass openssh-client sudo net-tools wget curl lava-tool
> > > > > > bash- completion iproute2
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 905946765c0f
> > > > > >
> > > > > > Step 15/73 : RUN apt-get -q=2 -V --no-install-recommends install
> > > > > > build-essential cmake bison flex automake libtool libelf-dev
> > > > > > libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev libxmuu-dev
> > > > > > libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> d50391129368
> > > > > >
> > > > > > Step 16/73 : RUN apt-get -q=2 -V --no-install-recommends install
> > > > > > iperf iperf3 netperf bzip2 bc python-matplotlib python-xmltodict
> > > > > > netpipe-tcp iputils-ping
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 85de295ba83e
> > > > > >
> > > > > > Step 17/73 : RUN pip install flake8
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> d06145e6d6e1
> > > > > >
> > > > > > Step 18/73 : RUN apt-get -q=2 -V --no-install-recommends install     python-serial     diffstat     vim     time
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 9a2efc657e33
> > > > > >
> > > > > > Step 19/73 : RUN apt-get -q=2 -V --no-install-recommends install     genromfs
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 27cc5f6384cd
> > > > > >
> > > > > > Step 20/73 : RUN /bin/bash -c 'echo "dash dash/sh boolean false" | debconf-set-selections ; dpkg-reconfigure dash'
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 70716880fca5
> > > > > >
> > > > > > Step 21/73 : RUN if [ -n "$HTTP_PROXY" ]; then echo "use_proxy =
> > > > > > on" >> /etc/wgetrc; fi
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 4024073b0421
> > > > > >
> > > > > > Step 22/73 : RUN if [ -n "$HTTP_PROXY" ]; then echo -e
> > > > > > "http_proxy=$HTTP_PROXY\nhttps_proxy=$HTTP_PROXY" >>
> > > > /etc/environment;
> > > > > fi
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 50ef3ed94126
> > > > > >
> > > > > > Step 23/73 : ARG user=jenkins
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> b0eeac245788
> > > > > >
> > > > > > Step 24/73 : ARG group=jenkins
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 2c173e8c5d8b
> > > > > >
> > > > > > Step 25/73 : ARG uid=1000
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> c342cf370184
> > > > > >
> > > > > > Step 26/73 : ARG gid=${uid}
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> af64dc4d22ee
> > > > > >
> > > > > > Step 27/73 : ARG JENKINS_PORT=8090
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 26be13d551d4
> > > > > >
> > > > > > Step 28/73 : ARG JENKINS_VERSION=2.164.2
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> b71e06c76efe
> > > > > >
> > > > > > Step 29/73 : ARG
> > > > > > JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 31390254def1
> > > > > >
> > > > > > Step 30/73 : ARG
> > > > > > JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenkins_
> > > > > > ${
> > > > > > JENKINS_VERSION}_all.deb
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 6f64d9baa5e3
> > > > > >
> > > > > > Step 31/73 : ARG JENKINS_UC=https://updates.jenkins.io
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 99c9ee9790f2
> > > > > >
> > > > > > Step 32/73 : ARG REF=/var/lib/jenkins/plugins
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 77b70527889b
> > > > > >
> > > > > > Step 33/73 : ENV JENKINS_HOME=/var/lib/jenkins
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 0165b5fd4726
> > > > > >
> > > > > > Step 34/73 : ENV JENKINS_PORT=$JENKINS_PORT
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> dd7bca1baa28
> > > > > >
> > > > > > Step 35/73 : RUN apt-get -q=2 -V --no-install-recommends install
> > > > > > default-jdk daemon psmisc adduser procps unzip
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> dffc7862fc82
> > > > > >
> > > > > > Step 36/73 : RUN pip install python-jenkins==1.4.0
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> cc0bd5cc453d
> > > > > >
> > > > > > Step 37/73 : RUN echo -e "JENKINS_PORT=$JENKINS_PORT" >>
> > > > > > /etc/environment
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 2bcaafc3c37b
> > > > > >
> > > > > > Step 38/73 : RUN getent group ${gid} >/dev/null || groupadd -g
> > > > > > ${gid} ${group}
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> 7220f09e21b0
> > > > > >
> > > > > > Step 39/73 : RUN useradd -l -m -d "${JENKINS_HOME}" -u ${uid} -g
> > > > > > ${gid} -G sudo -s /bin/bash ${user}
> > > > > >
> > > > > >  ---> Using cache
> > > > > >
> > > > > >  ---> e5d54809a63b
> > > > > >
> > > > > > Step 40/73 : RUN wget -nv ${JENKINS_URL}
> > > > > >
> > > > > >  ---> Running in e15776c15d7a
> > > > > >
> > > > > > https://pkg.jenkins.io/debian-stable/binary/jenkins_2.164.2_all.deb:
> > > > > >
> > > > > > 2020-07-18 17:38:37 ERROR 503: certificate has expired.
> > > > > >
> > > > > > The command '/bin/sh -c wget -nv ${JENKINS_URL}' returned a
> > > > > > non-zero code: 8
> > > > > >
> > > > > > ubuntu@ubuntu-ZBOX-EN72080V-EN72070V-EN52060V-EN51660T:~/fuego$
> > > > >
> > > > > I just tried a new install here, and didn't see the problem.  Here
> > > > > is an excerpt from the output from "./install.sh --no-cache --priv"
> > > > >
> > > > > ----
> > > > > Step 39/73 : RUN useradd -l -m -d "${JENKINS_HOME}" -u ${uid} -g
> > > > > ${gid} -G sudo -s /bin/bash ${user}  ---> Running in 90305799e925
> > > > > Removing intermediate container 90305799e925  ---> 0d62007a592a
> > > > > Step
> > > > > 40/73 : RUN wget -nv ${JENKINS_URL}  ---> Running in eb39cc8589c5
> > > > > 2020-07-20 19:40:44
> > > > > URL:https://prodjenkinsreleases.blob.core.windows.net/debian-stabl
> > > > > e/ jenkins_2.164.2_all.deb [76722062/76722062] ->
> > > > > "jenkins_2.164.2_all.deb" [1] Removing intermediate container
> > > > > eb39cc8589c5  ---> c1e4485e2b0c Step 41/73 : RUN echo
> > > > > "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum -c -
> > > > > ---> Running in b5a2e14aa4a7
> > > > > jenkins_2.164.2_all.deb: OK
> > > > > Removing intermediate container b5a2e14aa4a7  ---> 74724603692d
> > > > >
> > > > > I'm not having a problem with it here.  My host is "Ubuntu 16.04.6"
> > > > > and I'm not using proxies.
> > > > >
> > > > > I would think that the wget in step 40 would use certificates from
> > > > > inside the container (which should be the same for anyone doing a
> > > > > container build), but maybe I'm wrong about that.  There is some
> > > > > difference in your host setup compared to mine that is causing the
> > > > > difference in behavior.
> > > > >
> > > > > I see from some web research that it's possible to have wget skip
> > > > > certificate checks.  See this page:
> > > > > https://stackoverflow.com/questions/9224298/how-do-i-fix-certifica
> > > > > te
> > > > > -errors-when-running-wget-on-an-https-url-in-cygwin#14218279
> > > > >
> > > > > You might try changing this line in the Dockerfile, as a workaround:
> > > > >
> > > > > from
> > > > > RUN wget -nv ${JENKINS_URL}
> > > > > to
> > > > > RUN wget --no-check-certificate ${JENKINS_URL}
> > > > >
> > > > > and try ./install.sh again.
> > > > >
> > > > > Let me know what happens.
> > > > >  -- Tim
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Fuego mailing list
> > > > > [email protected]
> > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > > > _______________________________________________
> > > > Fuego mailing list
> > > > [email protected]
> > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > _______________________________________________
> > Fuego mailing list
> > [email protected]
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego