Re: [Fuego] Use common script to install fuego on host or docker system
<[email protected]> Wed, 16 Jun 2021 17:47:59 +0000
| Newsgroups | dev.linux.lists.fuego |
|---|---|
| Message-ID | <BYAPR13MB25033AF87D4435DFFE124EA4FD0F9@BYAPR13MB2503.namprd13.prod.outlook.com> |
Venkata, Did you have any responses or comments on my other feedback for the patch? -- Tim > -----Original Message----- > From: [email protected] <[email protected]> > > Hello Tim, > > I sent patch based on the master branch of this repository https://bitbucket.org/fuegotest/fuego/src/master/ > And the previous commit id was "3c813c717f1ebd7dc279ecdb0f288e4f9d30d6d7". > > Also created a PR (https://bitbucket.org/fuegotest/fuego/pull-requests/4) to visualize the changes. > > Please let me know if I need to send another patch with any specific branch or version? > > > Thanks, > Venkata. > -----Original Message----- > From: [email protected] <[email protected]> > Sent: 15 June 2021 05:39 > To: pyla venkata(TSIP) <[email protected]>; [email protected] > Cc: [email protected]; sangorrin daniel(サンゴリン ダニエル □SWC◯ACT) <[email protected]> > Subject: RE: [Fuego] Use common script to install fuego on host or docker system > > Please see comments inline below. > > I tried to apply this patch from the e-mail list using 'git am', but got a failure. > That made it a hard to experiment with and view the changes in > context. > > What version of the code is this patch based on? > > > -----Original Message----- > > From: [email protected] <[email protected]> > > Subject: [Fuego] Use common script to install fuego on host or docker system > > > > From: venkata pyla <[email protected]> > > > > Modified the install scripts to use common scripts for both types of > > installation, this way it will help to make sure both the environments > > are same and if any future modification will have single place of change. > > > > Now, for installing fuego on docker system the 'install.sh' script will > > use common script in Dockefile to install fuego related stuff. > > > > sameway for installing fuego on host system it can use the same common > > script to install fuego related stuff. > > > > Below are the common scripts that are used in both the installation types > > for each debian distribution > > * install-stretch.sh > > * install-buster.sh > > > > so finally, > > To install fuego on docker system use the below script with options > > $ ./install.sh <options> image_name <port> <distribution> <suite> > > e.g: ./install.sh debian_stretch 8090 debian stretch > > or > > ./install.sh debian_buster 8090 debian buster > > or > > ./install.sh --nojenkins debian_buster_nojenkins 8090 debian buster > This is a bit wordy, but OK. > > If we don't support any other distributions besides Debian at the moment, > then I'm not sure we need the distribution argument. Do you have any > non-debian distribution install scripts ready to submit soon? > > I'm not sure I follow calling the Debian release name the "<suite>". > > > > > To install fuego on host system use the below script with options > > $ ./install-fuego.sh <options> <-d distribution> <-s suite> <-p port> > "suite" seems like an odd name. Should this be distro "version" or "release"? > > > e.g: ./install-fuego.sh -d debian -s stretch -p 8090 > > or > > ./install-fuego.sh -d debian -s buster -p 8090 > > or > > ./install-fuego.sh -d debian -s buster -p 8090 --nojenkins > > > > Signed-off-by: venkata pyla <[email protected]> > > --- > > Dockerfile | 303 ++--------------- > > Dockerfile.nojenkins | 151 --------- > > README | 29 +- > > entrypoint.sh | 7 + > > fuego-host-scripts/docker-build-image.sh | 11 +- > > install-buster.sh | 294 ++++++++++++++++ > > install-debian.sh | 410 ++++++++++++++--------- > > install-fuego.sh | 109 ++++++ > > install-stretch.sh | 288 ++++++++++++++++ > > Is install-debian.sh the common parts for a Debian installation, with > buster and stretch having the parts that are unique to each of > those distros respectively? If we're going to refactor this, I > want to eliminate any duplication, so that package names are > not used in multiple places. This has been a headache to > maintain. > > It appears there is lots of duplication between install-buster.sh and > install-stretch.sh. Can we possible put this duplication into an > "install-debian-common.sh", and reserve install-buster.sh and > install-stretch.sh for only the parts unique to those distros? > > > install.sh | 12 +- > > uninstall-fuego.sh | 47 +++ > > Ooo - I like the sound of this uninstall script. > > > 11 files changed, 1057 insertions(+), 604 deletions(-) > > delete mode 100644 Dockerfile.nojenkins > > create mode 100755 entrypoint.sh > > create mode 100755 install-buster.sh > > create mode 100755 install-fuego.sh > > create mode 100755 install-stretch.sh > > create mode 100644 uninstall-fuego.sh > > > > > > diff --git a/Dockerfile b/Dockerfile > > index 7f63798..ac57cb2 100644 > > --- a/Dockerfile > > +++ b/Dockerfile > > @@ -2,9 +2,11 @@ > > # WARNING: this Dockerfile assumes that the container will be created with > > # several bind mounts (see docker-create-container.sh) > > # ============================================================================== > > -# FIXTHIS: build this as an extension of the nonjenkins image > > > > -FROM debian:stretch-slim > > +ARG SUITE=stretch > > +ARG DIST=debian > > + > > +FROM ${DIST}:${SUITE}-slim > > MAINTAINER [email protected] > > > > # ============================================================================== > > @@ -16,299 +18,32 @@ ENV http_proxy ${HTTP_PROXY} > > ENV https_proxy ${HTTP_PROXY} > > > > # ============================================================================== > > -# Prepare basic image > > +# Install fuego with the script > > # ============================================================================== > > > > -ARG DEBIAN_FRONTEND=noninteractive > > - > > -WORKDIR / > > -RUN echo deb http://deb.debian.org/debian stretch main non-free > /etc/apt/sources.list > > -RUN echo deb http://security.debian.org/debian-security stretch/updates main >> /etc/apt/sources.list > > -RUN if [ -n "$HTTP_PROXY" ]; then echo 'Acquire::http::proxy "'$HTTP_PROXY'";' > /etc/apt/apt.conf.d/80proxy; fi > > - > > -# Fuego python dependencies > > -# - python-lxml: ftc, loggen > > -# - python-simplejson: ftc > > -# - python-yaml: ftc > > -# - python-openpyxl: ftc (also LTP) > > -# - python-requests: ftc (also fuego_release_test) > > -# - python-reportlab: ftc > > -# - python-parsedatetime: ftc > > -# - python-pexpect: ssh_exec (part of ttc) > > -# - python-pip: to install filelock, flake8 > > -# - filelock: parser > > -RUN mkdir -p /usr/share/man/man1 > > -RUN apt-get update -q=2 && apt-get -q=2 -V --no-install-recommends install \ > > - python-lxml python-simplejson python-yaml python-openpyxl \ > > - python-requests python-reportlab python-parsedatetime \ > > - python-pexpect python-pip python-setuptools python-wheel > > -RUN pip install filelock > > - > > -# Fuego command dependencies > > -RUN apt-get -q=2 -V --no-install-recommends install \ > > - git sshpass openssh-client sudo net-tools wget curl lava-tool \ > > - bash-completion iproute2 > > - > > -# Default SDK for testing locally or on an x86 board > > -RUN apt-get -q=2 -V --no-install-recommends install \ > > - build-essential cmake bison flex automake kmod libtool \ > > - libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \ > > - libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools > > - > > -# Default test host dependencies > > -# - iperf iperf3 netperf: used as servers on the host > > -# - bzip2 bc: used for local tests by Functional.bzip2/bc > > -# - python-matplotlib: Benchmark.iperf3 parser > > -# - python-xmltodict: AGL tests > > -# - flake8: Functional.fuego_lint > > -# - netpipe-tcp - used by Benchmark.netpipe (provides the netpipe server) > > -# - iputils-ping - for /bin/ping command, used by some tests > > -# FIXTHIS: install dependencies dynamically on the tests that need them > > -RUN apt-get -q=2 -V --no-install-recommends install \ > > - iperf iperf3 netperf bzip2 bc python-matplotlib python-xmltodict \ > > - netpipe-tcp iputils-ping > > -RUN pip install flake8 > > - > > -# miscelaneous packages: > > -# python-serial - used by serio > > -# diffstat and vim - used by Tim > > -# time - useful for timing command duration > > -RUN apt-get -q=2 -V --no-install-recommends install \ > > - python-serial \ > > - diffstat \ > > - vim \ > > - time > > - > > -# install packages used by NuttX SDK > > -RUN apt-get -q=2 -V --no-install-recommends install \ > > - genromfs > > - > > -# FIXTHIS: determine if these tools are really necessary > > -#RUN apt-get -q=2 -V --no-install-recommends install \ > > -# apt-utils python-paramiko \ > > -# xmlstarlet rsync \ > > -# inotify-tools gettext netpipe-tcp \ > > -# at minicom lzop bsdmainutils \ > > -# mc netcat openssh-server > > - > > -# bsdmainutils provides hexdump, which is needed for netperf test > > -RUN apt-get -q=2 -V --no-install-recommends install \ > > - bsdmainutils > > - > > -RUN /bin/bash -c 'echo "dash dash/sh boolean false" | debconf-set-selections ; dpkg-reconfigure dash' > > -RUN if [ -n "$HTTP_PROXY" ]; then echo "use_proxy = on" >> /etc/wgetrc; fi > > -RUN if [ -n "$HTTP_PROXY" ]; then echo -e "http_proxy=$HTTP_PROXY\nhttps_proxy=$HTTP_PROXY" >> /etc/environment; fi > > - > > -# ============================================================================== > > -# Install Jenkins with the same UID/GID as the host user > > -# ============================================================================== > > +RUN mkdir /fuego > > +COPY . /fuego/ > > +WORKDIR /fuego > > > > -ARG user=jenkins > > -ARG group=jenkins > > -ARG uid=1000 > > -ARG gid=${uid} > > ARG JENKINS_PORT=8090 > > -# see https://updates.jenkins.io/download/war/ for latest Jenkins versions > > -#ARG JENKINS_VERSION=2.164.2 > > -#ARG JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab > > -#ARG JENKINS_VERSION=2.235.3 > > -#ARG JENKINS_SHA=89a37ed1b3c8c14ff94cc66c3f2254f281b60794 > > -ARG JENKINS_VERSION=2.249.3 > > -ARG JENKINS_SHA=534014c007edbb533a1833fe6f2dc115faf3faa2 > > - > > -ARG JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenkins_${JENKINS_VERSION}_all.deb > > -ARG JENKINS_UC=https://updates.jenkins.io > > -ARG REF=/var/lib/jenkins/plugins > > +ARG NO_JENKINS= > > +ARG SUITE=stretch > > +ARG DIST=debian > > +ARG uid= > > +ARG gid= > > ENV JENKINS_HOME=/var/lib/jenkins > > ENV JENKINS_PORT=$JENKINS_PORT > > +RUN ./install-$DIST.sh -s $SUITE -p $JENKINS_PORT -u $uid -g $gid $NO_JENKINS > > One question about this overall approach. Does this retain the ability to cache > operations, so that if something goes wrong with an install, you don't have to > start over from scratch. When building with docker, with multiple steps, it > caches the image at each step. When rebuilding, or when an error occurs, > this can save a lot of time. It also can introduce subtle bugs, but that's > a separate issue. > > This setup seems like it performs the bulk of the install as a single docker > step, is that right? > > > -# Jenkins dependencies > > -RUN apt-get -q=2 -V --no-install-recommends install \ > > - default-jdk daemon psmisc adduser procps unzip > > -RUN pip install python-jenkins==1.4.0 > > - > > -RUN echo -e "JENKINS_PORT=$JENKINS_PORT" >> /etc/environment > > -RUN getent group ${gid} >/dev/null || groupadd -g ${gid} ${group} > > -RUN useradd -l -m -d "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user} > > -RUN wget -nv ${JENKINS_URL} > > -RUN echo "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum -c - > > -# allow Jenkins to start and install plugins, as part of dpkg installation > > -RUN printf "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d > > -RUN dpkg -i jenkins_${JENKINS_VERSION}_all.deb > > -RUN rm jenkins_${JENKINS_VERSION}_all.deb > > - > > -# ============================================================================== > > -# get ttc script and helpers > > -# ============================================================================== > > -RUN git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc > > -RUN /usr/local/src/ttc/install.sh /usr/local/bin > > -# as of ttc version 2.0.2, this should not be needed > > -#RUN perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc > > - > > -# ============================================================================== > > -# get ebf script and helpers > > -# ============================================================================== > > -RUN apt-get -q=2 -V --no-install-recommends install jq > > -RUN git clone https://github.com/TimesysGit/board-farm-rest-api.git /usr/local/src/board-farm-rest-api ; \ > > - cp /usr/local/src/board-farm-rest-api/cli/ebf /usr/local/bin/ebf ; \ > > - chmod a+x /usr/local/bin/ebf > > - > > -# ============================================================================== > > -# Serial Config > > -# ============================================================================== > > - > > -RUN git clone https://github.com/frowand/serio.git /usr/local/src/serio > > -COPY frontend-install/0001-Fix-host-parsing-for-serial-device-with-in-name.patch \ > > - frontend-install/0002-Output-data-from-port-during-command-execution.patch \ > > - /tmp/ > > -RUN /bin/bash -c 'patch -d /usr/local/src/serio -p1 </tmp/0001-Fix-host-parsing-for-serial-device-with-in-name.patch ; \ > > - patch -d /usr/local/src/serio -p1 </tmp/0002-Output-data-from-port-during-command-execution.patch ; \ > > - chown -R jenkins /usr/local/src/serio ; \ > > - cp /usr/local/src/serio/serio /usr/local/bin/ ; \ > > - ln -s /usr/local/bin/serio /usr/local/bin/sercp ; \ > > - ln -s /usr/local/bin/serio /usr/local/bin/sersh' > > - > > -RUN /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; chown -R jenkins /usr/local/src/serlogin ; > cp > > /usr/local/src/serlogin/serlogin /usr/local/bin' > > - > > -# ============================================================================== > > -# fserver > > -# ============================================================================== > > - > > -RUN /bin/bash -c 'git clone https://github.com/tbird20d/fserver.git /usr/local/lib/fserver ; \ > > - ln -s /usr/local/lib/fserver/start_local_bg_server /usr/local/bin/start_local_bg_server' > > - > > -# ============================================================================== > > -# Jenkins post installation > > -# ============================================================================== > > - > > -RUN source /etc/default/jenkins && \ > > - JENKINS_ARGS="$JENKINS_ARGS --prefix=/fuego" && \ > > - sed -i -e "s#JENKINS_ARGS.*#JENKINS_ARGS\=\"${JENKINS_ARGS}\"#g" /etc/default/jenkins > > - > > -RUN source /etc/default/jenkins && \ > > - JAVA_ARGS="$JAVA_ARGS -Djenkins.install.runSetupWizard=false - > > Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true" && \ > > - if [ -n "$HTTP_PROXY" ]; then \ > > - PROXYSERVER=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\1/') && \ > > - PROXYPORT=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\2/') && \ > > - JAVA_ARGS="$JAVA_ARGS -Dhttp.proxyHost="${PROXYSERVER}" -Dhttp.proxyPort="${PROXYPORT}" - > > Dhttps.proxyHost="${PROXYSERVER}" -Dhttps.proxyPort="${PROXYPORT}; \ > > - fi && \ > > - sed -i -e "s#^JAVA_ARGS.*#JAVA_ARGS\=\"${JAVA_ARGS}\"#g" /etc/default/jenkins; > > - > > -RUN sed -i -e "s#8080#$JENKINS_PORT#g" /etc/default/jenkins > > - > > -# set up Jenkins plugins > > -COPY frontend-install/install-plugins.sh \ > > - frontend-install/jenkins-support \ > > - frontend-install/clitest \ > > - /usr/local/bin/ > > - > > -# start Jenkins onces to initialize directories? > > -RUN service jenkins start && \ > > - sleep 30 && \ > > - service jenkins stop > > - > > -# install plugins - these versions are for Jenkins version 2.164.2 > > -# Explicitly install script-security v1.68, otherwise > > -# v1.74 will automatically be installed as a dependency of > > -# the junit plugin. Make sure to install before junit plugin. > > -# Explicitly install junit:1.27 > > - > > -# install other plugins from Jenkins update center > > -# NOTE: not sure all of these are needed, but keep list > > -# as compatible as possible with 1.2.1 release for now > > -# Do NOT change the order of the plugins, unless you know what > > -# you are doing. Otherwise, the install-plugins.sh script will > > -# install the wrong versions of other plugins based on dependency > > -# information in the plugin files. > > - > > -#RUN /usr/local/bin/install-plugins.sh \ > > -# script-security:1.68 \ > > -# structs:1.20 \ > > -# workflow-step-api:2.22 \ > > -# workflow-api:2.40 \ > > -# junit:1.27 \ > > -# scm-api:2.6.3 \ > > -# ant:1.9 \ > > -# antisamy-markup-formatter:1.5 \ > > -# bouncycastle-api:2.17 \ > > -# command-launcher:1.3 \ > > -# description-setter:1.10 \ > > -# display-url-api:2.3.1 \ > > -# external-monitor-job:1.7 \ > > -# greenballs:1.15 \ > > -# icon-shim:2.0.3 \ > > -# javadoc:1.5 \ > > -# jdk-tool:1.2 \ > > -# ldap:1.20 \ > > -# mailer:1.23 \ > > -# matrix-auth:2.3 \ > > -# matrix-project:1.14 \ > > -# pam-auth:1.5 \ > > -# pegdown-formatter:1.3 \ > > -# windows-slaves:1.4 > > - > > -# install plugins for latest LTS Jenkins (currently 2.249.3) > > -RUN /usr/local/bin/install-plugins.sh \ > > - script-security \ > > - structs \ > > - workflow-step-api \ > > - workflow-api \ > > - junit \ > > - scm-api \ > > - ant \ > > - antisamy-markup-formatter \ > > - bouncycastle-api \ > > - command-launcher \ > > - description-setter \ > > - display-url-api \ > > - external-monitor-job \ > > - greenballs \ > > - icon-shim \ > > - javadoc \ > > - jdk-tool \ > > - mailer \ > > - matrix-auth \ > > - matrix-project \ > > - pam-auth \ > > - pegdown-formatter \ > > - windows-slaves > > +COPY entrypoint.sh /usr/bin/ > > > > -COPY frontend-install/plugins/flot-plotter-plugin/flot.hpi $JENKINS_HOME/plugins/flot.jpi > > - > > -# jenkins should automatically unzip any plugins in the plugin dir > > -# make a symlink for mod.js well after flot is installed > > -RUN service jenkins start && sleep 30 && \ > > - rm $JENKINS_HOME/plugins/flot/flot/mod.js && \ > > - ln -s /fuego-core/scripts/mod.js $JENKINS_HOME/plugins/flot/flot/mod.js > > - > > -RUN ln -s /fuego-rw/logs $JENKINS_HOME/userContent/fuego.logs > > -COPY docs/fuego-docs.pdf $JENKINS_HOME/userContent/docs/fuego-docs.pdf > > - > > -COPY frontend-install/config.xml $JENKINS_HOME/config.xml > > -COPY frontend-install/jenkins.model.JenkinsLocationConfiguration.xml > $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml > > -RUN sed -i -e "s#8080#$JENKINS_PORT#g" $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml > > - > > -RUN chown -R jenkins:jenkins $JENKINS_HOME/ > > - > > -# ============================================================================== > > -# ftc post installation > > -# ============================================================================== > > - > > -RUN ln -s /fuego-core/scripts/ftc /usr/local/bin/ > > -COPY fuego-core/scripts/ftc_completion.sh /etc/bash_completion.d/ftc > > -RUN echo ". /etc/bash_completion" >> /root/.bashrc > > - > > -# ============================================================================== > > -# Lava > > -# ============================================================================== > > - > > -RUN ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin > > -RUN ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin > > +# Delete the files that are copied for installation > > +RUN rm -rf /fuego > > I'm not sure what this is doing. It looks scary. Please explain. > > > +WORKDIR / > > > > # ============================================================================== > > # Setup startup command > > # ============================================================================== > > - > > # FIXTHIS: when running multiple Fuego containers, or if the host is already > > # running the netperf server, netperf will complain because the port is taken > > -ENTRYPOINT service jenkins start && service netperf start && iperf3 -V -s -D -f M && /bin/bash > > +ENTRYPOINT entrypoint.sh $NO_JENKINS && /bin/bash > > Isolating this to a separate script is nice! > > > diff --git a/Dockerfile.nojenkins b/Dockerfile.nojenkins > > deleted file mode 100644 > > index 56de98a..0000000 > > --- a/Dockerfile.nojenkins > > +++ /dev/null > > Getting rid of this separate Dockerfile is nice. > > > @@ -1,151 +0,0 @@ > > -# ============================================================================== > > -# WARNING: this Dockerfile assumes that the container will be created with > > -# several bind mounts (see docker-create-container.sh) > > -# ============================================================================== > > -# Usage: > > -# host$ vi fuego-ro/conf/fuego.conf > > -# jenkins_enabled=0 > > -# host$ ./fuego-host-scripts/docker-build-image.sh fuego-nojenkins 8090 Dockerfile.nojenkins > > -# host$ ./fuego-host-scripts/docker-create-container.sh fuego-nojenkins fuego-nojenkins-container > > -# host$ ./fuego-host-scripts/docker-start-container.sh fuego-nojenkins-container > > -# docker# ftc list-boards > > -# docker, bbb, ... > > -# docker# ftc list-tests > > -# Functional.hello_world, ... > > -# docker# ftc run-test -b docker -t Functional.hello_world > > -# log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1) > > - > > -FROM debian:stretch-slim > > -MAINTAINER [email protected] > > - > > -# ============================================================================== > > -# Proxy variables > > -# ============================================================================== > > - > > -ARG HTTP_PROXY > > -ENV http_proxy ${HTTP_PROXY} > > -ENV https_proxy ${HTTP_PROXY} > > - > > -# ============================================================================== > > -# Prepare basic image > > -# ============================================================================== > > - > > -ARG DEBIAN_FRONTEND=noninteractive > > - > > -WORKDIR / > > -RUN echo deb http://deb.debian.org/debian stretch main non-free > /etc/apt/sources.list > > -RUN echo deb http://security.debian.org/debian-security stretch/updates main >> /etc/apt/sources.list > > -RUN if [ -n "$HTTP_PROXY" ]; then echo 'Acquire::http::proxy "'$HTTP_PROXY'";' > /etc/apt/apt.conf.d/80proxy; fi > > - > > -# Fuego python dependencies > > -# - python-lxml: ftc, loggen > > -# - python-simplejson: ftc > > -# - python-yaml: ftc > > -# - python-openpyxl: ftc (also LTP) > > -# - python-requests: ftc (also fuego_release_test) > > -# - python-reportlab: ftc > > -# - python-parsedatetime: ftc > > -# - python-pexpect: ssh_exec (part of ttc) > > -# - python-pip: to install filelock, flake8 > > -# - filelock: parser > > -RUN mkdir -p /usr/share/man/man1 > > -RUN apt-get update -q=2 && apt-get -q=2 -V --no-install-recommends install \ > > - python-lxml python-simplejson python-yaml python-openpyxl \ > > - python-requests python-reportlab python-parsedatetime \ > > - python-pexpect python-pip python-setuptools python-wheel > > -RUN pip install filelock > > - > > -# Fuego command dependencies > > -RUN apt-get -q=2 -V --no-install-recommends install \ > > - git sshpass openssh-client sudo net-tools wget curl lava-tool \ > > - bash-completion iproute2 > > - > > -# Default SDK for testing locally or on an x86 board > > -RUN apt-get -q=2 -V --no-install-recommends install \ > > - build-essential cmake bison flex automake kmod libtool \ > > - libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \ > > - libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools > > - > > -# Default test host dependencies > > -# - iperf iperf3 netperf: used as servers on the host > > -# - bzip2 bc: used for local tests by Functional.bzip2/bc > > -# - python-matplotlib: Benchmark.iperf3 parser > > -# - python-xmltodict: AGL tests > > -# - flake8: Functional.fuego_lint > > -# - netpipe-tcp - used by Benchmark.netpipe (provides the netpipe server) > > -# - iputils-ping - for /bin/ping command, used by some tests > > -# FIXTHIS: install dependencies dynamically on the tests that need them > > -RUN apt-get -q=2 -V --no-install-recommends install \ > > - iperf iperf3 netperf bzip2 bc python-matplotlib python-xmltodict \ > > - netpipe-tcp iputils-ping > > -RUN pip install flake8 > > - > > -# miscelaneous packages: > > -# python-serial - used by serio > > -# diffstat and vim - used by Tim > > -# time - useful for timing command duration > > -RUN apt-get -q=2 -V --no-install-recommends install \ > > - python-serial \ > > - diffstat \ > > - vim \ > > - time > > - > > -# FIXTHIS: determine if these tools are really necessary > > -# RUN apt-get -q=2 -V --no-install-recommends install \ > > -# apt-utils python-paramiko \ > > -# xmlstarlet rsync \ > > -# inotify-tools gettext netpipe-tcp \ > > -# at minicom lzop bsdmainutils \ > > -# mc netcat openssh-server > > - > > -# bsdmainutils provides hexdump, which is needed for netperf test > > -RUN apt-get -q=2 -V --no-install-recommends install \ > > - bsdmainutils > > - > > -RUN /bin/bash -c 'echo "dash dash/sh boolean false" | debconf-set-selections ; dpkg-reconfigure dash' > > -RUN if [ -n "$HTTP_PROXY" ]; then echo "use_proxy = on" >> /etc/wgetrc; fi > > -RUN if [ -n "$HTTP_PROXY" ]; then echo -e "http_proxy=$HTTP_PROXY\nhttps_proxy=$HTTP_PROXY" >> /etc/environment; fi > > - > > -# ============================================================================== > > -# get ttc script and helpers > > -# ============================================================================== > > -RUN git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc > > -RUN /usr/local/src/ttc/install.sh /usr/local/bin > > -# as of ttc version 2.0.2, this should not be needed > > -#RUN perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc > > - > > -# ============================================================================== > > -# Serial Config > > -# ============================================================================== > > - > > -RUN /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; cp /usr/local/src/serio/serio /usr/local/bin/ ; ln -s > > /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh' > > - > > -RUN /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; cp /usr/local/src/serlogin/serlogin > > /usr/local/bin' > > - > > -# ============================================================================== > > -# fserver > > -# ============================================================================== > > - > > -RUN /bin/bash -c 'git clone https://github.com/tbird20d/fserver.git /usr/local/lib/fserver ; ln -s > /usr/local/lib/fserver/start_local_bg_server > > /usr/local/bin/start_local_bg_server' > > - > > -# ============================================================================== > > -# ftc post installation > > -# ============================================================================== > > - > > -RUN ln -s /fuego-core/scripts/ftc /usr/local/bin/ > > -COPY fuego-core/scripts/ftc_completion.sh /etc/bash_completion.d/ftc > > -RUN echo ". /etc/bash_completion" >> /root/.bashrc > > - > > -# ============================================================================== > > -# Lava > > -# ============================================================================== > > - > > -RUN ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin > > -RUN ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin > > - > > -# ============================================================================== > > -# Setup startup command > > -# ============================================================================== > > - > > -# FIXTHIS: when running multiple Fuego containers netperf will complain because the port is taken > > -ENTRYPOINT service netperf start && iperf3 -V -s -D -f M && /bin/bash > > diff --git a/README b/README > > index 882d84e..ce42e1e 100644 > > --- a/README > > +++ b/README > > @@ -44,6 +44,29 @@ You can change the name of the docker image and the port used by Jenkins > > by passing the corresponding parameters to install.sh. > > ./install.sh fuego-8082 8082 > > > > +To run fuego on different distribution versions like Debian(stretch, buster, etc.) > > + pass appropriate arguments to the install.sh script > > + ./install.sh <image name> <Jenkins port> <Distribution name> <suite> > > should probably be: > ./install.sh [[<image name>] [[<Jenkins port>] [<Distribution name> <suite>]]] > > > + > > +By default install.sh script will install fuego on Debian stretch version > > + to install fuego on Debian buster version pass below options > > + ./install.sh fuego_buster 8090 debian buster > > + > > +[Note] It is currently verified on Debian versions (stretch and buster), > > +and yet to provide install scripts for other distributions > > + > > +Installing Fuego to a host system > > +------------------------------------------------------------------------------- > > +Run install-fuego.sh to install fuego on host system, by default it assume the > > +host system is Debian stretch > > + ./install-fuego.sh > > + > > + to install on Debian buster version > > + ./install-fuego.sh -d debian -s buster -p 8090 > > + > > + to install without jenkins pass --nojenkins option to the script > > + ./install-fuego.sh --nojenkins > > + > > Running > > ------------------------------------------------------------------------------- > > To start your Fuego container, issue the following command top directory: > > @@ -85,9 +108,9 @@ Updating the Jenkins version > > ------------------------------------------------------------------------------- > > > > Modify the version and checksum > > - $ vi Dockerfile > > - ARG JENKINS_VERSION=2.32.1 > > - ARG JENKINS_SHA=3cbe9b6d4d82c975642ea4409e21e087 > > + $ vi install-debian.sh > > + JENKINS_VERSION=2.32.1 > > + JENKINS_SHA=3cbe9b6d4d82c975642ea4409e21e087 > > > > Re-build the flot plugin > > $ sudo apt-get install maven > > diff --git a/entrypoint.sh b/entrypoint.sh > > new file mode 100755 > > index 0000000..6eac895 > > --- /dev/null > > +++ b/entrypoint.sh > > @@ -0,0 +1,7 @@ > > +#!/bin/bash > > + > > +[ "$1" != "--nojenkins" ] && service jenkins start > > + > > +#TODO: netperf not present in buster, remove this condition once alternative found > > +[ -f /etc/init.d/netperf ] && service netperf start > > +iperf3 -V -s -D -f M > > diff --git a/fuego-host-scripts/docker-build-image.sh b/fuego-host-scripts/docker-build-image.sh > > index 5823357..6889c47 100755 > > --- a/fuego-host-scripts/docker-build-image.sh > > +++ b/fuego-host-scripts/docker-build-image.sh > > @@ -2,9 +2,10 @@ > > # $1 - name for the docker image (default: fuego) > > # $2 - port for jenkins (default: 8090) > > # $3 - Dockerfile or Dockerfile.nojenkins > > +# $4 - Debian code name on which to install (default: stretch) > > # > > # Example: > > -# ./fuego-host-scripts/docker-build-image.sh --no-cache myfuegoimg 8082 Dockerfile.nojenkins > > +# ./fuego-host-scripts/docker-build-image.sh --no-cache myfuegoimg 8082 debian stretch --nojenkins > > # > > if [ "$1" = "--no-cache" ]; then > > NO_CACHE=--no-cache > > @@ -13,7 +14,10 @@ fi > > > > DOCKERIMAGE=${1:-fuego} > > JENKINS_PORT=${2:-8090} > > -DOCKERFILE=${3:-Dockerfile} > > +DISTRIBUTION=${3:-debian} > > +SUITE=${4:-stretch} > > +NO_JENKINS=${5:""} > > +DOCKERFILE="Dockerfile" > > > > if [ "$(id -u)" == "0" ]; then > > JENKINS_UID=$(id -u $SUDO_USER) > > @@ -27,4 +31,7 @@ echo "Using Port $JENKINS_PORT" > > > > sudo docker build ${NO_CACHE} -t ${DOCKERIMAGE} --build-arg HTTP_PROXY=$http_proxy \ > > --build-arg uid=$JENKINS_UID --build-arg gid=$JENKINS_GID \ > > + --build-arg DIST=$DISTRIBUTION \ > > + --build-arg SUITE=$SUITE \ > > + --build-arg NO_JENKINS=$NO_JENKINS \ > > --build-arg JENKINS_PORT=$JENKINS_PORT -f $DOCKERFILE . > > diff --git a/install-buster.sh b/install-buster.sh > > new file mode 100755 > > index 0000000..f4ce65b > > --- /dev/null > > +++ b/install-buster.sh > > @@ -0,0 +1,294 @@ > > +#!/bin/bash > > +# 2019 (c) Toshiba corp. <[email protected]> > > +# > > +# Usage: > > +# $ sudo ./install-buster.sh [--help|-h] [--nojenkins] [<port>] > > +# > > +usage(){ > > + > > +cat <<HERE > > +Usage: sudo ./install-buster.sh [--help|-h] [--nojenkins] [<port>] > > + > > +Install fuego on the host Debian filesystem. > > + > > +options: > > + --help Show usage help > > + --nojenkins Install Fuego without Jenkins > > +HERE > > +} > > + > > +if [[ $EUID -ne 0 ]]; then > > + echo "Sorry, you need root permissions" > > + exit 1 > > +fi > > + > > +nojenkins=0 > > + > > +POSITIONAL=() > > +while [[ $# -gt 0 ]] ; do > > + case $1 in > > + --nojenkins) > > + nojenkins=1 > > + shift > > + ;; > > + --help | -h) > > + usage > > + exit 0 > > + ;; > > + *) > > + POSITIONAL+=("$1") # save argument for later > > + shift > > + esac > > +done > > +set -- "${POSITIONAL[@]}" # restore positional arguments > > + > > +port=${2:-8090} > > + > > +# ============================================================================== > > +# Install Fuego dependencies > > +# ============================================================================== > > + > > +mkdir -p /usr/share/man/man1 > > + > > +# netperf is in non-free > > +echo deb http://deb.debian.org/debian buster main non-free > /etc/apt/sources.list > > +echo deb http://security.debian.org/debian-security buster/updates main >> /etc/apt/sources.list > > + > > +apt-get update -q=2 > > + > > +# Fuego python dependencies > > +apt-get -q=2 -V --no-install-recommends install \ > > + python-lxml python-simplejson python-yaml python-openpyxl \ > > + python-requests python-reportlab python-parsedatetime \ > > + python-pexpect python-pip python-setuptools python-wheel > > +pip install filelock > > + > > Can we put this stuff that is common between install-buster.sh and install-stretch.sh > into an "install-debian-common.sh"? > > > +# Fuego command dependencies > > +apt-get -q=2 -V --no-install-recommends install \ > > + git sshpass openssh-client sudo net-tools wget curl lava-tool \ > > + bash-completion iproute2 > > + > > +# Default SDK for testing locally or on an x86 board > > +apt-get -q=2 -V --no-install-recommends install \ > > + build-essential cmake bison flex kmod automake libtool \ > > + libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \ > > + libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools > > + > > +# Default test host dependencies > > +apt-get -q=2 -V --no-install-recommends install \ > > + iperf iperf3 bzip2 bc python-matplotlib python-xmltodict \ > > + netpipe-tcp iputils-ping > > +pip install flake8 > > + > > +# miscelaneous packages: > > +# python-serial - used by serio > > +# diffstat and vim - used by Tim > > +# time - useful for timing command duration > > +apt-get -q=2 -V --no-install-recommends install \ > > + python-serial \ > > + diffstat \ > > + vim \ > > + time > > + > > +# install packages used by NuttX SDK > > +apt-get -q=2 -V --no-install-recommends install \ > > + genromfs > > + > > +# bsdmainutils provides hexdump, which is needed for netperf test > > +#apt-get -q=2 -V --no-install-recommends install \ > > +# bsdmainutils > > + > > +echo "dash dash/sh boolean false" | debconf-set-selections > > +DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash > > + > > +# TODO: check if proxy settings can set outside docker container > > +if [ -n "$http_proxy" ]; then > > + sed -i -e 's/#use_proxy = on/use_proxy = on/g' /etc/wgetrc > > + echo -e "http_proxy=$http_proxy\nhttps_proxy=$https_proxy" >> /etc/environment > > +fi > > This doesn't seem specific to Debian (/etc/wgetrc and /etc/environment). > Should this be moved to an 'install-common.sh' script? > > > + > > +# ============================================================================== > > +# Install Jenkins > > +# ============================================================================== > > + > > +if [ $nojenkins -eq 0 ]; then > > + #JENKINS_VERSION=2.164.2 > > + #JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab > > + #JENKINS_VERSION=2.249.3 > > + #JENKINS_SHA=534014c007edbb533a1833fe6f2dc115faf3faa2 > > + JENKINS_VERSION=2.164.3 > > + JENKINS_SHA=27f14f0ccca207660765205317295a5c29f4dd53 > > + JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenkins_${JENKINS_VERSION}_all.deb > > + export JENKINS_UC=https://updates.jenkins.io > > + REF=/var/lib/jenkins/plugins > > + JENKINS_HOME=/var/lib/jenkins > > + JENKINS_PORT=$port > > + > > + # Jenkins dependencies > > + apt-get -q=2 -V --no-install-recommends install \ > > + default-jdk daemon psmisc adduser procps unzip > > + pip install python-jenkins==1.4.0 > > + > > + echo -e "JENKINS_PORT=$JENKINS_PORT" >> /etc/environment > > + export JENKINS_PORT=$port > > + > > + groupadd jenkins > > + useradd -l -m -d "${JENKINS_HOME}" -g jenkins -G sudo -s /bin/bash jenkins > > + > > + wget -nv ${JENKINS_URL} > > + echo "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum -c - > > + > > + # allow Jenkins to start and install plugins, as part of dpkg installation > > + printf "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d > > + dpkg -i jenkins_${JENKINS_VERSION}_all.deb > > + rm jenkins_${JENKINS_VERSION}_all.deb > > + > > + # fix for Jenkins Java version check error > > + JAVA_VERSION_CHECK="\$\(\$JAVA -version 2\>\&1 \| sed -n \';s/.* version > > \"\\\(.*\\\)\\\.\\\(.*\\\)\\\..\*\\\..\*\".\*\/\\\1\\\2\/p;\'\)" > > + sed -i -e "s#^JAVA_VERSION=.*#JAVA_VERSION\=\"${JAVA_VERSION_CHECK}\"#g" /etc/init.d/jenkins > > + > > + source /etc/default/jenkins && \ > > + JENKINS_ARGS="$JENKINS_ARGS --prefix=/fuego" && \ > > + sed -i -e "s#JENKINS_ARGS.*#JENKINS_ARGS\=\"${JENKINS_ARGS}\"#g" /etc/default/jenkins > > + > > + source /etc/default/jenkins && \ > > + JAVA_ARGS="$JAVA_ARGS -Djenkins.install.runSetupWizard=false - > > Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true" && \ > > + if [ -n "$http_proxy" ]; then \ > > + PROXYSERVER=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\1/') && \ > > + PROXYPORT=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\2/') && \ > > + JAVA_ARGS="$JAVA_ARGS -Dhttp.proxyHost="${PROXYSERVER}" -Dhttp.proxyPort="${PROXYPORT}" - > > Dhttps.proxyHost="${PROXYSERVER}" -Dhttps.proxyPort="${PROXYPORT}; \ > > + fi && \ > > + sed -i -e "s#^JAVA_ARGS.*#JAVA_ARGS\=\"${JAVA_ARGS}\"#g" /etc/default/jenkins; > > + > > + sed -i -e "s#8080#$JENKINS_PORT#g" /etc/default/Jenkins > > The above looks like there's a lot of non-Debian-specific stuff intermingled with Debian-specific stuff. > I'm not sure what other distributions are intended to be supported, but wouldn't the user and > group manipulation be the same for those? > > > + > > + # set up Jenkins plugins > > + cp frontend-install/install-plugins.sh \ > > + frontend-install/jenkins-support \ > > + frontend-install/clitest \ > > + /usr/local/bin/ > > + > > + mkdir -p $JENKINS_HOME/userContent/docs > > + cp frontend-install/config.xml $JENKINS_HOME/config.xml > > + ln -s /fuego-rw/logs $JENKINS_HOME/userContent/fuego.logs > > + cp docs/fuego-docs.pdf $JENKINS_HOME/userContent/docs/fuego-docs.pdf > > + cp frontend-install/jenkins.model.JenkinsLocationConfiguration.xml > > $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml > > + sed -i -e "s#8080#$JENKINS_PORT#g" $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml > > + chown -R jenkins:jenkins $JENKINS_HOME/ > > + > > + # start and stop jenkins to pre-populate some settings > > + service jenkins start && \ > > + sleep 30 && \ > > + service jenkins stop > > + > > + # install other plugins from Jenkins update center > > + # NOTE: not sure all of these are needed, but keep list > > + # compatible with 1.2.1 release for now > > + #/usr/local/bin/install-plugins.sh \ > > + # ant:1.9 \ > > + # antisamy-markup-formatter:1.5 \ > > + # bouncycastle-api:2.17 \ > > + # command-launcher:1.3 \ > > + # description-setter:1.10 \ > > + # display-url-api:2.3.1 \ > > + # external-monitor-job:1.7 \ > > + # greenballs:1.15 \ > > + # icon-shim:2.0.3 \ > > + # javadoc:1.5 \ > > + # jdk-tool:1.2 \ > > + # junit:1.27 \ > > + # ldap:1.20 \ > > + # mailer:1.23 \ > > + # matrix-auth:2.3 \ > > + # matrix-project:1.14 \ > > + # pam-auth:1.4 \ > > + # pegdown-formatter:1.3 \ > > + # structs:1.17 \ > > + # windows-slaves:1.4 > > + > > + # install plugins for latest LTS Jenkins (currently 2.249.3) > > + /usr/local/bin/install-plugins.sh \ > > + script-security \ > > + structs \ > > + workflow-step-api \ > > + workflow-api \ > > + junit \ > > + scm-api \ > > + ant \ > > + antisamy-markup-formatter \ > > + bouncycastle-api \ > > + command-launcher \ > > + description-setter \ > > + display-url-api \ > > + external-monitor-job \ > > + greenballs \ > > + icon-shim \ > > + javadoc \ > > + jdk-tool \ > > + mailer \ > > + matrix-auth \ > > + matrix-project \ > > + pam-auth \ > > + pegdown-formatter \ > > + windows-slaves > > + > > + cp frontend-install/plugins/flot-plotter-plugin/flot.hpi $JENKINS_HOME/plugins/flot.jpi > > + > > + # make the mod.js symlink well after flot is installed > > + service jenkins start && sleep 30 && \ > > + rm $JENKINS_HOME/plugins/flot/flot/mod.js && \ > > + ln -s /fuego-core/scripts/mod.js $JENKINS_HOME/plugins/flot/flot/mod.js > > + > > + chown -R jenkins:jenkins $JENKINS_HOME/ > > +fi > > + > > +# ============================================================================== > > +# get ttc script and helpers > > +# ============================================================================== > > +git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc > > +/usr/local/src/ttc/install.sh /usr/local/bin > > +perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc > > + > The above and following are definitely not unique to Debian, or a specific version of Debian. > They should be put into an "install-common.sh" > > > +# ============================================================================== > > +# Serial Config > > +# ============================================================================== > > +if [ $nojenkins -eq 0 ]; then > > + /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; chown -R jenkins /usr/local/src/serio ; cp > > /usr/local/src/serio/serio /usr/local/bin/ ; ln -s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh' > > + /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; chown -R jenkins /usr/local/src/serlogin ; > > cp /usr/local/src/serlogin/serlogin /usr/local/bin' > > +else > > + /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; cp /usr/local/src/serio/serio /usr/local/bin/ ; ln - > > s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh' > > + /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; cp /usr/local/src/serlogin/serlogin > > /usr/local/bin' > > +fi > > +cp frontend-install/0001-Fix-host-parsing-for-serial-device-with-in-name.patch \ > > + frontend-install/0002-Output-data-from-port-during-command-execution.patch \ > > + /tmp/ > > +/bin/bash -c 'patch -d /usr/local/src/serio -p1 </tmp/0001-Fix-host-parsing-for-serial-device-with-in-name.patch ; \ > > + patch -d /usr/local/src/serio -p1 </tmp/0002-Output-data-from-port-during-command-execution.patch ;' > > + > > + > > +# ============================================================================== > > +# get ebf script and helpers > > +# ============================================================================== > > +apt-get -q=2 -V --no-install-recommends install jq > > +git clone https://github.com/TimesysGit/board-farm-rest-api.git /usr/local/src/board-farm-rest-api > > +cp /usr/local/src/board-farm-rest-api/cli/ebf /usr/local/bin/ebf > > +chmod a+x /usr/local/bin/ebf > > + > > +# ============================================================================== > > +# fserver > > +# ============================================================================== > > +git clone https://github.com/tbird20d/fserver.git /usr/local/lib/fserver > > +ln -s /usr/local/lib/fserver/start_local_bg_server /usr/local/bin/start_local_bg_server > > + > > +# ============================================================================== > > +# ftc post installation > > +# ============================================================================== > > +ln -s /fuego-core/scripts/ftc /usr/local/bin/ > > +cp fuego-core/scripts/ftc_completion.sh /etc/bash_completion.d/ftc > > +echo ". /etc/bash_completion" >> /root/.bashrc > > + > > +# ============================================================================== > > +# Lava > > +# ============================================================================== > > +ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin > > +ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin > > diff --git a/install-debian.sh b/install-debian.sh > > index ccca032..2d3ffd3 100755 > > --- a/install-debian.sh > > +++ b/install-debian.sh > > @@ -2,102 +2,163 @@ > > # 2019 (c) Toshiba corp. <[email protected]> > > # > > # Usage: > > -# $ sudo ./install-debian.sh [--help|-h] [--nojenkins] [<port>] > > +# $ sudo ./install-debian.sh [--help|-h] [-s <suite>] [-p <jenkins port>] [--nojenkins] > > # > > -if [ -n "$1" ]; then > > - if [ "$1" = "--help" -o "$1" = "-h" ]; then > > - cat <<HERE > > -Usage: sudo ./install-debian.sh [--help|-h] [--nojenkins] [<port>] > > - > > -Installs fuego on the host Debian filesystem. > > - > > -options: > > - --help Show usage help > > - --nojenkins Install Fuego without Jenkins > > -HERE > > - exit 0 > > - fi > > -fi > > + > > +usage(){ > > + echo "" > > + echo "Usage: sudo ./install-debian.sh [--help|-h] [-s <suite>] [-p <jenkins port>] [--nojenkins]" > > + echo "Install fuego on Debian filesystem." > > + echo "" > > + echo "options:" > > + echo " --help | -h Show usage help" > > + echo " --suite | -s <suite> Specify Debian suite" > > + echo " --port | -p <port> Jenkins port" > > + echo " --nojenkins Install Fuego without Jenkins" > > + echo "" > > +} > > > > if [[ $EUID -ne 0 ]]; then > > echo "Sorry, you need root permissions" > > exit 1 > > fi > > > > -if [ "$1" = "--nojenkins" ]; then > > - nojenkins=1 > > +nojenkins=0 > > +port=8090 > > +suite=stretch > > +user=jenkins > > +group=jenkins > > +uid=1000 > > +gid=1000 > > + > > +while [ "$1" != "" ]; do > > + case $1 in > > + -s | --suite ) > > + shift > > + suite=$1 > > + ;; > > + -p | --port ) > > + shift > > + port=$1 > > + ;; > > + -u | --uid ) > > + shift > > + uid=$1 > > + ;; > > + -g | --gid ) > > + shift > > + gid=$1 > > + ;; > > + --nojenkins ) > > + nojenkins=1 > > + ;; > > + -h | --help ) > > + usage > > + exit 0 > > + ;; > > + * ) > > + break > > + esac > > shift > > -else > > - nojenkins=0 > > - port=${1:-8090} > > -fi > > +done > > > > # ============================================================================== > > # Install Fuego dependencies > > # ============================================================================== > > > > # netperf is in non-free > > -echo deb http://deb.debian.org/debian stretch main non-free > /etc/apt/sources.list > > -echo deb http://security.debian.org/debian-security stretch/updates main >> /etc/apt/sources.list > > - > > -apt-get update -q=2 > > +echo deb http://deb.debian.org/debian $suite main non-free > /etc/apt/sources.list > > +echo deb http://security.debian.org/debian-security $suite/updates main >> /etc/apt/sources.list > > > > # Fuego python dependencies > > -apt-get -q=2 -V --no-install-recommends install \ > > +# - python-lxml: ftc, loggen > > +# - python-simplejson: ftc > > +# - python-yaml: ftc > > +# - python-openpyxl: ftc (also LTP) > > +# - python-requests: ftc (also fuego_release_test) > > +# - python-reportlab: ftc > > +# - python-parsedatetime: ftc > > +# - python-pexpect: ssh_exec (part of ttc) > > +# - python-pip: to install filelock, flake8 > > +# - filelock: parser > > +mkdir -p /usr/share/man/man1 > > +apt-get update -q=2 && apt-get -q=2 -V --no-install-recommends install \ > > python-lxml python-simplejson python-yaml python-openpyxl \ > > python-requests python-reportlab python-parsedatetime \ > > - python-pip > > + python-pexpect python-pip python-setuptools python-wheel > > pip install filelock > > > > # Fuego command dependencies > > apt-get -q=2 -V --no-install-recommends install \ > > git sshpass openssh-client sudo net-tools wget curl lava-tool \ > > - bash-completion > > + bash-completion iproute2 > > > > # Default SDK for testing locally or on an x86 board > > apt-get -q=2 -V --no-install-recommends install \ > > build-essential cmake bison flex automake kmod libtool \ > > libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \ > > - libxmuu-dev libglib2.0-dev libaio-dev u-boot-tools pkg-config > > + libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools > > > > # Default test host dependencies > > +# - iperf iperf3 netperf: used as servers on the host > > +# - bzip2 bc: used for local tests by Functional.bzip2/bc > > +# - python-matplotlib: Benchmark.iperf3 parser > > +# - python-xmltodict: AGL tests > > +# - flake8: Functional.fuego_lint > > +# - netpipe-tcp - used by Benchmark.netpipe (provides the netpipe server) > > +# - iputils-ping - for /bin/ping command, used by some tests > > +# FIXTHIS: install dependencies dynamically on the tests that need them > > apt-get -q=2 -V --no-install-recommends install \ > > - iperf iperf3 netperf bzip2 bc python-matplotlib python-xmltodict > > -pip install flake8 bsdmainutils > > + iperf iperf3 bzip2 bc python-matplotlib python-xmltodict \ > > + netpipe-tcp iputils-ping > > > > -echo "dash dash/sh boolean false" | debconf-set-selections > > -DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash > > -if [ -n "$http_proxy" ]; then > > - sed -i -e 's/#use_proxy = on/use_proxy = on/g' /etc/wgetrc > > - echo -e "http_proxy=$http_proxy\nhttps_proxy=$https_proxy" >> /etc/environment > > +#TODO: find alternate package for netperf in buster version > > +if [ $suite != "buster" ];then > > + apt-get -q=2 -V --no-install-recommends install netperf > > fi > > +pip install flake8 > > > > -# ============================================================================== > > -# Clone fuego and fuego-core > > -# ============================================================================== > > -if [ ! -d "/fuego" ]; then > > - cd / > > - git clone --branch next --depth=1 https://bitbucket.org/nirrognas/fuego.git > > - ln -s /fuego/fuego-ro /fuego-ro > > - ln -s /fuego/fuego-rw /fuego-rw > > - cd fuego > > - git clone --branch next --depth=1 https://bitbucket.org/nirrognas/fuego-core.git > > - ln -s /fuego/fuego-core /fuego-core > > -fi > > +# miscelaneous packages: > > +# python-serial - used by serio > > +# diffstat and vim - used by Tim > > +# time - useful for timing command duration > > +apt-get -q=2 -V --no-install-recommends install \ > > + python-serial \ > > + diffstat \ > > + vim \ > > + time > > + > > +# install packages used by NuttX SDK > > +apt-get -q=2 -V --no-install-recommends install \ > > + genromfs > > + > > +# FIXTHIS: determine if these tools are really necessary > > +#RUN apt-get -q=2 -V --no-install-recommends install \ > > +# apt-utils python-paramiko \ > > +# xmlstarlet rsync \ > > +# inotify-tools gettext netpipe-tcp \ > > +# at minicom lzop bsdmainutils \ > > +# mc netcat openssh-server > > + > > +# bsdmainutils provides hexdump, which is needed for netperf test > > +apt-get -q=2 -V --no-install-recommends install \ > > + bsdmainutils > > + > > +echo "dash dash/sh boolean false" | debconf-set-selections > > +DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash > > > > # ============================================================================== > > # Install Jenkins > > # ============================================================================== > > - > > if [ $nojenkins -eq 0 ]; then > > #JENKINS_VERSION=2.164.2 > > #JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab > > JENKINS_VERSION=2.249.3 > > JENKINS_SHA=534014c007edbb533a1833fe6f2dc115faf3faa2 > > JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenkins_${JENKINS_VERSION}_all.deb > > - JENKINS_UC=https://updates.jenkins.io > > - REF=/var/lib/jenkins/plugins > > - JENKINS_HOME=/var/lib/jenkins > > + export JENKINS_UC=https://updates.jenkins.io > > + export REF=/var/lib/jenkins/plugins > > + export JENKINS_HOME=/var/lib/jenkins > > JENKINS_PORT=$port > > > > # Jenkins dependencies > > @@ -106,10 +167,12 @@ if [ $nojenkins -eq 0 ]; then > > pip install python-jenkins==1.4.0 > > > > echo -e "JENKINS_PORT=$JENKINS_PORT" >> /etc/environment > > - groupadd jenkins > > - useradd -l -m -d "${JENKINS_HOME}" -g jenkins -G sudo -s /bin/bash jenkins > > - cd > > - wget ${JENKINS_URL} > > + export JENKINS_PORT=$port > > + > > + getent group ${gid} >/dev/null || groupadd -g ${gid} ${group} > > + useradd -l -m -d "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user} > > + > > + wget -nv ${JENKINS_URL} > > echo "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum -c - > > > > # allow Jenkins to start and install plugins, as part of dpkg installation > > @@ -117,11 +180,60 @@ if [ $nojenkins -eq 0 ]; then > > dpkg -i jenkins_${JENKINS_VERSION}_all.deb > > rm jenkins_${JENKINS_VERSION}_all.deb > > > > - # update ownership > > - chown -R jenkins:jenkins /fuego > > - chown -R jenkins:jenkins /fuego-ro > > - chown -R jenkins:jenkins /fuego-rw > > - chown -R jenkins:jenkins /fuego-core > > +fi > > + > > +# ============================================================================== > > +# get ttc script and helpers > > +# ============================================================================== > > +git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc > > +/usr/local/src/ttc/install.sh /usr/local/bin > > +# as of ttc version 2.0.2, this should not be needed > > +#perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc > > + > > +# ============================================================================== > > +# get ebf script and helpers > > +# ============================================================================== > > +if [ $nojenkins -eq 0 ]; then > > + apt-get -q=2 -V --no-install-recommends install jq > > + git clone https://github.com/TimesysGit/board-farm-rest-api.git /usr/local/src/board-farm-rest-api > > + cp /usr/local/src/board-farm-rest-api/cli/ebf /usr/local/bin/ebf > > + chmod a+x /usr/local/bin/ebf > > +fi > > + > > +# ============================================================================== > > +# Serial Config > > +# ============================================================================== > > +git clone https://github.com/frowand/serio.git /usr/local/src/serio > > +if [ $nojenkins -eq 0 ]; then > > + cp frontend-install/0001-Fix-host-parsing-for-serial-device-with-in-name.patch \ > > + frontend-install/0002-Output-data-from-port-during-command-execution.patch \ > > + /tmp/ > > + patch -d /usr/local/src/serio -p1 </tmp/0001-Fix-host-parsing-for-serial-device-with-in-name.patch > > + patch -d /usr/local/src/serio -p1 </tmp/0002-Output-data-from-port-during-command-execution.patch > > + chown -R jenkins /usr/local/src/serio > > +fi > > +cp /usr/local/src/serio/serio /usr/local/bin/ > > +ln -s /usr/local/bin/serio /usr/local/bin/sercp > > +ln -s /usr/local/bin/serio /usr/local/bin/sersh > > + > > +git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin > > +[ $nojenkins -eq 0 ] && chown -R jenkins /usr/local/src/serlogin > > +cp /usr/local/src/serlogin/serlogin /usr/local/bin > > + > > +# ============================================================================== > > +# fserver > > +# ============================================================================== > > +git clone https://github.com/tbird20d/fserver.git /usr/local/lib/fserver > > +ln -s /usr/local/lib/fserver/start_local_bg_server /usr/local/bin/start_local_bg_server > > + > > +# ============================================================================== > > +# Jenkins post installation > > +# ============================================================================== > > +if [ $nojenkins -eq 0 ]; then > > + > > + # Fix jenkins issue, Java version check fails when java version is 11 > > + JAVA_VERSION_CHECK="\$\(\$JAVA -version 2\>\&1 \| sed -n \';s/.* version \"\\\(\[^.-\]*\\\)\\\.\\\(\[^.- > > \]*\\\).*\\\".*/\\\1\\\2\/p;')" > > + sed -i -e "s#^JAVA_VERSION=.*#JAVA_VERSION\=\"${JAVA_VERSION_CHECK}\"#g" /etc/init.d/jenkins > > > > source /etc/default/jenkins && \ > > JENKINS_ARGS="$JENKINS_ARGS --prefix=/fuego" && \ > > @@ -138,116 +250,107 @@ if [ $nojenkins -eq 0 ]; then > > > > sed -i -e "s#8080#$JENKINS_PORT#g" /etc/default/jenkins > > > > - cp /fuego/frontend-install/install-plugins.sh \ > > - /fuego/frontend-install/jenkins-support \ > > - /fuego/frontend-install/clitest \ > > + # set up Jenkins plugins > > + cp frontend-install/install-plugins.sh \ > > + frontend-install/jenkins-support \ > > + frontend-install/clitest \ > > /usr/local/bin/ > > > > - cp /fuego/frontend-install/config.xml $JENKINS_HOME/config.xml > > - ln -s /fuego-rw/logs $JENKINS_HOME/userContent/fuego.logs > > - mkdir $JENKINS_HOME/userContent/docs > > - cp /fuego/docs/fuego-docs.pdf $JENKINS_HOME/userContent/docs/fuego-docs.pdf > > - jenkins cp /fuego/frontend-install/jenkins.model.JenkinsLocationConfiguration.xml > > $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml > > - sed -i -e "s#8080#$JENKINS_PORT#g" $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml > > - chown -R jenkins:jenkins $JENKINS_HOME/ > > - > > # start and stop jenkins to pre-populate some settings > > service jenkins start && \ > > sleep 30 && \ > > service jenkins stop > > > > + # install plugins - these versions are for Jenkins version 2.164.2 > > + # Explicitly install script-security v1.68, otherwise > > + # v1.74 will automatically be installed as a dependency of > > + # the junit plugin. Make sure to install before junit plugin. > > + # Explicitly install junit:1.27 > > + > > # install other plugins from Jenkins update center > > # NOTE: not sure all of these are needed, but keep list > > - # compatible with 1.2.1 release for now > > - #/usr/local/bin/install-plugins.sh \ > > - # ant:1.9 \ > > - # antisamy-markup-formatter:1.5 \ > > - # bouncycastle-api:2.17 \ > > - # command-launcher:1.3 \ > > - # description-setter:1.10 \ > > - # display-url-api:2.3.1 \ > > - # external-monitor-job:1.7 \ > > - # greenballs:1.15 \ > > - # icon-shim:2.0.3 \ > > - # javadoc:1.5 \ > > - # jdk-tool:1.2 \ > > - # junit:1.27 \ > > - # ldap:1.20 \ > > - # mailer:1.23 \ > > - # matrix-auth:2.3 \ > > - # matrix-project:1.14 \ > > - # pam-auth:1.4 \ > > - # pegdown-formatter:1.3 \ > > - # structs:1.17 \ > > - # windows-slaves:1.4 > > - > > - # install plugins for latest LTS Jenkins (currently 2.249.3) > > - /usr/local/bin/install-plugins.sh \ > > - script-security \ > > - structs \ > > - workflow-step-api \ > > - workflow-api \ > > - junit \ > > - scm-api \ > > - ant \ > > - antisamy-markup-formatter \ > > - bouncycastle-api \ > > - command-launcher \ > > - description-setter \ > > - display-url-api \ > > - external-monitor-job \ > > - greenballs \ > > - icon-shim \ > > - javadoc \ > > - jdk-tool \ > > - mailer \ > > - matrix-auth \ > > - matrix-project \ > > - pam-auth \ > > - pegdown-formatter \ > > - windows-slaves > > - > > - cp frontend-install/plugins/flot-plotter-plugin/flot.hpi $JENKINS_HOME/plugins/flot.jpi > > + # as compatible as possible with 1.2.1 release for now > > + # Do NOT change the order of the plugins, unless you know what > > + # you are doing. Otherwise, the install-plugins.sh script will > > + # install the wrong versions of other plugins based on dependency > > + # information in the plugin files. > > > > + #/usr/local/bin/install-plugins.sh \ > > + # script-security:1.68 \ > > + # structs:1.20 \ > > + # workflow-step-api:2.22 \ > > + # workflow-api:2.40 \ > > + # junit:1.27 \ > > + # scm-api:2.6.3 \ > > + # ant:1.9 \ > > + # antisamy-markup-formatter:1.5 \ > > + # bouncycastle-api:2.17 \ > > + # command-launcher:1.3 \ > > + # description-setter:1.10 \ > > + # display-url-api:2.3.1 \ > > + # external-monitor-job:1.7 \ > > + # greenballs:1.15 \ > > + # icon-shim:2.0.3 \ > > + # javadoc:1.5 \ > > + # jdk-tool:1.2 \ > > + # ldap:1.20 \ > > + # mailer:1.23 \ > > + # matrix-auth:2.3 \ > > + # matrix-project:1.14 \ > > + # pam-auth:1.5 \ > > + # pegdown-formatter:1.3 \ > > + # windows-slaves:1.4 > > + > > + # install plugins for latest LTS Jenkins (currently 2.249.3) > > + /usr/local/bin/install-plugins.sh \ > > + script-security \ > > + structs \ > > + workflow-step-api \ > > + workflow-api \ > > + junit \ > > + scm-api \ > > + ant \ > > + antisamy-markup-formatter \ > > + bouncycastle-api \ > > + command-launcher \ > > + description-setter \ > > + display-url-api \ > > + external-monitor-job \ > > + greenballs \ > > + icon-shim \ > > + javadoc \ > > + jdk-tool \ > > + mailer \ > > + matrix-auth \ > > + matrix-project \ > > + pam-auth \ > > + pegdown-formatter \ > > + windows-slaves > > + > > + cp frontend-install/plugins/flot-plotter-plugin/flot.hpi $JENKINS_HOME/plugins/flot.jpi > > + > > + # jenkins should automatically unzip any plugins in the plugin dir > > # make the mod.js symlink well after flot is installed > > service jenkins start && sleep 30 && \ > > rm $JENKINS_HOME/plugins/flot/flot/mod.js && \ > > ln -s /fuego-core/scripts/mod.js $JENKINS_HOME/plugins/flot/flot/mod.js > > > > - chown -R jenkins:jenkins $JENKINS_HOME/ > > -else > > - sed -i -e 's/jenkins_enabled=1/jenkins_enabled=0/g' /fuego-ro/conf/fuego.conf > > -fi > > + mkdir -p $JENKINS_HOME/userContent/docs > > + ln -s /fuego-rw/logs $JENKINS_HOME/userContent/fuego.logs > > + cp docs/fuego-docs.pdf $JENKINS_HOME/userContent/docs/fuego-docs.pdf > > > > -# ============================================================================== > > -# get ttc script and helpers > > -# ============================================================================== > > -git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc > > -/usr/local/src/ttc/install.sh /usr/local/bin > > -perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc > > + cp frontend-install/config.xml $JENKINS_HOME/config.xml > > + cp frontend-install/jenkins.model.JenkinsLocationConfiguration.xml > > $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml > > + sed -i -e "s#8080#$JENKINS_PORT#g" $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml > > > > -# ============================================================================== > > -# Serial Config > > -# ============================================================================== > > -if [ $nojenkins -eq 0 ]; then > > - /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; chown -R jenkins /usr/local/src/serio ; cp > > /usr/local/src/serio/serio /usr/local/bin/ ; ln -s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh' > > - /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; chown -R jenkins /usr/local/src/serlogin ; > > cp /usr/local/src/serlogin/serlogin /usr/local/bin' > > -else > > - /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; cp /usr/local/src/serio/serio /usr/local/bin/ ; ln - > > s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh' > > - /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; cp /usr/local/src/serlogin/serlogin > > /usr/local/bin' > > + chown -R jenkins:jenkins $JENKINS_HOME/ > > fi > > > > -# ============================================================================== > > -# fserver > > -# ============================================================================== > > -git clone https://github.com/tbird20d/fserver.git /usr/local/lib/fserver > > -ln -s /usr/local/lib/fserver/start_local_bg_server /usr/local/bin/start_local_bg_server > > - > > # ============================================================================== > > # ftc post installation > > # ============================================================================== > > ln -s /fuego-core/scripts/ftc /usr/local/bin/ > > -cp /fuego-core/scripts/ftc_completion.sh /etc/bash_completion.d/ftc > > +cp fuego-core/scripts/ftc_completion.sh /etc/bash_completion.d/ftc > > echo ". /etc/bash_completion" >> /root/.bashrc > > > > # ============================================================================== > > @@ -255,14 +358,3 @@ echo ". /etc/bash_completion" >> /root/.bashrc > > # ============================================================================== > > ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin > > ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin > > - > > -# ============================================================================== > > -# Small guide > > -# ============================================================================== > > -echo "Run 'service netperf start' to start a netperf server" > > -echo "Run 'iperf3 -V -s -D -f M' to start an iperf3 server" > > -echo "Run 'ftc list-boards' to see the available boards" > > -echo "Run 'ftc list-tests' to see the available tests" > > -echo "Run 'ftc run-test -b local -t Functional.hello_world' to run a hello world" > > -echo "Run 'ftc run-test -b local -t Benchmark.Dhrystone -s 500M' to run Dhrystone" > > -echo "Run 'ftc gen-report' to get results" > > diff --git a/install-fuego.sh b/install-fuego.sh > > new file mode 100755 > > index 0000000..7b3e010 > > --- /dev/null > > +++ b/install-fuego.sh > > @@ -0,0 +1,109 @@ > > +#!/bin/bash > > +# 2019 (c) Toshiba corp. <[email protected]> > > +# > > +# Usage: > > +# $ sudo ./install-fuego.sh [--help|-h] [-d <distribution>] [-s <suite>] [-p <jenkins port>] [--nojenkins] > > +# > > + > > +usage(){ > > + echo "" > > + echo "Usage: sudo ./install-fuego.sh [--help|-h] [-d <distribution>] [-s <suite>] [-p <jenkins port>] [--nojenkins]" > > + echo "Install fuego on Debian filesystem." > > + echo "" > > + echo "options:" > > + echo " --help | -h Show usage help" > > + echo " --dist | -d <distribution> Specify distribution" > > + echo " --suite | -s <suite> Specify suite" > > + echo " --port | -p <port> Jenkins port" > > + echo " --nojenkins Install Fuego without Jenkins" > > + echo "" > > +} > > + > > +if [[ $EUID -ne 0 ]]; then > > + echo "Sorry, you need root permissions" > > + exit 1 > > +fi > > + > > +nojenkins= > > +port=8090 > > +dist=debian > > +suite=stretch > > +fuego_dir=$(pwd) > > + > > +while [ "$1" != "" ]; do > > + case $1 in > > + -d | --dist ) > > + shift > > + dist=$1 > > + ;; > > + -s | --suite ) > > + shift > > + suite=$1 > > + ;; > > + -p | --port ) > > + shift > > + port=$1 > > + ;; > > + --nojenkins ) > > + nojenkins="--nojenkins" > > + ;; > > + -h | --help ) > > + usage > > + exit 0 > > + ;; > > + * ) > > + break > > + esac > > + shift > > +done > > + > > +# ============================================================================== > > +# Clone fuego-core > > +# ============================================================================== > > +if [ ! -f "fuego-core/scripts/ftc" ]; then > > + # set fuego-core branch to same as current fuego branch > > + # get current git branch > > + set -o noglob > > + while IFS=" " read -r part1 part2 ; do > > + if [ $part1 = "*" ] ; then > > + branch=$part2 > > + fi > > + done < <(git branch) > > + set +o noglob > > + git clone -b $branch https://bitbucket.org/fuegotest/fuego-core.git > > +fi > > + > > +# ============================================================================== > > +# Install fuego on host system > > +# ============================================================================== > > +./install-$dist.sh -s $suite -p $port $nojenkins > > + > > +systemctl daemon-reload > > +service jenkins restart > > + > > +# provide permission to current user > > +usermod -aG jenkins $USER > > +chmod -R 775 $fuego_dir/fuego-rw > > + > > +chown -R jenkins:jenkins $fuego_dir/fuego-ro > > +chown -R jenkins:jenkins $fuego_dir/fuego-rw > > +chown -R jenkins:jenkins $fuego_dir/fuego-core > > + > > +# ============================================================================== > > +# Create Links to fuego folders > > +# ============================================================================== > > +ln -s $fuego_dir/fuego-ro / > > +ln -s $fuego_dir/fuego-rw / > > +ln -s $fuego_dir/fuego-core / > > + > > +# ============================================================================== > > +# Small guide > > +# ============================================================================== > > +echo "Logout and Login current user session to work properly" > > +echo "Run 'service netperf start' to start a netperf server" > > +echo "Run 'iperf3 -V -s -D -f M' to start an iperf3 server" > > +echo "Run 'ftc list-boards' to see the available boards" > > +echo "Run 'ftc list-tests' to see the available tests" > > +echo "Run 'ftc run-test -b local -t Functional.hello_world' to run a hello world" > > +echo "Run 'ftc run-test -b local -t Benchmark.Dhrystone -s 500M' to run Dhrystone" > > +echo "Run 'ftc gen-report' to get results" > > diff --git a/install-stretch.sh b/install-stretch.sh > > new file mode 100755 > > index 0000000..4f6f724 > > --- /dev/null > > +++ b/install-stretch.sh > > @@ -0,0 +1,288 @@ > > +#!/bin/bash > > +# 2019 (c) Toshiba corp. <[email protected]> > > +# > > +# Usage: > > +# $ sudo ./install-<debian-suit>.sh [--help|-h] [--nojenkins] [<port>] > > +# > > +usage(){ > > + > > +cat <<HERE > > +Usage: sudo ./install-<debian-suit>.sh [--help|-h] [--nojenkins] [<port>] > > + > > +Install fuego on the host Debian filesystem. > > + > > +options: > > + --help Show usage help > > + --nojenkins Install Fuego without Jenkins > > +HERE > > +} > > + > > +if [[ $EUID -ne 0 ]]; then > > + echo "Sorry, you need root permissions" > > + exit 1 > > +fi > > + > > +nojenkins=0 > > + > > +POSITIONAL=() > > +while [[ $# -gt 0 ]] ; do > > + case $1 in > > + --nojenkins) > > + nojenkins=1 > > + shift > > + ;; > > + --help | -h) > > + usage > > + exit 0 > > + ;; > > + *) > > + POSITIONAL+=("$1") # save argument for later > > + shift > > + esac > > +done > > +set -- "${POSITIONAL[@]}" # restore positional arguments > > + > > +port=${2:-8090} > > + > > +# ============================================================================== > > +# Install Fuego dependencies > > +# ============================================================================== > > + > > +mkdir -p /usr/share/man/man1 > > + > > +# netperf is in non-free > > +echo deb http://deb.debian.org/debian stretch main non-free > /etc/apt/sources.list > > +echo deb http://security.debian.org/debian-security stretch/updates main >> /etc/apt/sources.list > > + > > +apt-get update -q=2 > > + > > +# Fuego python dependencies > > +apt-get -q=2 -V --no-install-recommends install \ > > + python-lxml python-simplejson python-yaml python-openpyxl \ > > + python-requests python-reportlab python-parsedatetime \ > > + python-pexpect python-pip python-setuptools python-wheel > > +pip install filelock > > + > > +# Fuego command dependencies > > +apt-get -q=2 -V --no-install-recommends install \ > > + git sshpass openssh-client sudo net-tools wget curl lava-tool \ > > + bash-completion iproute2 > > + > > +# Default SDK for testing locally or on an x86 board > > +apt-get -q=2 -V --no-install-recommends install \ > > + build-essential cmake bison flex kmod automake libtool \ > > + libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \ > > + libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools > > + > > +# Default test host dependencies > > +apt-get -q=2 -V --no-install-recommends install \ > > + iperf iperf3 netperf bzip2 bc python-matplotlib python-xmltodict \ > > + netpipe-tcp iputils-ping > > +pip install flake8 > > + > > +# miscelaneous packages: > > +# python-serial - used by serio > > +# diffstat and vim - used by Tim > > +# time - useful for timing command duration > > +apt-get -q=2 -V --no-install-recommends install \ > > + python-serial \ > > + diffstat \ > > + vim \ > > + time > > + > > +# install packages used by NuttX SDK > > +apt-get -q=2 -V --no-install-recommends install \ > > + genromfs > > + > > +# bsdmainutils provides hexdump, which is needed for netperf test > > +apt-get -q=2 -V --no-install-recommends install \ > > + bsdmainutils > > + > > +echo "dash dash/sh boolean false" | debconf-set-selections > > +DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash > > + > > +# TODO: check if proxy settings can set outside docker container > > +#if [ -n "$http_proxy" ]; then > > +# sed -i -e 's/#use_proxy = on/use_proxy = on/g' /etc/wgetrc > > +# echo -e "http_proxy=$http_proxy\nhttps_proxy=$https_proxy" >> /etc/environment > > +#fi > > + > > +# ============================================================================== > > +# Install Jenkins > > +# ============================================================================== > > + > > +if [ $nojenkins -eq 0 ]; then > > + #JENKINS_VERSION=2.164.2 > > + #JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab > > + JENKINS_VERSION=2.249.3 > > + JENKINS_SHA=534014c007edbb533a1833fe6f2dc115faf3faa2 > > + JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenkins_${JENKINS_VERSION}_all.deb > > + export JENKINS_UC=https://updates.jenkins.io > > + REF=/var/lib/jenkins/plugins > > + JENKINS_HOME=/var/lib/jenkins > > + JENKINS_PORT=$port > > + > > + # Jenkins dependencies > > + apt-get -q=2 -V --no-install-recommends install \ > > + default-jdk daemon psmisc adduser procps unzip > > + pip install python-jenkins==1.4.0 > > + > > + echo -e "JENKINS_PORT=$JENKINS_PORT" >> /etc/environment > > + export JENKINS_PORT=$port > > + > > + groupadd jenkins > > + useradd -l -m -d "${JENKINS_HOME}" -g jenkins -G sudo -s /bin/bash jenkins > > + > > + wget -nv ${JENKINS_URL} > > + echo "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum -c - > > + > > + # allow Jenkins to start and install plugins, as part of dpkg installation > > + printf "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d > > + dpkg -i jenkins_${JENKINS_VERSION}_all.deb > > + rm jenkins_${JENKINS_VERSION}_all.deb > > + > > + source /etc/default/jenkins && \ > > + JENKINS_ARGS="$JENKINS_ARGS --prefix=/fuego" && \ > > + sed -i -e "s#JENKINS_ARGS.*#JENKINS_ARGS\=\"${JENKINS_ARGS}\"#g" /etc/default/jenkins > > + > > + source /etc/default/jenkins && \ > > + JAVA_ARGS="$JAVA_ARGS -Djenkins.install.runSetupWizard=false - > > Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true" && \ > > + if [ -n "$http_proxy" ]; then \ > > + PROXYSERVER=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\1/') && \ > > + PROXYPORT=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\2/') && \ > > + JAVA_ARGS="$JAVA_ARGS -Dhttp.proxyHost="${PROXYSERVER}" -Dhttp.proxyPort="${PROXYPORT}" - > > Dhttps.proxyHost="${PROXYSERVER}" -Dhttps.proxyPort="${PROXYPORT}; \ > > + fi && \ > > + sed -i -e "s#^JAVA_ARGS.*#JAVA_ARGS\=\"${JAVA_ARGS}\"#g" /etc/default/jenkins; > > + > > + sed -i -e "s#8080#$JENKINS_PORT#g" /etc/default/jenkins > > + > > + # set up Jenkins plugins > > + cp frontend-install/install-plugins.sh \ > > + frontend-install/jenkins-support \ > > + frontend-install/clitest \ > > + /usr/local/bin/ > > + > > + mkdir -p $JENKINS_HOME/userContent/docs > > + cp frontend-install/config.xml $JENKINS_HOME/config.xml > > + ln -s /fuego-rw/logs $JENKINS_HOME/userContent/fuego.logs > > + cp docs/fuego-docs.pdf $JENKINS_HOME/userContent/docs/fuego-docs.pdf > > + cp frontend-install/jenkins.model.JenkinsLocationConfiguration.xml > > $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml > > + sed -i -e "s#8080#$JENKINS_PORT#g" $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml > > + chown -R jenkins:jenkins $JENKINS_HOME/ > > + > > + # start and stop jenkins to pre-populate some settings > > + service jenkins start && \ > > + sleep 30 && \ > > + service jenkins stop > > + > > + # install other plugins from Jenkins update center > > + # NOTE: not sure all of these are needed, but keep list > > + # compatible with 1.2.1 release for now > > + #/usr/local/bin/install-plugins.sh \ > > + # ant:1.9 \ > > + # antisamy-markup-formatter:1.5 \ > > + # bouncycastle-api:2.17 \ > > + # command-launcher:1.3 \ > > + # description-setter:1.10 \ > > + # display-url-api:2.3.1 \ > > + # external-monitor-job:1.7 \ > > + # greenballs:1.15 \ > > + # icon-shim:2.0.3 \ > > + # javadoc:1.5 \ > > + # jdk-tool:1.2 \ > > + # junit:1.27 \ > > + # ldap:1.20 \ > > + # mailer:1.23 \ > > + # matrix-auth:2.3 \ > > + # matrix-project:1.14 \ > > + # pam-auth:1.4 \ > > + # pegdown-formatter:1.3 \ > > + # structs:1.17 \ > > + # windows-slaves:1.4 > > + > > + # install plugins for latest LTS Jenkins (currently 2.249.3) > > + /usr/local/bin/install-plugins.sh \ > > + script-security \ > > + structs \ > > + workflow-step-api \ > > + workflow-api \ > > + junit \ > > + scm-api \ > > + ant \ > > + antisamy-markup-formatter \ > > + bouncycastle-api \ > > + command-launcher \ > > + description-setter \ > > + display-url-api \ > > + external-monitor-job \ > > + greenballs \ > > + icon-shim \ > > + javadoc \ > > + jdk-tool \ > > + mailer \ > > + matrix-auth \ > > + matrix-project \ > > + pam-auth \ > > + pegdown-formatter \ > > + windows-slaves > > + > > + cp frontend-install/plugins/flot-plotter-plugin/flot.hpi $JENKINS_HOME/plugins/flot.jpi > > + > > + # make the mod.js symlink well after flot is installed > > + service jenkins start && sleep 30 && \ > > + rm $JENKINS_HOME/plugins/flot/flot/mod.js && \ > > + ln -s /fuego-core/scripts/mod.js $JENKINS_HOME/plugins/flot/flot/mod.js > > + > > + chown -R jenkins:jenkins $JENKINS_HOME/ > > +fi > > + > > +# ============================================================================== > > +# get ttc script and helpers > > +# ============================================================================== > > +git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc > > +/usr/local/src/ttc/install.sh /usr/local/bin > > +perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc > > + > > +# ============================================================================== > > +# Serial Config > > +# ============================================================================== > > +if [ $nojenkins -eq 0 ]; then > > + /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; chown -R jenkins /usr/local/src/serio ; cp > > /usr/local/src/serio/serio /usr/local/bin/ ; ln -s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh' > > + /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; chown -R jenkins /usr/local/src/serlogin ; > > cp /usr/local/src/serlogin/serlogin /usr/local/bin' > > +else > > + /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; cp /usr/local/src/serio/serio /usr/local/bin/ ; ln - > > s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh' > > + /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; cp /usr/local/src/serlogin/serlogin > > /usr/local/bin' > > +fi > > +cp frontend-install/0001-Fix-host-parsing-for-serial-device-with-in-name.patch \ > > + frontend-install/0002-Output-data-from-port-during-command-execution.patch \ > > + /tmp/ > > +/bin/bash -c 'patch -d /usr/local/src/serio -p1 </tmp/0001-Fix-host-parsing-for-serial-device-with-in-name.patch ; \ > > + patch -d /usr/local/src/serio -p1 </tmp/0002-Output-data-from-port-during-command-execution.patch ;' > > + > > + > > +# ============================================================================== > > +# get ebf script and helpers > > +# ============================================================================== > > +apt-get -q=2 -V --no-install-recommends install jq > > +git clone https://github.com/TimesysGit/board-farm-rest-api.git /usr/local/src/board-farm-rest-api > > +cp /usr/local/src/board-farm-rest-api/cli/ebf /usr/local/bin/ebf > > +chmod a+x /usr/local/bin/ebf > > + > > +# ============================================================================== > > +# fserver > > +# ============================================================================== > > +git clone https://github.com/tbird20d/fserver.git /usr/local/lib/fserver > > +ln -s /usr/local/lib/fserver/start_local_bg_server /usr/local/bin/start_local_bg_server > > + > > +# ============================================================================== > > +# ftc post installation > > +# ============================================================================== > > +ln -s /fuego-core/scripts/ftc /usr/local/bin/ > > +cp fuego-core/scripts/ftc_completion.sh /etc/bash_completion.d/ftc > > +echo ". /etc/bash_completion" >> /root/.bashrc > > + > > +# ============================================================================== > > +# Lava > > +# ============================================================================== > > +ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin > > +ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin > > diff --git a/install.sh b/install.sh > > index a5a90b0..259c887 100755 > > --- a/install.sh > > +++ b/install.sh > > @@ -1,13 +1,13 @@ > > #!/bin/bash > > # > > -# install.sh [--help] [--priv] [--no-cache] [--nojenkins] [<image_name>] [<port>] > > +# install.sh [--help] [--priv] [--no-cache] [--nojenkins] [<image_name>] [<port>] [<distribution>] [<suit>] > > # > > > > # $1 is the exit code after usage is shown > > usage() { > > exit_code="$1" > > cat <<HERE > > -Usage: install.sh [--help] [--priv] [--no-cache] [--nojenkins] [<image_name>] [<port>] > > +Usage: install.sh [--help] [--priv] [--no-cache] [--nojenkins] [<image_name>] [<port>] [<distribution>] [<suit>] > > > > Create the docker image and container with the Fuego test distribution. > > If no <image_name> is provided, the image will be named 'fuego'. > > @@ -34,7 +34,7 @@ fi > > > > priv=0 > > NOCACHE="" > > -dockerfile="Dockerfile" > > +NOJENKINS="" > > > > POSITIONAL=() > > while [[ $# -gt 0 ]] ; do > > @@ -48,7 +48,7 @@ while [[ $# -gt 0 ]] ; do > > shift > > ;; > > --nojenkins) > > - dockerfile="Dockerfile.nojenkins" > > + NOJENKINS="--nojenkins" > > shift > > ;; > > *) > > @@ -60,6 +60,8 @@ set -- "${POSITIONAL[@]}" # restore positional arguments > > > > image_name=${1:-fuego} > > jenkins_port=${2:-8090} > > +distribution=${3:-debian} > > +suite=${4:-stretch} > > > > container_name="${image_name}-container" > > > > @@ -79,7 +81,7 @@ fi > > > > set -e > > > > -source fuego-host-scripts/docker-build-image.sh $NOCACHE ${image_name} ${jenkins_port} ${dockerfile} > > +source fuego-host-scripts/docker-build-image.sh $NOCACHE ${image_name} ${jenkins_port} ${distribution} ${suite} ${NOJENKINS} > > if [ "$priv" == "0" ]; then > > fuego-host-scripts/docker-create-container.sh ${image_name} ${container_name} > > else > > diff --git a/uninstall-fuego.sh b/uninstall-fuego.sh > > new file mode 100644 > > index 0000000..b899e7c > > --- /dev/null > > +++ b/uninstall-fuego.sh > > @@ -0,0 +1,47 @@ > > +#!/bin/bash > > + > > +# Remove Jenkins plugins > > +[ -d /usr/share/jenkins/ref/plugins/ ] && rm -rf /usr/share/jenkins/ref/plugins > > + > > +# uninstall Jenkins > > +service jenkins stop > > +dpkg -P jenkins > > + > > +[ -f /usr/local/bin/install-plugins.sh ] && rm -f /usr/local/bin/install-plugins.sh > > +[ -f /usr/local/bin/jenkins-support ] && rm -rf /usr/local/bin/jenkins-support > > +[ -f /usr/local/bin/clitest ] && rm -rf /usr/local/bin/clitest > > +[ -d /var/lib/jenkins ] && rm -rf /var/lib/jenkins > > +sed -i '/^JENKINS_PORT=/d' /etc/environment > > + > > +# uninstall ttc script > > +[ -d /usr/local/src/ttc ] && rm -rf /usr/local/src/ttc > > + > > +# uninstall ebf script > > +[ -d /usr/local/src/board-farm-rest-api ] && rm -rf /usr/local/src/board-farm-rest-api > > +[ -f /usr/local/bin/ebf ] && rm -f /usr/local/bin/ebf > > + > > +# uninstall serial config > > +[ -d /usr/local/src/serio ] && rm -rf /usr/local/src/serio > > +[ -f /usr/local/bin/serio ] && rm -rf /usr/local/bin/serio > > +[ -f /usr/local/bin/sercp ] && rm -rf /usr/local/bin/sercp > > +[ -f /usr/local/bin/sersh ] && rm -rf /usr/local/bin/sersh > > +[ -d /usr/local/src/serlogin ] && rm -rf /usr/local/src/serlogin > > +[ -f /usr/local/bin/serlogin ] && rm -rf /usr/local/bin/serlogin > > + > > +# uninstall fserver > > +[ -d /usr/local/lib/fserver ] && rm -f /usr/local/bin/fserver > > +[ -f /usr/local/bin/start_local_bg_server ] && rm -rf /usr/local/bin/start_local_bg_server > > + > > +# uninstall ftc > > +[ -f /usr/local/bin/ftc ] && rm -rf /usr/local/bin/ftc > > +[ -f /etc/bash_completion.d/ftc ] && rm -rf /etc/bash_completion.d/ftc > > + > > +# remove lava target files > > +[ -f /usr/local/bin/fuego-lava-target-setup ] && rm -rf /usr/local/bin/fuego-lava-target-setup > > +[ -f /usr/local/bin/fuego-lava-target-teardown ] && rm -rf /usr/local/bin/fuego-lava-target-teardown > > + > > +# remove links > > +[ -d /fuego-ro ] && rm /fuego-ro > > +[ -d /fuego-rw ] && rm /fuego-rw > > +[ -d /fuego-core ] && rm /fuego-core > > + > > -- > > 2.20.1 > > > > Overall - this looks like a good start. But I'd like to see the duplicate sections of install-buster.sh > and install-stretch.sh put into a single location "install-debian-common.sh". Also, I'd like to see > the non-debian stuff (ttc, serio, ebf, etc.) put into "install-common.sh". > > Can you please re-do this with that refactoring in mind? > > Thanks, > -- Tim