[Fuego] [PATCH] Docs: Convert Fuego tbwiki pages into .rst files

From: Pooja More <[email protected]>

Following pages are converted:
Docker_Tips
Troubleshooting_Guide
Updating_Fuego

Signed-off-by: Pooja More <[email protected]>
---
 docs/rst_src/Docker_Tips.rst           | 176 ++++++++++++++++++++++++++++
 docs/rst_src/Troubleshooting_Guide.rst | 204 +++++++++++++++++++++++++++++++++
 docs/rst_src/Updating_Fuego.rst        | 154 +++++++++++++++++++++++++
 3 files changed, 534 insertions(+)
 create mode 100644 docs/rst_src/Docker_Tips.rst
 create mode 100644 docs/rst_src/Troubleshooting_Guide.rst
 create mode 100644 docs/rst_src/Updating_Fuego.rst

diff --git a/docs/rst_src/Docker_Tips.rst b/docs/rst_src/Docker_Tips.rst
new file mode 100644
index 0000000..5c9b200
--- /dev/null
+++ b/docs/rst_src/Docker_Tips.rst
@@ -0,0 +1,176 @@
+#############
+Docker Tips
+#############
+
+Here are some tips for using docker with Fuego:
+
+============
+Starting
+============
+
+After the container is created, you should start it by running:
+
+::
+
+  ./start.sh
+
+This calls:
+
+::
+
+  ./fuego-host-scripts/docker-start-containter.sh
+
+With no arguments, this will start the container named
+``fuego-container``.  However, if you have a multiple fuego containers,
+or have a container with a different name, you can specify the
+container name on the start.sh command line.
+
+Special privileged container
+=================================
+
+To run the container in a special privileged mode that allows access
+to host USB devices (needed for accessing Android targets and
+USB-SERIAL devices), create it using the "--priv" command line option
+to ``./install.sh``:
+
+::
+
+  ./install.sh --priv
+
+This will call
+``./fuego-host-scripts/docker-create-usb-privileged-container.sh``
+to create a container with extra privileges to certain host devices.
+
+This script includes a set of devices that the container is granted
+access to.  However, you may need to add a new device to the list.  To
+add a new device to be accessible inside the docker container, please
+edit the following line in ``docker-create-usb-privileged-container.sh``:
+
+::
+
+ CONTAINER_ID=`sudo docker create -it --privileged -v /dev/bus/usb:/dev/bus/usb -v /dev/ttyACM0:/dev/ttyACM0 ... --net="host" fuego`
+
+
+With above, only "ttyUSBx" and "ttyACM0" will be detected and accessible inside
+the docker container.
+
+.. note::
+
+   As of February, 2017, this script was in the next branch of the fuego repository.
+
+
+============================
+Operations while running
+============================
+
+ * Show the running docker ID
+
+    * sudo docker ps
+
+ * Execute a command in the container
+
+    * docker exec <id> <some_command>
+
+ * Attach another shell inside the container
+
+    * sudo docker exec -i -t <id> bash
+    * There is a helper script called 'fuegosh' which can be used to get a shell
+      inside a currently running Fuego container
+
+      * see fuego-core/scripts/fuegosh
+
+ * Access docker container using ssh
+
+    * ssh user@<ip_addr> -p 2222
+    * sshd is running on 2222 in the container, if the default sshd_config is used
+
+ * Copy files to the container
+
+    * docker cp foo <id>:/path/to/dest
+
+ * Copy files from the container
+
+    * docker cp <id>:/path/to/src/foo bar
+
+===========
+Exiting
+===========
+
+To exit the docker container, just exit the primary shell that started
+with the container was started.
+
+==============
+Persistence
+==============
+
+The Fuego container uses docker bind mounts so that some files persist
+in the host filesystem, even when the container is not running.
+
+In the host system, these are under ``fuego-ro``, ``fuego-rw`` and
+``fuego-core`` in the directory where the container was created.
+
+Here are some files that persist:
+
+ * ``fuego-ro/boards*`` - for board definition files
+ * ``fuego-ro/conf/ttc.conf`` - for use with ttc targets
+ * ``fuego-ro/toolchains`` - this is where toolchains and SDKs can be installed
+ * ``fuego-ro/toolchains/tools.sh`` - this file has the multiplexor for the different
+   toolchains (on the PLATFORM variable)
+ * ``fuego-rw/logs`` - this has logs from executed test runs
+ * ``fuego-rw/work``
+ * ``fuego-rw/buildzone`` - this is where test programs are built
+ * ``fuego-rw/test`` - place where the board 'docker' places test materials
+
+===================================================
+How to determine if you're inside the container
+===================================================
+
+ * grep -q docker /proc/1/cgroup ; echo $?
+
+   * Will be 0 if inside the container, 1 if on host
+
+==========================
+Cleaning up old images
+==========================
+
+I build lots of docker images, and they leave lots of data around.
+
+ * **docker ps -a**  - show docker containers on your system, and their images
+ * **docker images** - show images on your system, and their age and size
+ * **docker rmi <id>** - remove an image (you must remove any containers using this image first)
+ * **docker rm <id>** - remove a container
+
+====================================================
+Copy/Replace a file into a non-running container
+====================================================
+
+Background:
+
+Consider a case where you make some changes to
+``/etc/default/jenkins``
+file when your container is running, and then you restart the
+container. Unfortunately your container may not start because of an
+issue in the ``/etc/default/jenkins`` file. How do you fix it as the
+container itself is not running?
+
+Solution:
+
+Get the container id (of the non-running container) via
+
+::
+
+  $ 'docker ps -a' command
+
+
+Replace the faulty file with original/corrected one via 'docker cp'
+command as shown in the example below.
+
+::
+
+  $ sudo docker cp jenkins 6b4e6e63rfg7:/etc/default/
+
+
+where '6b4e6e63rfg7' is the container id of the non-running container
+
+Now you will able to start the docker container successfully.
+
diff --git a/docs/rst_src/Troubleshooting_Guide.rst b/docs/rst_src/Troubleshooting_Guide.rst
new file mode 100644
index 0000000..eb2b251
--- /dev/null
+++ b/docs/rst_src/Troubleshooting_Guide.rst
@@ -0,0 +1,204 @@
+######################
+Troubleshooting Guide
+######################
+
+This page describes problems encountered using Fuego, and their solutions.
+
+.. note::
+
+   for Editors: please put each issue in it's own page section.
+
+
+================
+Installation
+================
+
+Problem with default Jenkins port
+======================================
+
+Fuego has Jenkins default to using port 8090 on the host system.
+However, if you have something else already running on port 8090, you
+may wish to change this.
+
+You can change the Jenkins port during installation of Fuego,
+using an argument to the install.sh script.  For example,
+to install Fuego with Jenkins configured to use port 9999, use
+the following command during installation:
+
+::
+
+  $ ./install.sh fuego 9999
+
+
+To change the Jenkins port for an already-built Fuego container,
+start the container, and inside the container edit the file:
+
+ * /etc/default/jenkins
+
+Change the line that says: HTTP_PORT=8090
+
+Change to port to whatever your like.
+
+Also, check the line that defines JENKINS_ARGS.  Mine looked like this:
+
+::
+
+  JENKINS_ARGS="--webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego"
+
+
+Change this line to read as follows:
+
+::
+
+  JENKINS_ARGS="--webroot=/var/cache/jenkins/war --httpPort=$HTTP_PORT --prefix=/fuego"
+
+Then restart Jenkins:
+
+ * $ service jenkins restart
+
+
+Problem creating docker file
+================================
+
+Make sure you are running on a 64-bit version of the Linux kernel on
+your host machine.
+
+Problem starting Jenkins after initial container creation
+==============================================================
+
+Doug Crawford reported a problem starting Jenkins in the container
+after his initial build.
+
+
+::
+
+  $ sudo ./docker-create-container.sh
+  Created JTA container 6a420f901af7847f2afa3100d3fb3852b71bc65f92aecd13a9aefe0823d42b77
+  $ sudo ./docker-start-container.sh Starting JTA container
+  6a420f901af7847f2afa3100d3fb3852b71bc65f92aecd13a9aefe0823d42b77
+  [....] Starting Jenkins Continuous Integration Server: jenkinssu: System error  failed!
+  [ ok ] Starting OpenBSD Secure Shell server: sshd.
+  [ ok ] Starting network benchmark server.
+
+
+The error string is jenkinssu: System error
+
+Takuo Kogushi provides the following response:
+
+I had the same issue. I did some search in the net and found it is not
+a problem of fuego itself.  As far as I know there are two
+workarounds;
+
+ 1) Rebuild and install libpam with --disable-audit option (in the container) or
+ 2) Modify ``docker-create-container.sh`` to add --pid="host" option to docker
+    create command
+
+Here is a patch provided by Koguchi-san:
+
+::
+
+  diff --git a/fuego-host-scripts/docker-create-container.sh b/fuego-host-scripts/docker-create-container.sh
+  index 2ea7961..24663d6 100755
+  --- a/fuego-host-scripts/docker-create-container.sh
+  +++ b/fuego-host-scripts/docker-create-container.sh
+  @@ -7,7 +7,7 @@ while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symli  done  DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
+
+  -CONTAINER_ID=`sudo docker create -it -v $DIR/../userdata:/userdata --net="host" fuego`
+  +CONTAINER_ID=`sudo docker create -it -v $DIR/../userdata:/userdata --pid="host" --net="host" fuego`
+   CONTAINER_ID_FILE="$DIR/../last_fuego_container.id"
+   echo "Created Fuego container $CONTAINER_ID"
+   echo $CONTAINER_ID > $DIR/../last_fuego_container.id
+
+Actually I have not tried the first one and do not know if there is
+any side effects for the second.
+
+
+This may be related to this docker bug:
+`<https://github.com/docker/docker/issues/5899>`_
+
+Problem with bad port on ssh connection
+=============================================
+
+``ovgen.py`` doesn't parse SSH_PORT from:
+
+ * ``/home/jenkins/fuego/engine/scripts/overlays/base/base-board.fuegoclass``
+
+because it is missing double quotes.
+
+The symptom is the following:
+
+You see the following in the test log for some test you tried to run:
+
+::
+
+  +++ sshpass -e ssh -o ServerAliveInterval=30 -o
+  StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
+  ConnectTimeout=15 -p [email protected] true
+  Bad port '[email protected]'
+  +++ abort_job 'Cannot connect to 10.0.0.1 via ssh'
+  +++ set +
+
+The error string here is "Bad port '[email protected]'"
+
+This occurs because the port is empty.  It should have been passed to
+the ssh command after the '-p' command line option, but since it is
+empty, it uses the account-name@address combination as the argument.
+
+The reason it is empty is that a bug in the ``base-board.fuegoclass`` is
+missing the double-quotes.
+
+This is fixed in the fuegotest repository with the following commit:
+
+ * `<https://bitbucket.org/fuegotest/fuego-core/commits/abb2e7161ba66017a267c09897e5db4d938ab214>`_
+
+
+===========
+General
+===========
+
+Timeout executing ssh commands
+====================================
+
+In some cases, the ssh command used by Fuego takes a very long time to
+connect.  There is a timeout for the ssh commands, specified as 15
+seconds in the cogent repository and 30 seconds in the fuegotest
+repository.
+
+The timeout for ssh commands is specified in the file
+
+ * ``/home/jenkins/fuego/engine/scripts/overlays/base/base-params.fuegoclass``
+
+You can change ConnectTimeout to something longer by editing the file.
+
+::
+
+  FIXTHIS - make ConnectTimeout for ssh connections a board-level test variable
+
+ssh commands taking a long time
+=====================================
+
+Sometimes, even if the command does not time, the SSH operations
+on the target take a very long time for each operation.
+
+The symptom is that when you are watching the console output for a
+test, the test stops at the point of each SSH connection to the
+target.
+
+One cause of long ssh connection times can be that the target ssh
+server (sshd) is configured to do DNS lookups on each inbound
+connection.
+
+To turn this off, on the target, edit the file:
+
+ * ``/etc/ssh/sshd_config``
+
+and add the line:
+
+::
+
+  UseDNS no
+
+
+This line can be added anywhere in the file, but I recommend adding
+it right after the UsePrivilegeSeparation line (if that's there).
+
diff --git a/docs/rst_src/Updating_Fuego.rst b/docs/rst_src/Updating_Fuego.rst
new file mode 100644
index 0000000..a0c675a
--- /dev/null
+++ b/docs/rst_src/Updating_Fuego.rst
@@ -0,0 +1,154 @@
+
+##############
+Updating Fuego
+##############
+
+Here are some notes about updating Fuego, and some tips for when it
+might be necessary and when not.
+
+=================
+Introduction
+=================
+
+Fuego consists of two repositories - ``fuego`` and ``fuego-core``.  The
+contents of the ``fuego`` repository are primarily focused on the
+creation and management of the docker container, and the ``fuego`` Linux
+distribution inside it, and on global configuration for the Fuego
+system (including fuego configuration, board definitions and
+toolchains).
+
+The ``fuego-core`` repository has the core engine of Fuego, including
+the 'ftc' command, the core scripts, and the Fuego tests themselves
+(including source code in many cases).  This repository may get
+updates more frequently as tests are added or as the core framework of
+Fuego is fixes, extended and enhanced.
+
+One of the goals of having separate repositories (which are a bit of a
+pain to keep synchronized) is to make it possible to update the test
+framework and tests in ``fuego-core`` without having to update the
+``fuego`` repository or rebuild the docker container.
+
+===================
+Upgrading Fuego
+===================
+
+'pull'ing fuego-core
+==========================
+
+In many cases, you can upgrade the Fuego test framework merely by
+doing a 'git pull' on the ``fuego-core`` repository, on your host
+machine.  This will pull new features and new tests into your
+``fuego-core`` repository.  The new scripts, tools and tests in this
+repository will become visible inside your container under the
+directory: ``/fuego-core``.
+
+You can even do this while the docker container is running.  However,
+you should not do a 'git pull' on ``fuego-core`` while a test is
+running.  That might change the scripts and tools in the middle of a
+test, which would lead to unpredictable behavior.
+
+If you have local modifications to existing tests, you may need to
+'git stash' those modifications, and merge them with the new code, in
+order to proceed with the update.  New tests that you have created
+should be in their own directories, and should not be affected by a
+'git pull'.
+
+For Fuego releases, we strive to preserve backwards compatibility with
+core APIs, so that existing tests will not break when a new core
+framework is installed.  However, in some rare cases this is
+unavoidable.  These situations will be noted in the release notes for
+a particular release.  See :ref:`Releases` for links to information about
+each release.  In particular, a large amount of framework refactoring
+occurred in the 1.1 and 1.2 releases.
+
+In rare cases, which will be announced on the Fuego mailing list and
+noted in the release notes, a change will be made in the framework
+that is incompatible with the current format of nodes, jobs or builds,
+as held by the Jenkins server.  In this case it becomes impossible to
+use pre-existing test data with the new framework, and it may become
+necessary to shelve that data and start a new instance (either of the
+docker container or of Jenkins).
+
+
+'pull'ing fuego
+=====================
+
+As new features are added to Fuego, sometimes it becomes necessary to
+alter the way the docker container is built, or to add additional
+Debian packages to the 'fuego' distribtion of Linux that is inside the
+container.  These types of changes sometimes require that a new
+container be built with the new attributes.  However, building a new
+container eliminates the Jenkins node, job, and build data that is in
+the current container.  For this reason, it is desirable to avoid
+rebuilding the container, if possible.
+
+In many cases it is possible to pull a new ``fuego`` repository and NOT
+have to rebuild the container, by just implementing manually whatever
+was changed.  For example, the most common change to ``fuego`` is in the
+Dockerfile, to add a new package to the fuego distribution of Linux
+inside the container.  While you could rebuild the container from
+scratch after such a change, you can also manually just do an 'apt-get
+install <new-package>' inside the running docker container.  This will
+provide the same functionality for your existing docker container that
+a new one would have (providing that new library, tool or feature).
+
+In some cases, it is possible to implement other changes as well.  For
+example, if a tool is placed in a new location by an updated
+Dockerfile, then you could manually move the tool in your docker
+container, for the same effect.  The details of this operation depend
+on what has changed.  You can do a 'git log' in the 'fuego' repository
+for details about the changes made, and decide if you can effect those
+changes in your existing container, without having to rebuild a new
+one.  If you have any questions, please ask them on the Fuego mailing
+list, and we will try to assist.
+
+
+
+Preserving old containers
+==============================
+
+Please note that you do not have to destroy or remove a container when
+you create a new one.  By convention the fuego docker image is called
+``fuego`` and the fuego docker container is called ``fuego-container``.
+You can specify different names when you create a new image and
+container, but the preferred method of dealing with this is to rename
+the existing image and container, and create new ones with the default
+names.  If you plan to preserve an image and container, you need to
+preserve the ``fuego`` and ``fuego-core`` repositories in their same
+directories, or docker will get confused.  That is, if you want to
+upgrade and create a new docker container, while still preserving the
+old container, you should 'git clone' the repositories to a new
+directory location in your host filesystem.  Note in this case, that
+you should not have both the new container and the old container
+running at the same time, as there will be conflicts over TCP port
+numbers and other host resources. The old test data (from the other
+container) will not be visible along with any new data in the new
+container (ie in the Jenkins interface).  However this does provide a
+mechanism to preserve your data from previous tests.
+
+Note that Fuego 'run' data is outside of the Jenkins directory, and
+stored on the host filesystem in ``fuego-rw/logs``, so this data is always
+available even when the docker container is rebuilt.  You should be
+careful, however, as Jenkins job IDs will be reused, starting from 1,
+for any new jobs executed with a fresh container and instance of
+Jenkins.  These may overwrite the result directories from previous
+runs, if you re-use the same ``fuego/fuego-rw`` directory.  This is yet
+another reason to use new repository directories for a new docker
+container build (and Fuego instance).
+
+
+==================
+Fuego versions
+==================
+
+Please note that this discussion applies more generally to major Fuego
+releases.  For Fuego, and major release is considered one where the
+second digit of the version number is the same (the '1' or '2' in 1.1
+and 1.2).
+
+If an API-incompatible change occurs within a major release, this is
+considered a regression and we will try to fix it.
+
+Our hope is that Fuego is starting to settle down a bit in the 1.2
+release, and that API-incompatible changes will be more rare after
+that release.
-- 
2.7.4


-- 






This
message contains confidential information and is intended only 
for the
individual(s) named. If you are not the intended
recipient, you are 
notified that disclosing, copying, distributing or taking any
action in 
reliance on the contents of this mail and attached file/s is strictly
prohibited. Please notify the
sender immediately and delete this e-mail 
from your system. E-mail transmission
cannot be guaranteed to be secured or 
error-free as information could be
intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain
viruses. The sender therefore does 
not accept liability for any errors or
omissions in the contents of this 
message, which arise as a result of e-mail
transmission.