[Fuego] Fwd: RST_Docs conversion
Pooja Sanjay More <[email protected]> Wed, 16 Sep 2020 12:59:02 +0530
| Newsgroups | dev.linux.lists.fuego |
|---|---|
| Message-ID | <CAO08rzt1OzNS7PfoUtucJrEanvHn34GG6Ca6aRLL0ZDV8LBdAA@mail.gmail.com> |
---------- Forwarded message --------- From: Pooja Sanjay More <[email protected]> Date: Wed, Sep 16, 2020 at 12:40 PM Subject: Re: RST_Docs conversion To: Bird, Tim <[email protected]> Cc: Amit Agarwal <[email protected]>, Srivatsan S < [email protected]> Hi, Below are a few patches attached. I have created the patches for files according to the categorized area mentioned on rst_page_conversion_list. 0001-docs-pages-categorized-as-Explanation.patch -- Contains changes to Architecture.rst, FAQ.rst, OSS_Test_Vision.rst 0001-docs-rst-files-for-pages-categorized-as-for-users.patch --Contains changes to Installing_feugo.rst, Adding_a_board, Adding_a_toolchain, Adding_test_jobs_to_Jenkins, Adding_views_to_Jenkins. 0001-docs-rst-files-for-pages-categorized-as-Tutorials-- Contains changes to Fuego_Quickstart_Guide, Raspberry_Pi_Fuego_Setup, Quick_Setup_Guide, Using_the_qemuarm_target. It is observed that avocado documentation uses a different sphinx theme. Hence, to have the look and feel of avocado, the below theme can be used in sphinx: <https://github.com/readthedocs/sphinx_rtd_theme> Thanks and regards, Pooja More On Mon, Sep 14, 2020 at 10:43 PM Bird, Tim <[email protected]> wrote: > > -----Original Message----- > > From: Pooja Sanjay More > > > > Hi, > > > > As mentioned in the comments to make changes in the fuego repository > itself and commit the same. I'm not able to push the changes to > > fuego repository as it needs permission. Please provide the same for my > credentials: > > I'm not planning on giving you push capability to the main repository (on > bitbucket). > > I'd like the development to be in your own fork of the fuego repository, > with proposed > changes submitted to me as patches on a mailing list. > > What I'm trying to accomplish here is to give you some training on how > patches are > submitted to an upstream project that is primarily e-mail oriented. I am > hoping > that once you get familiar with the infrastructure required and the > methods used > to submit patches this way, you will be able to submit patches to other > projects > (specifically the Linux kernel) in the future. In this regard, Fuego is a > stepping > stone (or training ground, if you will). I am indeed grateful for your > work on > the Fuego documentation, and part of my "payment" back to you will be > to give you experience and mentorship that will help you become more > involved > in other upstream projects. > > > > > > > Mail ID: [email protected] <mailto: > [email protected]> > > user_Name: Pooja_More > > > > Henceforth, I plan to push all changes to the repo and the review can > happen from there on. > > Just commit them to your local repository, and use 'git format-patch' to > create patches to submit > to me via e-mail. If you have questions about this, let me know and we > can go over it > by email or in a conference call. > > > > > Also about index.rst content, as adding files to toctree makes a long > list of headings in a table of content , so purposely kept hidden. Rest > > of the comments will be followed. > > Yes. It does create a really long list. I potentially needs to be > trimmed down to be manageable. > I had hoped that the toctree would show up on the left side of the page as > a navigation with collapsible > headings. > > I'd eventually like the docs to look something like this: > https://avocado-framework.readthedocs.io/en/82.0/ (in terms of layout > navigation - obviously > our content would be different). > > That's how the Avocado docs look on readthedocs.io. It looks different > when built locally. > It appears that for a local build, the left sidebar menus don't expand and > collapse the same > way. Anyway, we don't have anything like that, but I'd like to compare > our doc build with > theirs and see if we can imitate what they are doing. Eventually I'd like > to put Fuego > documentation on readthedocs.io, similar to other projects. > > > > > > > Thanks and Regards, > > Pooja More > > > > On Sat, Sep 12, 2020 at 5:21 AM Bird, Tim <[email protected] <mailto: > [email protected]> > wrote: > > > > > > > -----Original Message----- > > > From: Pooja Sanjay More > > > Hi Tim, > > > > > > Below is the list of pages that have been converted wherein > your earlier feedback about filenames has been taken care of. > > Accordingly, > > > the wiki-naming has been retained. Also there was no mapping > provided for 'Title' and hence added the mapping for 'Title' on > > the 'Markup > > > Mapping' page and the rest of the conventions are followed in > the same manner. > > > > > > Html pages have been built using the "conf.py" file from fuego > repo. > > > > > > The Documentation page contains only hyperlinks so it has been > converted partially. > > > > > > Please find the zip for .rst files and corresponding html pages > and also the image folder. > > > > Receiving zip files to my work email is quite cumbersome to deal > with. Can we please > > do the following going forward? > > - perform the work in the fuego repository, by adding or editing > files in the > > docs/rst_src directory > > - create a commit for the changes > > - e-mail the commit to [email protected] <mailto: > [email protected]> , but also copy the fuego mailing list: > > cc: [email protected] <mailto: > [email protected]> > > - I have a separate e-mail account receiving e-mail from > [email protected] > > <mailto:[email protected]> > > which I use to process patches to Fuego > > - preferably, the commit would be in 'git send-email' format (or > 'git format-patch' format) > > - I prefer to receive commits inline, but if you send them as > attachments I can deal with it > > - it would be nice to get the commits as a series, with one > commit per added file, but > > you could send a commit with multiple files > > - please send them as you work on them, instead of in big > batches. Big batches are > > hard to process, because I need to allocate a large block of > time, instead of just fitting > > in a quick review where I can in my work schedule > > - you shouldn't need to send html, as it can be made from the > sources in the fuego > > docs directory with "make htmldocs" > > - I've already populated the directory fuego/docs/images with > images from the wiki, > > so you don't need to send images (unless you create new ones, > which is currently > > out of scope for the project) > > > > This will help the review flow considerably. > > > > > Converted pages: > > > Artwork > > > Documentation > > > FrontPage > > > Architecture > > > FAQ > > > OSS_Test_Vision > > > Fuego_Quickstart_Guide > > > Raspberry_Pi_Fuego_Setup > > > Quick_Setup_Guide > > > Using_the_qemuarm_target > > > Installing_Fuego > > > Adding_a_board > > > Adding_a_toolchain > > > Adding_test_jobs_to_Jenkins > > > Adding_views_to_Jenkins > > > Adding_or_Customizing_a_Distribution > > > Integration_with_ttc > > > Test_variables > > > Working_with_remote_boards > > > Adding_a_new_test > > > Fuego_naming_rules > > > Using_Batch_Tests > > > Parser_module_API > > > Building_Documentation > > > License_And_Contribution_Policy > > > > > > > > > > > > Thanks and regards, > > > Pooja More > > > > > > > > > This message contains confidential information and is intended > only for the individual(s) named. If you are not the intended > > recipient, you > > > are notified that disclosing, copying, distributing or taking > any action in reliance on the contents of this mail and attached file/s > > is strictly > > > prohibited. Please notify the sender immediately and delete this > e-mail from your system. E-mail transmission cannot be > > guaranteed to be > > > secured or error-free as information could be intercepted, > corrupted, lost, destroyed, arrive late or incomplete, or contain > > viruses. The > > > sender therefore does not accept liability for any errors or > omissions in the contents of this message, which arise as a result of e- > > mail > > > transmission. > > > > > > > > > > This message contains confidential information and is intended only for > the individual(s) named. If you are not the intended recipient, you > > are notified that disclosing, copying, distributing or taking any action > in reliance on the contents of this mail and attached file/s is strictly > > prohibited. Please notify the sender immediately and delete this e-mail > from your system. E-mail transmission cannot be guaranteed to be > > secured or error-free as information could be intercepted, corrupted, > lost, destroyed, arrive late or incomplete, or contain viruses. The > > sender therefore does not accept liability for any errors or omissions > in the contents of this message, which arise as a result of e-mail > > transmission. > > -- This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission.
0001-docs-rst-files-for-pages-categorized-as-Tutorials.patch
(application/x-patch, 25 KB)
From bc37ae224aff74a689e6a467fcef976ad7f45ddf Mon Sep 17 00:00:00 2001 From: Pooja_More <[email protected]> Date: Tue, 15 Sep 2020 23:53:07 +0530 Subject: [PATCH] docs:rst files for pages categorized as Tutorials --- docs/rst_src/Fuego_Quickstart_Guide.rst | 255 ++++++++++++++++++++++++++++++ docs/rst_src/Quick_Setup_Guide.rst | 160 +++++++++++++++++++ docs/rst_src/Raspberry_Pi_Fuego_Setup.rst | 224 ++++++++++++++++++++++++++ docs/rst_src/Using_the_qemuarm_target.rst | 135 ++++++++++++++++ 4 files changed, 774 insertions(+) create mode 100644 docs/rst_src/Fuego_Quickstart_Guide.rst create mode 100644 docs/rst_src/Quick_Setup_Guide.rst create mode 100644 docs/rst_src/Raspberry_Pi_Fuego_Setup.rst create mode 100644 docs/rst_src/Using_the_qemuarm_target.rst diff --git a/docs/rst_src/Fuego_Quickstart_Guide.rst b/docs/rst_src/Fuego_Quickstart_Guide.rst new file mode 100644 index 0000000..a40b308 --- /dev/null +++ b/docs/rst_src/Fuego_Quickstart_Guide.rst @@ -0,0 +1,255 @@ +.. _quickstart: + +####################### +Fuego Quickstart Guide +####################### + + +Running tests from Fuego on your hardware can be accomplished in a few +simple steps. + +*Note: this is the quickstart guide. More details and explanations* +*can be found on the* :ref:`Installing Fuego <installfuego>` page. + +========= +Overview +========= + +The overview of the steps is: + * 1. install pre-requisite software + * 2. download the fuego repository + * 3. build your fuego container + * 4. start the container + * 5. access the interface + * 6. add your board to fuego + * 7. run a test + +These steps are described below. + +=============================== +Install pre-requisite software +=============================== + +To retrieve the fuego software and create the docker image for it, you +need to have git and docker installed on your system. + +On Ubuntu, try the following commands::: + + $ sudo apt install git docker.io + +=================================== +Download, build, start and access +=================================== + +To accomplish the last 6 steps, do the following from a Linux command +prompt::: + + $ git clone https://bitbucket.org/fuegotest/fuego.git + $ cd fuego + $ ./install.sh + $ ./start.sh + $ firefox http://localhost:8090/fuego + + +The fourth step (with ./install.sh) will take some time - about 45 +minutes on my machine. This is the main step that builds the Fuego +docker container. + +When you run the 'start.sh' script, the terminal where this is run +will be placed at a shell prompt, as the root user, inside the docker +container. The container will run until you exit this shell. You +should leave it running for the duration of your testing. + +*Note:If you are experimenting with the unreleased version of Fuego* +*in the'next' branch, then please replace the 'git clone' command in* +*the instructions above with these:* + + * git clone -b next https://bitbucket.org/fuegotest/fuego.git + +On the last step, to access the Fuego interface you can use any +browser - not just Firefox. By default the Fuego interface runs on +your host machine, on port 8090, with URL path "/fuego". + +In your browser, you should see a screen similar to the following: + + .. image:: ../images/fuego-1.1-jenkins-dashboard-new.png + :width: 900 + +We will now add items to Fuego (and this screen) so you can begin +testing. + +========================== +Add your board to fuego +========================== + +To add your own board to Fuego, there are three main steps: + * 1. create a test directory on the target + * 2. create a board file (on the host) + * 3. add your board to the Jenkins interface + +You can find detailed instructions for adding a board at: +:ref:`Adding a board <addingboard>` + +However, here is a quick list of steps you can do to add +a your own board, and a sample 'docker' board to Fuego: + +Create a test directory on your board +======================================== + +Login to your board, and create a directory to use for testing::: + + $ ssh root@your_board + <board>$ mkdir /home/a + <board>$ exit + + +If not using ssh, use whatever method you normally use to +access the board. + +Create board file +=================== + +Now, create your board file. The board file resides in +<fuego-dir>/fuego-ro/boards, and has a filename with the name of the +board, with the extension ".board". + +Do the following: :: + + $ cd fuego-ro/boards + $ cp template-dev.board myboard.board + $ vi myboard.board + + +Edit the variables in the board file to match your board. +Most variables can be left alone, but you will need +to change the IPADDR, TOOLCHAIN and ARCHITECTURE variables, +and set the BOARD_TESTDIR to the directory +you just created above. + +For other variables in the board file, or specifically to use +a different transport than SSH, see more complete instructions +at: :ref:`Adding a board <addingboard>` + +Add boards to the Jenkins interface +==================================== + +Finally, add the board in the Jenkins interface. + +In the Jenkins interface, boards are referred to as "Nodes". + +At the container shell prompt, run the following command: + + * (container prompt)$ ftc add-nodes -b myboard docker + +This will add your board as a node, as well as a 'docker' node in the +Jenkins interface. + +===================== +Install a toolchain +===================== + +If you just wish to run experiment with Fuego, without installing your +own board, you can use the existing 'docker' board. This will run the +tests inside the docker container on your host machine. This requires +little setup, and is intended to let people try Fuego to see how the +interface and tests work, without having to set up their own board. + +If you are running an ARM board with a Debian-based distribution on it, +you can install the Debian ARM cross-compilers into the docker container +with the following command (inside the container): + + * (container prompt)$ /fuego-ro/toolchains/install_armhf_toolchain.sh + +If you are installing a some other kind of board (different +architecture, different root filesystem layout, or different shared +library set), you will need to install a toolchain for your board +inside the docker container. + +Please follow the instructions at: +:ref:`Adding a toolchain <addtoolchain>` to do this. + +====================== +Now select some tests +====================== + +In order to execute tests using the Jenkins interface, you need to +create Jenkins "jobs" for them. You can do this using the 'ftc +add-jobs' command. + +These commands are also executed at the shell prompt in the docker +container. + +You can add jobs individually, or you can add a set of jobs all at +once based on something called a 'testplan'. A testplan is a list of +Fuego tests with some options for each one. You can see the list of +testplans in your system with the following command: + + * (container prompt)$ ftc list-plans + +To create a set of jobs for the 'docker' board on the system, do the +following: + + * (container prompt)$ ftc add-jobs -b docker -p testplan_docker + +To create a set of jobs for your own board (assuming you called it +'myboard'), do the following: + + * (container prompt)$ ftc add-jobs -b myboard -p testplan_smoketest + +The "smoketest" testplan has about 20 tests that exercise a variety of +features in a Linux system. After running these commands, a set of +jobs will appear in the Jenkins interface. + +Once this is done, your Jenkins interface should look something like +this: + +.. image:: ../images/fuego-1.1-jenkins-dashboard-beaglebone-jobs.png + :width: 900 + +============= +Run a test +============= + +To run a job manually, you can do the following: + * Go to the Jenkins dashboard (in the main Jenkins web page), + * Select the job (which includes the board name and the test name) + * Click “Build job” (Jenkins refers to running a test as "building" + it.) + +You can also click on the circle with a green triangle, on the far +right of the line with the job name, in the Jenkins dashboard. + +When the test has completed, the status will be shown by a colored +ball by the side of the test in the dashboard. Blue means success, +red means failure, and grey means the test did not complete (was not +run or was aborted). You can get details about the test run by +clicking on the link in the history list. + +================== +Additional Notes +================== + +Other variables in the board file +================================== + +Depending on the test you want to run, you may need to define some +other variables that are specific to your board or the configuration +of the filesystem on it. Please see +:ref:`Adding a board <addingboard>` for detailed instructions and a +full list of variables that may be used on the target. + +the Jenkins interface +======================== + +See :ref:`Jenkins User Interface <jUsrinterface>` for more screenshots +of the Jenkins web interface. This will help familiarize you with +some of the features of Jenkins, if you are new to using this tool. + +================= +Troubleshooting +================= + +If you have problems installing or using Fuego, please see our +:ref:`Troubleshooting Guide <troubleshootingguide>` + + diff --git a/docs/rst_src/Quick_Setup_Guide.rst b/docs/rst_src/Quick_Setup_Guide.rst new file mode 100644 index 0000000..7a817e7 --- /dev/null +++ b/docs/rst_src/Quick_Setup_Guide.rst @@ -0,0 +1,160 @@ +.. _quickSetupguide: + +################## +Quick Setup Guide +################## + +This page has some really quick setup instructions if you just want to +get a taste of what Fuego is like. This allows you to experiment with +Fuego and try out some tests to see what it looks like and how it +works, without investing a lot of time. + +In this configuration, we will show you how to install Fuego and run a +test on a 'docker' board, which is the docker container where Fuego +itself is running on your host machine. + +Obviously, this is not useful for testing any real hardware. It is +intended only as a demonstration of Fuego functionality. + +For instructions to set up a real board, try the :ref:`Fuego +Quickstart Guide <quickstart>` or the :ref:`Installing Fuego +<installfuego>` page. + +============= +Overview +============= + +The overview of the steps is: + * 1. install pre-requisite software + * 2. download the Fuego repository + * 3. build your Fuego container + * 4. start the container + * 5. add the 'docker' board to Jenkins + * 6. add some sample tests + * 7. access the Jenkins interface + * 8. run a test + +These steps are described below. + +================== +Step details +================== + +To retrieve the Fuego software and create the docker image for it, you +need to have git and docker installed on your system. + +On Ubuntu, try the following commands: :: + + $ sudo apt install git docker.io + + +To download Fuego, and build and start the container, +type the following commands at a Linux shell prompt: :: + + $ git clone https://bitbucket.org/fuegotest/fuego.git + $ cd fuego + $ ./install.sh + $ ./start.sh + + +The third step (with ./install.sh) will take some time - about 45 +minutes on an average Linux machine. This is building the "Fuego" +distribution of Linux (based on Debian) and putting it in the Fuego +docker container. You will also need a connection to the Internet +with fairly decent bandwidth. + +When you run the 'start.sh' script, the terminal will be placed at a +shell prompt, as the root user inside the docker container. The +container will run until you exit this shell. You should leave it +running for the duration of your testing. + +The next steps populate the Jenkins system objects used for testing: + +At the shell prompt inside the container type the following: :: + + <container-prompt># ftc add-node -b docker + <container-prompt># ftc add-jobs -b docker -t Functional.batch_smoketest + + +This will add the 'docker' node (representing the 'docker' Fuego +board) in the Jenkins interface and a small set of tests. + +The "smoketest" batch test has about 20 tests that exercise a variety +of features in a Linux system. After running these commands, a set of +jobs will appear in the Jenkins interface. :: + + + $ firefox http://localhost:8090/fuego + + +To access the Fuego interface (Jenkins) you can use any browser - +not just Firefox. +By default the Fuego interface runs on your host machine, on +port 8090, with URL path "/fuego". + +In your browser, you should see a screen similar to the following: + + .. image:: ../images/fuego-1.1-jenkins-dashboard-beaglebone-jobs.png + :width: 900 + + +================= +Run a test +================= + +To run a job manually, do the following: + * Go to the Jenkins dashboard (in the main Jenkins web page), + * Select the job (which includes the board name and the test name) + * Click “Build job” (Jenkins refers to running a test as "building" + it.) + +A few very simple jobs you might start with are: + * Functional.hello_world + * Benchmark.Drhystone + +You can also start a test manually by clicking on the circle with +a green triangle, on the far right of the line with the job name, +in the Jenkins dashboard. + +When you run a test, the test software is built from source, +sent to the machine (in this case the Fuego docker container), and +executed. Then the results are collected, analyzed, and displayed +in the Jenkins interface. + +When the test has completed, the status will be shown by a colored +ball by the side of the test in the dashboard. Green means success, +red means failure, and grey means the test did not complete +(it was not run or it was aborted). You can get details about the test +run by clicking on the links in the history list. You can see the +test log (what the actual test program +output on the target), by clicking on "testlog". You can see the steps +Fuego took to execute the test by clicking on the "console log" link on +the job page. And you can see the formatted results for a job, and job +details (like start time, test information, board information, and +results) in the 'run.json' file. + +================== +What do do next? +================== + +In order to use Fuego in a real Continuus Integration loop, you need +to do a few things: + + * configure Fuego to work with your own board or product + * customize benchmark thresholds and functional baselines for your + board + * configure Fuego jobs to be triggered after the board is installed + with new software to test + +Fuego does not currently have support for provisioning boards (that +is, installing the "software under test" to the board). Usually, +Fuego users create their own Jenkins job which provisions the board, +and then triggers Fuego jobs, after the new software is installed on +the board. + +See further instructions see the :ref:`Fuego Quickstart Guide +<quickstart>`, :ref:`Adding a board <addingboard>`, :ref:`Adding a +toolchain <addtoolchain>` or the :ref:`Installing Fuego +<installfuego>` page. + + diff --git a/docs/rst_src/Raspberry_Pi_Fuego_Setup.rst b/docs/rst_src/Raspberry_Pi_Fuego_Setup.rst new file mode 100644 index 0000000..7ce2a43 --- /dev/null +++ b/docs/rst_src/Raspberry_Pi_Fuego_Setup.rst @@ -0,0 +1,224 @@ +.. _raspPiFuegoSetup: + +######################### +Raspberry Pi Fuego Setup +######################### + +This is a list of instructions for setting up a Raspberry Pi board for +use with Fuego. These instructions will help you set up the ssh +server, used by Fuego to communicate with the board, and the test +directory on the machine, that Fuego will use to store programs and +files during a test. + +These instructions and the screen shots are for a Raspberry Pi Model 3 +B, running "Raspbian 9 (stretch)". + +This assumes that the Raspberry Pi is already installed, and that +networking is already configured and running. + +============================== +Obtain your network address +============================== + +First, determine what your Pi's network address is. You can see this +by using the command 'ifconfig' in a terminal window, and checking for +the 'inet' address. + +Or, move your mouse cursor over the network icon in the desktop panel +bar. If you leave the mouse there for a second or two, a box will +appear showing information about your current network connection. + +This is what the network information box looks like (in the upper +right corner of this screen shot): + +.. image:: ../images/rpi-network-address.png + :height: 400 + +In this case, my network address is 10.0.1.103. Your address might +start with 192.168, which is common for home or local networks. + +Note this address for use later. + + +========================== +Configure the SSH server +========================== + +In order for other machines to access the Pi remotely, you need to +enable the ssh server. + +This is done by enabling the SSH interface in the Raspberry Pi +Configuration dialog. + +To access this dialog, click on the raspberry logo in the upper right +corner of the main desktop window. Then click on "Preferences", then +on "Raspberry Pi Configuration". In the dialog that appears, click on +the "Interfaces" tab, and on the list of interfaces click on the +"Enable" radio button for the SSH interface. + +Here is the menu: + +.. image:: ../images/rpi-config-menu.png + :height: 400 + + +The configuration dialog looks something like this: + +.. image:: ../images/raspberry-pi-configuration-interfaces-ssh-enable.png + :height: 400 + +Try connecting +================ + +Now, close this dialog, and make sure you can access the Pi using SSH +from your host machine. + + +Try the following command, from your host machine: + + * ssh pi@<''your_address''> + +You will be asked for the password for the 'pi' user account. + +If you successfully log in, you will be at a shell prompt. + +Configure sshd for root access (if applicable) +================================================ + +If you intend to execute Fuego tests as root, you should configure +the SSH server to permit root login. + +This is not recommended on machines that are in production, as it is +a significant security risk. However, for test machines it may be +acceptable to allow root access over ssh. + +To do this, on the Raspberry Pi, with root permissions, edit the file +/etc/ssh/sshd_config and add the following line: :: + + PermitRootLogin yes + + +.. Note:: + + You need to stop and start the ssh server, or reboot the board, + in order to have this new configuration take effect.'' + +======================== +Make a test directory +======================== + +You can use any directory you like for executing tests from Fuego. +However, we recommend using the '/home/fuego' directory. +These instructions indicate how to create that directory. + +If you are using root as your test user account, then create +a directory on the Raspberry Pi for test programs and files. + +If logged in as the 'pi' account, then switch to root (using +something like 'sudo su'), and type, +at a shell prompt: + +``$ mkdir /home/fuego`` + +If you do not wish to use the root account for testing, then +you should create a 'fuego' account for testing. To do this, +use the 'adduser' program. You will be prompted for some +information. + +``$ adduser fuego`` + +Answer the questions, including setting the password for this account. +Remember the password you select, and use that in the board file when +configuring Fuego to access this board. + +This will create the directory ``/home/fuego``. + +The directory ``/home/fuego`` is what should be used in +the board file when configuring Fuego to operate with this board. + +============================ +Add the board file to Fuego +============================ + +Now that you have set up the Raspberry Pi board, add the board file to +Fuego. Assuming your IP address is 10.0.1.17, and you wish to log in +as root, you would create a board file +called "rpi.board", and located at :: + + <fuego-top-dir>/fuego-ro/boards/rpi.board + +with the following contents: :: + + inherit "base-board" + include "base-params" + + IPADDR="10.0.1.17" + LOGIN="root" + BOARD_TESTDIR="/home/fuego" + PASSWORD="put-the-root-password-here" + TOOLCHAIN="debian-armhf" + TRANSPORT="ssh" + ARCHITECTURE="arm" + FUEGO_TARGET_TMP="/home/fuego" + +.. Note:: + + Of course, use the correct root password for your board + +================================== +Add the toolchain to Fuego +================================== + +The Raspbery Pi board is an ARM 32-bit platform. + +Add the 'debian-armhf' toolchain to the Fuego docker container, using +the helper script in the fuego-ro/toolchains directory. + +Inside the Fuego container, run: :: + + $ /fuego-ro/toolchains/install_cross_toolchain.sh armhf + + +======================================== +Add a node and jobs for the board +======================================== + +Inside the Fuego container, run: :: + + $ ftc add-node -b rpi + + +Add the tests you want to run, as Jenkins jobs. You should always +add the "fuego_board_check" test, as a way to automatically determine +that status of a board. + +Inside the Fuego container, run: :: + + $ ftc add-job -b rpi -t Functional.fuego_board_check + + +An easy way to populate Jenkins with a set of tests is to install a +batch test. + +Install the "smoketest" batch test, as follows: + +Inside the Fuego container, run: :: + + $ ftc add-jobs -b rpi -t Functional.batch_smoketest + +============================= +Run a board check +============================= + +To see if everything is set up correctly, execute the test: +Functional.fuego_board_check. + +In the Jenkins interface, select +"rpi.default.Functional.fuego_board_check" and select the menu item +"Build Now" on the left hand side of the screen. + +Wait a few moments for the test to complete. when the test completes, +check the log for the test by clicking on the link to the 'testlog'. + + + diff --git a/docs/rst_src/Using_the_qemuarm_target.rst b/docs/rst_src/Using_the_qemuarm_target.rst new file mode 100644 index 0000000..0978199 --- /dev/null +++ b/docs/rst_src/Using_the_qemuarm_target.rst @@ -0,0 +1,135 @@ +.. _qemuarmtarget: + +########################## +Using the qemuarm target +########################## + +Here are some quick instructions for using the qemuarm target that is +preinstalled in fuego. + +Fuego does not ship with a qemuarm image in the repository, but +assumes that you have built one with the Yocto Project. + +If you don't have one lying around, you will need to build one. Then +you should follow the other steps on this page to configure it to run +with Fuego. + +========================= +Build a qemuarm image +========================= + +Here are some quick steps for building a qemuarm image using the Yocto +Project: (See the `Project Quick Start +<http://www.yoctoproject.org/docs/2.1/yocto-project-qs/ +yocto-project-qs.html|Yocto>`_, +for more information) + +Note that these steps are for Ubuntu. + + * make sure you have required packages for building the software + + * sudo apt-get install gawk wget git-core diffstat unzip texinfo + gcc-multilib build-essential chrpath socat libsdl1.2-dev xterm + + * install the qemu software + + * sudo apt-get install qemu-user + + * download the latest stable release of the Yocto Project + + * git clone git://git.yoctoproject.org/poky + + * configure for building the qemuarm target + + * cd poky + * source oe-init-build-env build-qemuarm build-qemuarm + * edit conf/local.conf + + * Under the comment about "Machine Selection", uncomment the line + 'MACHINE ?= "qemuarm"' + + * build a minimal image (this will take a while) + + * bitbake core-image-minimal + +============================ +Running the qemuarm image +============================ + +You can run the emulator, using the image you just built: + + * run the emulator + + * runqemu qemuarm + + * find the address and ssh port for the image + + * inside the image, do 'ifconfig eth0' + +==================== +Test connectivity +==================== + +From the host, verify that the networking is running: + + * ping 192.168.7.2 + * ssh [email protected] + +Of course, substitute the correct IP address in the commands above. + +Once you know that things are working, directly connecting from the +host to the qemuarm image, make sure the correct values are in the +qemu-arm.board file. You can edit this file inside the fuego +container at /fuego-ro/boards/qemu-arm.board, or on your host in +fuego-ro/boards/qemu-arm.board + +Here are the values you should set: + + * IPADDR="192.168.7.2" + * SSH_PORT=22 + * LOGIN="root" + * PASSWORD="" + +========================== +Test building software +========================== + +It is important to be able to build the test software for the image +you are using with qemu. + +The toolchain used to compile programs for a board is controlled via +the PLATFORM variable in the board file. Currently the qemu-arm.board +file specifies PLATFORM="qemu-armv7hf". Unfortunately, in my own +testing that toolchain won't produce a binary that runs with a +core-image-minimal image from YP Poky. + +You may need to install your Yocto Project SDK into fuego, in order to +successfully build programs for the platforms. + +See :ref:`Adding a toolchain <addtoolchain>` for information about how +to do that. + +Try building a simple program, like hello_world, as a test for the new +system, and see what happens. + + + + + + + + + + + + + + + + + + + + + + -- 2.7.4
0001-docs-rst-files-for-pages-categorized-as-for-users.patch
(application/x-patch, 50.6 KB)
From 083aeb221850a93766875a8055734eb484a77e89 Mon Sep 17 00:00:00 2001 From: Pooja_More <[email protected]> Date: Wed, 16 Sep 2020 09:53:37 +0530 Subject: [PATCH] docs:rst files for pages categorized as for users --- docs/rst_src/Adding_a_board.rst | 239 +++++++++------ docs/rst_src/Adding_a_toolchain.rst | 224 ++++++++++++++ docs/rst_src/Adding_test_jobs_to_Jenkins.rst | 139 +++++++++ docs/rst_src/Adding_views_to_Jenkins.rst | 116 ++++++++ docs/rst_src/Installing_Fuego.rst | 430 +++++++++++++++++++++++++++ 5 files changed, 1049 insertions(+), 99 deletions(-) create mode 100644 docs/rst_src/Adding_a_toolchain.rst create mode 100644 docs/rst_src/Adding_test_jobs_to_Jenkins.rst create mode 100644 docs/rst_src/Adding_views_to_Jenkins.rst create mode 100644 docs/rst_src/Installing_Fuego.rst diff --git a/docs/rst_src/Adding_a_board.rst b/docs/rst_src/Adding_a_board.rst index 2b5246b..82cd197 100644 --- a/docs/rst_src/Adding_a_board.rst +++ b/docs/rst_src/Adding_a_board.rst @@ -11,25 +11,28 @@ Overview To add your own board to Fuego, there are five main steps: - * 1. Make sure you can access the target via ssh, serial or some other connection - * 2. Decide whether to use an existing user account, or to create a user account specifically for testing - * 3. create a test directory on the target + * 1. Make sure you can access the target via ssh, serial or some + other connection + * 2. Decide whether to use an existing user account, or to create a + user account specifically for testing + * 3. create a test directory on the target * 4. create a board file (on the host) * 5. add your board as a node in the Jenkins interface 1 - Set up communication to the target board ============================================== -In order for Fuego to test a board, it needs to communicate with it from -the host machine where Fuego is running. +In order for Fuego to test a board, it needs to communicate with it +from the host machine where Fuego is running. The most common way to do this is to use 'ssh' access over a network connection. The target board needs to run an ssh server, and the host machine connects to it using the 'ssh' client. -The method of setting an ssh server up on a board varies from system to system, -but sample instructions for setting up an ssh server on a raspberry pi are -located here: :ref:`Raspberry Pi Fuego Setup <raspPiFuegoSetup>` +The method of setting an ssh server up on a board varies from system +to system, but sample instructions for setting up an ssh server on a +raspberry pi are located here: +:ref:`Raspberry Pi Fuego Setup <raspPiFuegoSetup>` Another method that can work is to use a serial connection between the host and the board's serial console. Setting this up is outside @@ -41,8 +44,8 @@ package to accomplish this. I On your target board, a user account is required in order to run tests. -The user account used by Fuego is determined by your board file, which you -will configure manually in step 4. You need +The user account used by Fuego is determined by your board file, which +you will configure manually in step 4. You need to decide which account to use. There are three options: * use the root account @@ -51,21 +54,23 @@ to decide which account to use. There are three options: There are pros and cons to each approach. -My personal preference is to use the root account. Several tests in Fuego -require root privileges. If you are working with a test board, that you -can re-install easily, using the 'root' account will allow you to run the -greatest number of tests. However, this should not be used to test machines -that are in production. A Fuego test can run all kinds of commands, and -you should not trust that tests will not destroy your machine (either -accidentally or via some malicious intent). +My personal preference is to use the root account. Several tests in +Fuego require root privileges. If you are working with a test board, +that you can re-install easily, using the 'root' account will allow +you to run the greatest number of tests. However, this should not be +used to test machines that are in production. A Fuego test can run +all kinds of commands, and you should not trust that tests will not +destroy your machine (either accidentally or via some malicious +intent). -If you don't use 'root', then you can either use an existing account, or -create a new account. In most circumstances it is worthwhile to create a new -account dedicated to testing. However, you may not have sufficient privileges -on your board to do this. +If you don't use 'root', then you can either use an existing account, +or create a new account. In most circumstances it is worthwhile to +create a new account dedicated to testing. However, you may not have +sufficient privileges on your board to do this. -In any event, at this point, decide which account you will use for testing -with Fuego, and note it to include in the board file, described later. +In any event, at this point, decide which account you will use for +testing with Fuego, and note it to include in the board file, +described later. 3 - Create test directory on target @@ -99,7 +104,8 @@ Create board file Now, create your board file. The board files reside in <fuego-source-dir>/fuego-ro/boards, and -each file has a filename with the name of the board, with the extension ".board". +each file has a filename with the name of the board, with the +extension ".board". The easiest way to create a board file is to copy an existing one, and edit the variables to match those of your board. The following @@ -130,7 +136,8 @@ with that transport type. * TRANSPORT - this specifies the transport to use with the target - * there are three transport types currently supported: 'ssh', 'serial', 'ttc' + * there are three transport types currently supported: 'ssh', + 'serial', 'ttc' * Most boards will use the 'ssh' or 'serial' transport type * ex: TRANSPORT="ssh" @@ -147,12 +154,13 @@ For targets using ssh: * SSH_PORT * SSH_KEY -IPADDR is the network address of your board. SSH_PORT is the port where -the ssh daemon is listening for connections. By default this is 22, but -you should set this to whatever your target board uses. SSH_KEY is the -absolute path where an SSH key file -may be found (to allow password-less access to a target machine). An -example would be: +IPADDR is the network address of your board. SSH_PORT is the port +where the ssh daemon is listening for connections. By default this is +22, but you should set this to whatever your target board uses. +SSH_KEY is the absolute path where an SSH key file may be found (to +allow password-less access to a target machine). + +An example would be: * SSH_KEY="/fuego-ro/boards/myboard_id_rsa" @@ -164,28 +172,40 @@ For targets using serial: * BAUD * IO_TIME_SERIAL -SERIAL is serial port name used to access the target from the host. This -is the name of the serial device node on the host (or in the container). -this is specified without the /dev/ prefix. Some examples are: +SERIAL is serial port name used to access the target from the host. +This is the name of the serial device node on the host (or in the +container).this is specified without the /dev/ prefix. + +Some examples are: * ttyACM0 * ttyACM1 * ttyUSB0 -BAUD is the baud-rate used for the serial communication, for eg. "115200". +BAUD is the baud-rate used for the serial communication, for eg. +"115200". -IO_TIME_SERIAL is the time required to catch the command's response from the target. This is specified as a decimal fraction of a second, and is usually -very short. A time that usually works is "0.1" seconds. +IO_TIME_SERIAL is the time required to catch the command's response +from the target. This is specified as a decimal fraction of a second, +and is usually very short. A time that usually works is "0.1" +seconds. * ex: IO_TIME_SERIAL="0.1" -This value directly impacts the speed of operations over the serial port, so -it should be adjusted with caution. However, if you find that some operations -are not working over the serial port, try increasing this value (in small increments - 0.15, 0.2, etc.) - -*Note: In the case of TRANSPORT="serial", Please make sure that docker container and Fuego have sufficient permissions to access the specified serial port. You may need to modify docker-create-usb-privileged-container.sh prior to making your docker image, in order to make sure the container can access the ports. Also, if check that the host filesystem permissions on the device node (e.g /dev/ttyACM0 allows access. From inside the container -you can try using the sersh or sercp commands directly, to test access to -the target.* +This value directly impacts the speed of operations over the serial +port, so it should be adjusted with caution. However, if you find +that some operations are not working over the serial port, try +increasing this value (in small increments - 0.15, 0.2, etc.) + +*Note: In the case of TRANSPORT="serial", Please make sure that docker +container and Fuego have sufficient permissions to access the +specified serial port. You may need to modify +docker-create-usb-privileged-container.sh prior to making your docker +image, in order to make sure the container can access the ports. +Also, if check that the host filesystem permissions on the device node +(e.g /dev/ttyACM0 allows access. From inside the container you can try +using the sersh or sercp commands directly, to test access to the +target.* For targets using ttc: @@ -203,44 +223,52 @@ Other parameters * DISTRIB * BOARD_CONTROL -The BOARD_TESTDIR directory is an absolute path in the filesystem on the -target board where the Fuego tests are run. -Normally this is set to something like "/home/fuego", but you can set it to -anything. The user you specify for LOGIN should have access rights to -this directory. +The BOARD_TESTDIR directory is an absolute path in the filesystem on +the target board where the Fuego tests are run. +Normally this is set to something like "/home/fuego", but you can set +it to anything. The user you specify for LOGIN should have access +rights to this directory. -The ARCHITECTURE is a string describing the architecture used by toolchains to build the tests for the target. +The ARCHITECTURE is a string describing the architecture used by +toolchains to build the tests for the target. -The TOOLCHAIN variable indicates the toolchain to use to build the tests -for the target. If you are using an ARM target, set this to "qemu-armv7hf". -This is a default ARM toolchain installed in the docker container, and should -work for most ARM boards. +The TOOLCHAIN variable indicates the toolchain to use to build the +tests for the target. If you are using an ARM target, set this to +"qemu-armv7hf". This is a default ARM toolchain installed in the +docker container, and should work for most ARM boards. -If you are not using ARM, or for some reason the pre-installed arm toolchains -don't work for the Linux distribution installed on your board, then -you will need to install your own SDK or toolchain. In this case, follow -the steps in [[Adding a toolchain]], then come back to this step and set -the TOOLCHAIN variable to the name you used for that operation. +If you are not using ARM, or for some reason the pre-installed arm +toolchains don't work for the Linux distribution installed on your +board, then you will need to install your own SDK or toolchain. +In this case, follow the steps in [[Adding a toolchain]], then come +back to this step and set the TOOLCHAIN variable to the name you used +for that operation. For other variables in the board file, see the section below. The DISTRIB variable specifies attributes of the Linux distribution -running on the board, that are used by Fuego. Currently, this is mainly -used to tell Fuego what kind of system logger the operating system on -the board has. Here are some options that are available: +running on the board, that are used by Fuego. Currently, this is +mainly used to tell Fuego what kind of system logger the operating +system on the board has. + +Here are some options that are available: - * base.dist - a "standard" distribution that implements syslogd-style system logging. It should have the commands: logread, logger, and /var/log/messages - * nologread.dist - a distribution that has no 'logread' command, but does have /var/log/messages - * nosyslogd.dist - a distribution that does not have syslogd-style system logging. + * base.dist - a "standard" distribution that implements syslogd-style + * system logging. It should have the commands: logread, logger, and + * /var/log/messages nologread.dist - a distribution that has no + * 'logread' command, but does have /var/log/messages nosyslogd.dist - + * a distribution that does not have syslogd-style system logging. -If DISTRIB is not specified, Fuego will default to using "nosyslogd.dist". +If DISTRIB is not specified, Fuego will default to using +"nosyslogd.dist". -The BOARD_CONTROL variable specifies the name of the system used to control -board hardware operations. When Fuego is used in conjunction with board -control hardware, it can automate more testing functionality. Specifically, -it can reboot the board, or re-provision the board, as needed for testing. -As of the 1.3 release, Fuego only supports the 'ttc' board control system. -Other board control systems will be introduced and supported over time. +The BOARD_CONTROL variable specifies the name of the system used to +control board hardware operations. When Fuego is used in conjunction +with board control hardware, it can automate more testing +functionality. Specifically, it can reboot the board, or re-provision +the board, as needed for testing. As of the 1.3 release, Fuego only +supports the 'ttc' board control system. Other board control systems +will be introduced and supported over time. Add node to Jenkins interface ================================ @@ -253,13 +281,17 @@ You can see a list of the boards that Fuego knows about using: * $ ftc list-boards -When you run this command, you should see the name of the board you just -created. +When you run this command, you should see the name of the board you +just created. + +You can see the nodes that have already been installed in Jenkins +with: -You can see the nodes that have already been installed in Jenkins with: * $ ftc list-nodes -To actually add the board as a node in jenkins, inside the docker container, run the following command at a shell prompt: +To actually add the board as a node in jenkins, inside the docker +container, run the following command at a shell prompt: + * $ ftc add-nodes -b <board_name> ============================== @@ -267,11 +299,13 @@ Board-specific test variables ============================== The following other variables can also be defined in the board file: + * MAX_REBOOT_RETRIES * FUEGO_TARGET_TMP * FUEGO_BUILD_FLAGS -See :ref:`Variables <variables>` for the definition and usage of these variables. +See :ref:`Variables <variables>` for the definition and usage of these +variables. General Variables ==================== @@ -279,33 +313,36 @@ General Variables File System test variables (SATA, USB, MMC) ============================================= -If running filesystem tests, you will want to declare the Linux device name -and mountpoint path, for the filesystems to be tested. There are three -different device/mountpoint options available depending on the testplan you -select (SATA, USB, or MMC). Your board may have all of these types of -storage available, or only one. +If running filesystem tests, you will want to declare the Linux device +name and mountpoint path, for the filesystems to be tested. There are +three different device/mountpoint options available depending on the +testplan you select (SATA, USB, or MMC). Your board may have all of +these types of storage available, or only one. To prepare to run a test on a filesystem on a sata device, define the SATA device and mountpoint variables for your board. -For example, if you had a SATA device with a mountable filesystem accessible -on device /dev/sdb1, and you have a directory on your target of /mnt/sata -that can be used to mount this device at, you could declare the following -variables in your board file. +For example, if you had a SATA device with a mountable filesystem +accessible on device /dev/sdb1, and you have a directory on your +target of /mnt/sata that can be used to mount this device at, you +could declare the following variables in your board file. * SATA_DEV="/dev/sdb1" * SATA_MP="/mnt/sata" -You can define variables with similar names (USB_DEV and USB_MP, or MMC_DEV and MMC_MP) for USB-based filesystems or MMC-based filesystems. +You can define variables with similar names (USB_DEV and USB_MP, or +MMC_DEV and MMC_MP) for USB-based filesystems or MMC-based +filesystems. LTP test variables ====================== -LTP (the Linux Test Project) test suite is a large collection of tests that -require some specialized handling, due to the complexity and diversity of -the suite. LTP has a large number of tests, some of which may not work correctly on your board. Some of the LTP tests -depend on the kernel configuration or on aspects of your Linux distribution -or your configuration. +LTP (the Linux Test Project) test suite is a large collection of tests +that require some specialized handling, due to the complexity and +diversity of the suite. LTP has a large number of tests, some of which +may not work correctly on your board. Some of the LTP tests depend on +the kernel configuration or on aspects of your Linux distribution or +your configuration. You can control whether the LTP posix test succeeds by indicating the number of positive and negative results you expect for your board. @@ -314,19 +351,23 @@ These numbers are indicated in test variables in the board file: * LTP_OPEN_POSIX_SUBTEST_COUNT_POS * LTP_OPEN_POSIX_SUBTEST_COUNT_NEG -You should run the LTP test yourself once, to see what your baseline values -should be, then set these to the correct values for your board (configuration -and setup). +You should run the LTP test yourself once, to see what your baseline +values should be, then set these to the correct values for your board +(configuration and setup). Then, Fuego will report any deviation from your accepted numbers, for LTP tests on your board. LTP may also use these other test variables defined in the board file: - * FUNCTIONAL_LTP_HOMEDIR - If this variable is set, it indicates where a pre-installed version of LTP resides in the board's filesystem. This can be used to avoid a lengthy deploy phase on each execution of LTP. - * FUNCTIONAL_LTP_BOARD_SKIPLIST - This variable has a list of individual LTP test programs to skip. + * FUNCTIONAL_LTP_HOMEDIR - If this variable is set, it indicates + where a pre-installed version of LTP resides in the board's + filesystem. This can be used to avoid a lengthy deploy phase on + each execution of LTP. + * FUNCTIONAL_LTP_BOARD_SKIPLIST - This variable has a list of + individual LTP test programs to skip. -See :ref:`Functional.LTP <functionalLTP>` for more information about the LTP test, and test -variables used by it. +See :ref:`Functional.LTP <functionalLTP>` for more information about +the LTP test, and test variables used by it. diff --git a/docs/rst_src/Adding_a_toolchain.rst b/docs/rst_src/Adding_a_toolchain.rst new file mode 100644 index 0000000..773d378 --- /dev/null +++ b/docs/rst_src/Adding_a_toolchain.rst @@ -0,0 +1,224 @@ +.. _addtoolchain: + + +################# +Adding Toolchain +################# + +================== +Introduction +================== + +In order to build tests for your target board, you need to install a +toolchain (often in the form of an SDK) into the Fuego system, and let +Fuego know how to access it. + +Adding a toolchain to Fuego consists of these steps: + + * 1. obtain (generate or retrieve) the toolchain + * 2. copy the toolchain to the container + * 3. install the toolchain inside the container + * 4. create a -tools.sh file for the toolchain + * 5. reference the toolchain in the appropriate board file + +======================== +Obtain a toolchain +======================== + +First, you need to obtain a toolchain that will work with your board. +You should have a toolchain that produces software which will work +with the Linux distribution on your board. This is usually obtained +from your build tool, if you are building the distribution yourself, +or from your semiconductor supplier or embedded Linux OS vendor, if +you have been provided the Linux distribution from an external source. + + +Installing a Debian cross-toolchain target +============================================== + +If you are using an Debian-based target, then to get started, you may +use a script to install a cross-compiler toolchain into the container. +For example, for an ARM target, you might want to install the Debian +armv7hf toolchain. You can even try a Debian toolchain with other +Linux distributions. However, if you are not using Debian on your +target board, there is no guarantee that this will produce correct +software for your board. It is much better to install your own SDK +for your board into the fuego system. + +To install a Debian cross toolchain into the container, get to the +shell prompt in the container and use the following script: + + * /fuego-ro/toolchains/install_cross_toolchain.sh + +To use the script, pass it the argument naming the cross-compile +architecture you are using. Available values are: + + * arm64 armel armhf mips mipsel powerpc ppc64el + +Execute the script, inside the docker container, with a single +command-line option to indicate the cross-toolchain to install. You +can use the script more than once, if you wish to install multiple +toolchains. + +Example: + + * # /fuego-ro/toolchains/install_cross_toolchain.sh armhf + +The Debian packages for the specified toolchain will be installed into +the docker container. + +Building a Yocto Project SDK +=============================== + +When you build an image in the Yocto Project, you can also build an +SDK to go with that image using the '-c do_populate_sdk' build step +with bitbake. + +To build the SDK in Yocto Project, inside your yocto build directory +do: + + * bitbake <image-name> -c do_populate_sdk + +This will build an SDK archive (containing the toolchain, header files +and libraries needed for creating software on your target, and put it +into the directory <build-root>/tmp/deploy/sdk/ + +For example, if you are building the 'core-image-minimal' image, you +would execute: :: + + $ bitbake core-image-minimal -c do_populate_sdk + +At this step look in tmp/deploy/sdk and note the name of the sdk +install package (the file ending with .sh). + +=========================================== +Install the SDK in the docker container +=========================================== + +To allow fuego to use the SDK, you need to install it into the fuego +docker container. First, transfer the SDK into the container using +docker cp. + +With the container running, on the host machine do: + + * docker ps (note the container id) + * docker cp tmp/deploy/sdk/<sdk-install-package> <container-id>:/tmp + +This last command will place the SDK install package into the /tmp +directory in the container. + +Now, install the SDK into the container, whereever you would like. +Many toolchains install themselves under /opt. + +At the shell inside the container, run the SDK install script +(which is a self-extracting archive): + + * /tmp/poky-....sh + + * during the installation, select a toolchain installation + location, like: /opt/poky/2.0.1 + +These instructions are for an SDK built by the Yocto Project. Similar +instructions would apply for installing a different toolchain or SDK. +That is, get the SDK into the container, then install it inside the +container. + +============================================== +Create a -tools.sh file for the toolchain +============================================== + +Now, fuego needs to be told how to interact with the toolchain. +During test execution, the fuego system determines what toolchain to +use based on the value of the TOOLCHAIN variable in the board file for +the target under test. The TOOLCHAIN variable is a string that is +used to select the appropriate '<TOOLCHAIN>-tools.sh' file in +/fuego-ro/toolchains. + +You need to determine a name for this TOOLCHAIN, and then create a +file with that name, called $TOOLCHAIN-tools.sh. So, for example if +you created an SDK with poky for the qemuarm image, you might call the +TOOLCHAIN "poky-qemuarm". You would create a file called +"poky-qemuarm-tools.sh" + +The -tools.sh file is used by Fuego to define the environment +variables needed to interact with the SDK. This includes things like +CC, AR, and LD. The complete list of variables that this script +neeeds to provide are described on the page [[tools.sh]] + +Inside the -tools.sh file, you execute instructions that will set the +environment variables needed to build software with that SDK. For an +SDK built by the Yocto Project, this involves setting a few variables, +and calling the environment-setup... script that comes with the SDK. +For SDKs from other sources, you can define the needed variables by +directly exporting them. + +Here is an example of the tools.sh script for poky-qemuarm. This is +in the sample file /fuego-ro/toolchains/poky-qemuarm-tools.sh: :: + + + # fuego toolchain script + # this sets up the environment needed for fuego to use a + # toolchain + # this includes the following variables: + # CC, CXX, CPP, CXXCPP, CONFIGURE_FLAGS, AS, LD, ARCH + # CROSS_COMPILE, PREFIX, HOST, SDKROOT + # CFLAGS and LDFLAGS are optional + # + # this script is sourced by /fuego-ro/toolchains/tools.sh + + POKY_SDK_ROOT=/opt/poky/2.0.1 + export SDKROOT=${POKY_SDK_ROOT}/sysroots/ + armv5e-poky-linux-gnueabi + + # the Yocto project environment setup script changes PATH so + # that python uses + # libs from sysroot, which is not what we want, so save the + # original path + # and use it later + ORIG_PATH=$PATH + + PREFIX=arm-poky-linux-gnueabi + source ${POKY_SDK_ROOT}/environment-setup-armv5e- + poky-linux-gnueabi + + HOST=arm-poky-linux-gnueabi + + # don't use PYTHONHOME from environment setup script + unset PYTHONHOME + env -u PYTHONHOME + + + +=============================================== +Reference the toolchain in a board file +=============================================== + +Now, to use that SDK for building test software for a particular +target board, set the value of the TOOLCHAIN variable in the board +file for that target. + +Edit the board file: + * vi /fuego-ro/boards/myboard.board + +And add (or edit) the line: + + * TOOLCHAIN="poky-qemuarm" + +============ +Notes +============ + +Python execution +================== + +You may notice that some of the example scripts set the environment +variable ORIG_PATH. This is used by the function +[[function_run_python|run_python]] internally to execute the +container's default python interpreter, instead of the interpreter +that was built by the Yocto Project. + + + + + + diff --git a/docs/rst_src/Adding_test_jobs_to_Jenkins.rst b/docs/rst_src/Adding_test_jobs_to_Jenkins.rst new file mode 100644 index 0000000..c2801df --- /dev/null +++ b/docs/rst_src/Adding_test_jobs_to_Jenkins.rst @@ -0,0 +1,139 @@ +.. _addtestjob: + +############################ +Adding test jobs to jenkins +############################ + +Before performing any tests with Fuego, you first need to +add Jenkins jobs for those tests in Jenkins. + +To add jobs to Jenkins, you use the 'ftc' command line tool. + +Fuego comes with over a hundred different tests, and not +all of them will be useful for your environment or testing needs. + +In order to add jobs to Jenkins, you first need to have +created a Jenkins node for the board for which you wish to add +the test. If you have not already added a board definition, +or added your board to Jenkins, please see: +:ref:`Adding a board <addingboard>` + +One your board is defined as a Jenkins node, you can add test +jobs for it. + +There are two ways of adding test jobs, individually, and +using testplans. In both cases, you use the 'ftc add-jobs' +command. + +============================ +Selecting tests or plans +============================ + +The list of all tests that are available can be seen +by running the command 'ftc list-tests'. + +Run this command inside the docker container, by going to +the shell prompt inside the Fuego docker container, and typing :: + + + (container_prompt)$ ftc list-tests + + +To see the list of plans that come pre-configured with Fuego, +use the command 'ftc list-plans'. + + (container_prompt)$ ftc list-plans + + +A plan lists a set of tests to execute. You can examine the +list of tests that a testplan includes, by examining the testplan +file. The testplan files are in JSON format, and are in the +directory ``fuego-core/engine/overlays/testplans``. + +============================ +Adding individual tests +============================ + +To add an individual test, add it using the 'ftc add-jobs' +command. For example, to add the test "Functional.hello_world" +for the board "beaglebone", you would use the following command: :: + + + (container prompt)$ ftc add-job -b beaglebone -t + Functional.hello_world + + +Configuring job options +========================= + +When Fuego executes a test job, several options are available to +control aspects of job execution. These can be configued on the +'ftc add-job' command line. + +The options available are: + + * timeout + * rebuild flag + * reboot flag + * precleanup flag + * postcleanup flag + +See 'ftc add-jobs help' for details about these options and how to +specify them. + +Adding tests for more than one board +====================================== + +If you want to add tests for more than one board at a time, you can do +so by specifying multiple board names after the '-b' option with +'ftc add-jobs'.Board names should be a single string argument, with +individual board names separated by commas. + +For example, the following would add a job for Functional.hello_world +to each of the boards rpi1, rpi2 and beaglebone. :: + + + (container prompt)$ ftc add-job -b rpi1,rpi2,beaglebone -t + Functional.hello_world + + + +================================ +Adding jobs based on testplans +================================ + +A testplan is a list of Fuego tests with some options for each one. +You can see the list of testplans in your +system with the following command: :: + + + (container prompt)$ ftc list-plans + + +To create a set of jobs related to docker image testing, for the +'docker' board on the system, do the following: :: + + + (container prompt)$ ftc add-jobs -b docker -p testplan_docker + + +To create a set of jobs for a board called 'beaglebone', +do the following: :: + + + (container prompt)$ ftc add-jobs -b myboard -p testplan_smoketest + + +The "smoketest" testplan has about 20 tests that exercise a variety of +features on a Linux system. After running these commands, a set of +jobs will appear in the Jenkins interface. + +Once this is done, your Jenkins interface should look something like +this: + +.. image:: ../images/fuego-1.1-jenkins-dashboard-beaglebone-jobs.png + :width: 900 + + + + diff --git a/docs/rst_src/Adding_views_to_Jenkins.rst b/docs/rst_src/Adding_views_to_Jenkins.rst new file mode 100644 index 0000000..8d0a679 --- /dev/null +++ b/docs/rst_src/Adding_views_to_Jenkins.rst @@ -0,0 +1,116 @@ +.. _addingViewstoJenkins: + + +######################### +Adding views to Jenkins +######################### + +It is useful to organize your Jenkins test jobs into "views". These +appear as tabs in the main Jenkins interface. Jenkins always provides +a tab that lists all of the installed jobs, call "All". Other views +that you create will appear on tabs next to this, on the main Jenkins +page. + +You can define new Jenkins views using the Jenkins interface, but +Fuego provides a command that allows you to easily create views for +boards, or for sets of related tests (by name and wildcard), from the +Linux command line (inside the container). + +The usage line for this command is: :: + + Usage: ftc add-view <view-name> [<job_spec>] + + +The view-name parameter indicates the name of the view in Jenkins, and +the job-spec parameter is used to select the jobs which appear in that +view. + +If the job_spec is provided and starts with an '=', then it is +interpreted as one or more specific job names. Otherwise, the view is +created using a regular expression statement that Jenkins uses to +select the jobs to include in the view. + +====================== +Adding a board view +====================== + +By convention, most Fuego users populate their Jenkins interface with +a view for each board in their system (well, for labs with a small +number of boards, anyway). + +The simplest way to add a view for a board is to just specify the +board name, like so: :: + + (container_prompt)$ ftc add-view myboard + + +When no job specification is provided, the 'add-view' command +will create one by prefixing the view name with +wildcards. For the example above, the job spec would consist +of the regular expression ".*myboard.*". + +Customizing regular expressions +================================== + +Note that if your board name is not unique enough, or is a string +contained in some tests, then you might see some test jobs listed that +were not specific to that board. For example, if you had a board name +"Bench", then a view you created with the view-name of "Bench", would +also include Benchmarks. You can work around this by specifying a +more details regular expression for your job spec. + +For example: :: + + (container_prompt)$ ftc add-view Bench "Bench.*" + + +This would only include the jobs that started with "Bench" in the +"Bench" view. Benchmark jobs for other boards would not be included, +since they only have "Benchmark" somewhere in the middle of their job +name - not at the beginning. + +=============================================== +Add view by test name regular expression +=============================================== + +This command would create a view to show LTP results for multiple +boards: :: + + (container_prompt)$ ftc add-view LTP + +This example creates a view for "fuego" tests. This view +would include any job that has the word "fuego" as part of it. +By convention, all Fuego self-tests have part of their name +prefixed with *"fuego_"*. :: + + + (container_prompt)$ ftc add-view fuego ".*fuego_.*" + + +And the following command will show all the batch jobs defined in the +system: :: + + (container_prompt)$ ftc add-view .*.batch + + + +====================== +Add specific jobs +====================== + +If the job specification starts with "=", it is a comma-separated +list of job names. The job names must be complete, including the +board name, spec name and full test name. :: + + + (container_prompt)$ ftc add-view network-tests =docker.default. + Functional.ipv6connect,docker.default.Functional.netperf + + +In this command, the view would be named "network-tests", and it would +show the jobs "docker.default.Functional.ipv6connect" and +"docker.default.Functional.netperf". + + + + diff --git a/docs/rst_src/Installing_Fuego.rst b/docs/rst_src/Installing_Fuego.rst new file mode 100644 index 0000000..482f3c1 --- /dev/null +++ b/docs/rst_src/Installing_Fuego.rst @@ -0,0 +1,430 @@ +.. _installfuego: + +################### +Installing Fuego +################### + +This page describes the steps to install Fuego on your Linux machine. +It includes detailed descriptions of the operations, for both users +and developers. + +If you are interested in a quick outline of steps, please see the +:ref:`Fuego Quickstart Guide <quickstart>` instead. + +=========== +Overview +=========== + +The overview of the steps is: + + * 1. install pre-requisite software + * 2. download the Fuego repository + * 3. build your Fuego container + * 4. start the container + * 5. access the Jenkins interface + +================================= +Install pre-requisite software +================================= + +To retrieve the Fuego software and create the docker image for it, you +need to have git and docker installed on your system. + +On Ubuntu, try the following commands: :: + + $ sudo apt-get install git + $ sudo apt-get install docker.io + + +These commands may be different for other distributions of Linux +(such as Fedora, RedHat, CentOS, Mint, etc.) + +============================= +Overview of remaining steps +============================= + +Steps 2 through 5 of the installation can be performed with the +following Linux commands: :: + + + $ git clone https://bitbucket.org/fuegotest/fuego.git + $ cd fuego + $ ./install.sh + $ ./start.sh + $ firefox http://localhost:8090/fuego + + +These steps and commands will be described in the sections that follow. + +================================= +Install the Fuego repositories +================================= + +The Fuego system is available in 2 git repositories. One +repository is called 'fuego' and the other is 'fuego-core'. +The fuego-core repository should be installed inside the +fuego directory, at the top level of that repository's directory +structure (parallel to fuego-ro and fuego-rw). + +The reason there are two repositories is that the different +repositories hold different pieces of the system, and this allows +for them to be upgraded independently of each other. + +The repositories are currently hosted on bitbucket.org, under +the the 'fuegotest' account. + +Fuego repository +====================== + +The 'fuego' repository has the code and files used to build the +Fuego docker container. It also has the fuego-ro directory, which +has board definition files, various configuration files, +miscellaneous scripts, and other items which are used by Fuego +for container management or other purposes. + +Fuego-core repository +======================== + +The 'fuego-core' repository has the code which implements the +core of the Fuego test execution engine, as well as the pre-packaged +tests included with the system. This includes the overlay generator, +the results parser, the Fuego shell function library, the directory +of tests, and the main Fuego command line tool 'ftc'. + +Downloading the repository +============================ + +You can use 'git clone' to download the main 'fuego' repository, like +so: :: + + + $ git clone https://bitbucket.org/fuegotest/fuego.git + $ cd fuego + + +After downloading the repositories, switch to the 'fuego' directory, +as shown in the example. + +Note that these git commands will download the 'master' branch of the +repository, which is the current main released version of Fuego. + +Downloading a different branch +-------------------------------- + +*NOTE:* If you are experimenting with an unreleased version of Fuego +in the 'next' branch, then please replace the 'git clone' command in +the instructions above with these: :: + + $ git clone -b next https://bitbucket.org/fuegotest/fuego.git + $ cd fuego + + +This uses '-b next' to indicate a different branch to check out during +the clone operation. + +============================ +Create the Fuego container +============================ + +The third step of the installation is to run install.sh to create the +Fuego docker container. While in the 'fuego' directory, +run the script from the current directory, like so: :: + + + $ ./install.sh + + +install.sh uses docker and the Dockerfile in the fuego directory to +create a docker container with the Fuego Linux distribution. + +This operation may take a long time. It takes about 45 minutes on my +machine. This is due to building a nearly complete distribution of +Linux, from binary packages obtained from the Internet. + +This step requires Internet access. You need to make sure that +you have proxy access to the Internet if you are behind a corporate +firewall. + +Please see the section "Alternative Installation Configuratons" below +for other arguments to *install.sh*, or for alternative installation +scripts. + + +Fuego Linux distribution +=========================== + +The Fuego Linux distribution is a distribution of Linux based on +Debian Linux, with many additional packages and tools installed. +These additional packages and tools are required for aspects of Fuego +operation, and to support host-side processes and services needed by +the tests included with Fuego. + +For example, the Fuego distribution includes + * the 'Jenkins' continuous integration server + * the 'netperf' server, for testing network performance. + * the 'ttc' command, which is a tool for board farm management + * the python 'jenkins' module, for interacting with Fuego's Jenkins + instance + * and many other tools, programs and modules used by Fuego and its tests + +Fuego commands execute inside the Fuego docker container, and Fuego +operations initiate in the container, but may access hardware (such as +USB ports, networking, or serial ports) that are outside the container. + +Configuring for 'privileged' hardware access +============================================== + +In many configurations, Fuego can perform its operations using only +network operations. However, depending on the configuration of your +boards, or your lab hardware, and the relationship between your host +and target computers used for testing, you may need to access other +hardware on your host machine. + +To do that, you can create a 'privileged' Fuego container, using +the '--priv' options with install.sh: :: + + $ ./install.sh --priv + + +Customizing the privileged container +------------------------------------- + +Note that using '--priv' causes install.sh to use a different +container creation script. Normally (in the non --priv case), +install.sh uses ``fuego-host-scripts/docker-create-container.sh``. + +When --priv is used, Fuego uses +``fuego-host-scripts/docker-create-usb-privileged-container.sh``. + + +``docker-create-usb-privileged-container.sh`` can be edited, before +running install.sh, to change the set of hardware devices +that the docker container will have privileged access to. + +This is done by adding more bind mount options to the 'docker create' +command inside this script. Explaining exactly how to do this is +outside the scope of this documentation. Please see documentation and +online resources for the 'docker' system for information about this. + +The script currently creates bind mounts for: + * /dev/bus/usb - USB ports, and newly created ports + * /dev/ttyACM0 - serial port 0 + * /dev/ttyACM1 - serial port 1 + * /dev/serial - general serial ports, and newly created ports + +If you experience problems with Fuego accessing hardware on your host +system, you may need to build the Fuego docker container using +additional bind mounts that are specific to your configuration. Do so +by editing docker-create-used-privileged-container.sh, removing the +old container, and re-running './install.sh --priv' to build a new +container with the desired privileges. + +Using an different container name +====================================== + +By default, install.sh creates a docker image called 'fuego' and a +docker container called 'fuego-container'. There are some situations +where it is desirable to use different names. For example, having +different container names is useful for Fuego self-testing. It can +also used to do A/B testing when migrating from one release of Fuego +to the next. + +You can provide a different name for the Fuego image and container, by +supplying one on the command line for install.sh, like so: + + $ ./install.sh my-fuego + + +This would create a docker image named 'my-fuego' and a docker +container named 'my-fuego-container' + + +=========================== +Start the Fuego container +=========================== + +To start the Fuego docker container, use the 'start.sh' script. + + + $ ./start.sh + +Using a different container name +================================== + +By default, start.sh will start the container named 'fuego-container' +(which is the default Fuego docker container name). However, if you +created a different container name, you can specify the name +on the command line, like so: :: + + $ ./start.sh my-fuego-container + + +When you run the 'start.sh', the terminal where the script +is run will be placed at a shell prompt inside the docker container. +The session will be logged in as the root user inside the container. +The container will run until you exit this top-level shell. +Therefore, you should leave it (the shell and the terminal that your +ran 'start.sh' from) running for the duration of your testing. + +========================================= +Access the Fuego Jenkins web interface +========================================= + +Fuego includes a version of Jenkins and a set of plugins as part of +its system. Jenkins is running inside the Fuego docker container. By +default the Fuego Jenkins interface runs on port 8090, with an URL +path "/fuego". + +Here is an example showing use of firefox to access the Jenkins +interface with Fuego :: + + $ firefox http://localhost:8090/fuego + +To access the Fuego interface you can use any browser - not just +Firefox. + +In your browser, you should see a screen similar to the following: + +.. image:: ../images/fuego-1.1-jenkins-dashboard-new.png + :width: 900 + +Note that this web interface is available from any machine that has +access to your host machine via the network. This means that test +operations and test results are available to anyone with access to +your machine. You can configure Jenkins with different security to +avoid this. + +====================================== +Access the Fuego docker command line +====================================== + +For some Fuego operations, it is handy to use the command line (shell +prompt) inside the docker container. In particular, parts of the +remaining setup of your Fuego system involve running the 'ftc' command +line tool. + +Some 'ftc' commands can be run outside the container, but other require +that you execute the command inside the container. + +To start another shell prompt inside the currently running Fuego docker +container, you can use the script 'fuegosh'. This helper script is +located at: ``fuego/fuego-ro/scripts/fuegosh``. +You may find it convenient to copy this script to +a 'bin' directory on your system (either /usr/local/bin or ~/bin) that +is already in your PATH. + +If you run 'fuegosh', it will place you at a shell prompt inside the +Fuego docker container, like so: :: + + $ fuegosh + root@hostname:/# + + +=================== +Remaining steps +=================== + +Fuego is now installed and ready for test operations. However, some +steps remain in order to use it with your hardware. You need to: + + * add one or more hardware boards (board definition files) + * add a toolchain + * populate the Jenkins interface with test jobs + +These steps are described in subsequent sections of this documentation. + +See: + * :ref:`Adding a board <addingboard>` + * :ref:`Adding a toolchain <addtoolchain>` + * :ref:`Adding test jobs to Jenkins <addtestjob>` + +================================================ +Alternative installation configurations +================================================ + +The default installation of Fuego installs the entire Fuego system, +including Jenkins and the Fuego core, into a docker container running +on a host system, which Jenkins running on port 8090. However, it is +possible to install Fuego in other configurations. + +The configuration alternatives that are supported are: + * install using a different TCP/IP port for Jenkins + * install without the Jenkins server + * install directly to your host (not inside a container) + +with a different Jenkins TCP/IP port +=========================================== + +By default the Fuego uses TCP/IP port 8090, but this can be changed to +another port. This can be used to avoid a conflict with a service +already using port 8090 on your host machine, or so that multiple +instances of Fuego can be run simultaneously. + +To use a different port than 8090 for Jenkins, specify it after the +image name on the command line when you run install.sh. Note that this +means that you must specify a Docker image name in order to specify a +non-default port. For example: :: + + + $ ./install.sh fuego 7777 + + +This would install Fuego, with an docker image name of 'fuego', a +docker container name of 'fuego-container', and with Jenkins +configured to run on port 7777 + +without Jenkins +================== + +Some Fuego users have their own front-ends or back-ends, and don't +need to use the Jenkins CI server to control Fuego tests, or visualize +Fuego test results. ``install.sh`` supports the option '--nojenkins' +which produces a docker container without the Jenkins server. This +reduces the overhead of the docker container by quite a bit, for those +users. + +Inside the docker container, the Fuego core is still available. +Boards, toolchains, and tests are configured normally, but the 'ftc' +command line tool is used to execute tests. There is no need to use +any of the 'ftc' functions to manage nodes, jobs or views in the +Jenkins system. 'ftc' is used to directly execute tests using 'ftc +run-test', and results can be queried using 'ftc list-runs' and 'ftc +gen-report'. + +When using Fuego with a different results visualization backend, the +user will use 'ftc put-run' to send the test result data to the +configured back end. + +without a container +=========================== + +Usually, for security and test reproducibility reasons, Fuego is +executed inside a docker container on your host machine. That is, the +default installation of Fuego will create a docker container using all +the software that is needed for Fuego's tests. However, in some +configurations it is desirable to execute Fuego directly on a host +machine (not inside a docker container). A user may have a dedicated +machine, or they may want to avoid the overhead of running a docker +container. + +A separate install script, called 'install-debian.sh' can be used in +place of 'install.sh' to install the Fuego system onto a Debian-based +Linux distribution. + +Please note that installing without a container is not advised unless +you know exactly what you are doing. In this configuration, Fuego will +not be able to manage host-side test dependencies for you correctly. + +Please note also that executing without a container presents a +possible security risk for your host. Fuego tests can run arbitrary +bash instruction sequences as part of their execution. So there is a +danger when running tests from unknown third parties that they will +execute something on your test host that breaches the security, or +that inadvertently damages you filesystem or data. + +However, despite these drawbacks, there are test scenarios (such as +installing Fuego directly to a target board), where this configuration +makes sense. + + + -- 2.7.4
0001-docs-pages-categorized-as-Explanation.patch
(application/x-patch, 35.7 KB)
From 018987133169dd64b6124581a92c6848e2a35ff6 Mon Sep 17 00:00:00 2001 From: Pooja_More <[email protected]> Date: Tue, 15 Sep 2020 17:52:55 +0530 Subject: [PATCH] docs:pages categorized as Explanation --- docs/rst_src/Architecture.rst | 551 +++++++++++++++++++++++++++++++++++++++ docs/rst_src/FAQ.rst | 48 ++++ docs/rst_src/OSS_Test_Vision.rst | 348 +++++++++++++++++++++++++ 3 files changed, 947 insertions(+) create mode 100644 docs/rst_src/Architecture.rst create mode 100644 docs/rst_src/FAQ.rst create mode 100644 docs/rst_src/OSS_Test_Vision.rst diff --git a/docs/rst_src/Architecture.rst b/docs/rst_src/Architecture.rst new file mode 100644 index 0000000..e85d9c7 --- /dev/null +++ b/docs/rst_src/Architecture.rst @@ -0,0 +1,551 @@ +.. _architecture: + + +################ +Architecture +################ + +Fuego consists of a continuous integration system, along with some +pre-packaged test programs and a shell-based test harness, running in +a Docker container.:: + + Fuego = (Jenkins + abstraction scripts + pre-packed tests) + inside a container + +Here's a diagram with an overview of Fuego elements: + +.. image:: ../images/Fuego-architecture.png + :width: 600 + +================= +Major elements +================= + +The major elements in the Fuego architecture are: + + * host system + + * container build system + * fuego container instance + + * Jenkins continuous integration system + + * web-based user interface (web server on port 8090) + * plugins + + * test programs + * abstraction scripts (test scripts) + * build environment (not shown in the diagram above) + + * target system + * web client, for interaction with the system + +============== +Jenkins +============== + +The main interface for Fuego is provided by the Jenkins continuous +integration system. + +The basic function of Jenkins is to automatically launch test jobs, +usually in response to changes in the software. However, it can +launch test jobs based on a variety of triggers, including when a user +manually schedules a test to run. + +Jenkins is too big a system to describe in detail here, but it has +many features and is very popular. It has an ecosystem of plugins for +all kinds of extended functionality, such as integration with +different source code management systems, results plotting, e-mail +notifications of regressions, and more. + +Fuego installs several plugins that are used by various aspects of the +system. + +Jenkins is used to: + * Start tests + * Schedule tests for automatic execution + * Shows test results (particularly over time) + * Flag regressions in test results + +Note that the interface between Jenkins and the test programs is +provided by a set of scripts (one per test, along with a set of +scripts that comprise the core of the system) written in shell script +language. + +The interface between Jenkins and these core scripts is documented at +:ref:`Core interfaces <coreinterface>`. + +This overall architecture means that when items are added into the +system (for example boards, toolchains, or tests), information has to +be supplied to both systems (the Jenkins system and the core script +system). + +========================= +Pre-packaged tests +========================= + +========================= +Abstraction scripts +========================= + +Fuego uses a set of shell script fragments to support abstractions for + + * building test programs from source, + * deploying them to target (installing them) + * executing the tests + * copy files to and from the target + * reading the test log + * parsing the log to determine pass or fail conditions for tests + * parsing the log for results to display in charts + +========================== +Container +========================== + +By default, Fuego runs inside a Docker container. This provides two +benefits: + + * It makes it easy to run the system on a variety of different Linux + distributions + * It makes the build environment for the test programs consistent + +=========================== + Hardware configuration +=========================== + +Fuego supports testing of embedded Linux by fully supporting a +host/target configuration for building, deploying and executing tests. + +Many Linux test systems assume that the system-under-test is a full +desktop or server system, with sufficient horsepower to build tests +and run them locally. Fuego assumes the opposite - that embedded +targets will be underpowered and may not have the normal complement of +utilities and tools available for performing tests + +============================ +Different objects in Fuego +============================ + +It is useful to give an overview of the major objects used in Fuego, +as they will be referenced many times: + +Fuego core objects: + + * board - a description of the device under test + * test - materials forconducting a test + * spec - one or more sets of variables for describing a test variant + * plan - a collection of tests, with additional test settings for + their execution + * run - the results from + a individual execution of a test on a board + +Jenkins objects: + + * node - the Jenkins object corresponding to a Fuego board + * job - a Jenkins object corresponding to a combination of board, + spec, and test + * build - the test results, from Jenkins perspective - corresponding + to a Fuego 'run' + +There are both a front-end and a back-end to the system, and different +names are used to describe the front-end and back-end objects used by +the system, to avoid confusion. In general, Jenkins objects have +rough counterparts in +the Fuego system: + + +------------------+-------------------------------+ + | Jenkins object | corresponds to fuego object | + +==================+===============================+ + | node | board | + +------------------+-------------------------------+ + | job | test | + +------------------+-------------------------------+ + | build | run | + +------------------+-------------------------------+ + +======================= + Jenkins operations +======================= + +How does Jenkins work? + * When the a job is initiated, Jenkins starts a slave process to run + the test that corresponds to that job + * Jenkins records stdout from slave process + * the slave (slave.jar) runs a script specified in the config.xml for + the job + + * this script sources functions from the scripts and overlays + directory of Fuego, and does the actual building, deploying and + test executing + * Also, the script does results analysis on the test logs, and + calls the post_test operation to collect additional information + and clean up after the test + + * while a test is running, Jenkins accumulates the log output from + the generated test script and displays it to the user (if they are + watching the console log) + + * Jenkins provides a web UI for browsing the nodes, jobs, and test + results (builds), and displaying graphs for benchmark data + +====================== + Fuego operations +====================== + +How do the Fuego scripts work? + +====================== +Test execution +====================== + + * each test has a base script, that defines a few functions specific + to that test (see below) + * upon execution, this base script loads additional test variables + and function definitions from other files using something called + the overlay generator + * the overlay generator creates a script containing test variables + for this test run + + * the script is created in the run directory for the test + * the script is called prolog.sh + * the overlay generator is called ovgen.py + + * the base script (with the test variable script sourced into it) + runs on the host, and uses fuego functions to perform different + phases of the test + * for a detailed flow graph of normal test execution see: + :ref:`test execution flow outline <Outline>` + +================================ +test variable file generation +================================ + + * the generator takes the following as input: + + * environment variables passed by Jenkins + * board file for the target (specified with NODE_NAME) + * tools.sh (vars from tools.sh are selected with TOOLCHAIN, from + the board file) + * the distribution file, and (selected with DISTRIB) + * the testplans for the test (selected with TESTPLAN) + * test specs for the test + + * the generator produces the test variable file + * the test variable file is in "run" directory for a test, and has + the name: prolog.sh + * this generation happens on the host, inside the docker container + * the test variable file has functions which are available to be + called by the base test script + +.. image:: ../images/fuego-script-generation.png + :width: 600 + +Input +====== + + * input descriptions: + + * the board file has variables defining attributes of the board, + like the toolchain, network address, method of accessing the + board, etc. + * tools.sh has variables which are used for identifying the + toolchain used to build binary test programs + + * it uses the TOOLCHAIN variable to determine the set of + variables to define + + * a testplan lists multiple tests to run + + * it specifies a test name and spec for each one + + * a spec files hold the a set of variable declarations which are + used by the tests themselves. + These are put into environment variables on the target. + + * ovgen.py reads the plans, board files, distrib files and specs, + and produces + a single prolog.sh file that has all the information for the test + + * Each test in the system has a fuego shell script + + * this must have the same name as the base name of the test: + + * \<base_test_name>.sh + + * Most (but not all) tests have an additional test program + + * this program is executed on the board (the device under test) + * it is often a compiled program, or set of programs + * it can be a simple shell script + * it is optional - sometime the base script can execute the needed + commands for a test without an additional program placed on the + board + + * the base script declares the tarfile for the test, and has + functions for: test_build(), test_deploy() and test_run() + + * the test script is run on host (in the container) + + * but it can include commands that will run on the board + + * tarball has the tarfile + * test_build() has commands (which run in the container) to compile + the test program + * test_deploy() has commands to put the test programs on the target + * test_run() has commands to define variables, execute the actual + test, and log the results. + + * the test program is run on the target + + * this is the actual test program that runs and produces a result + +==================== +fuego test phases +==================== + +A test execution in fuego runs through several phases, some of which +are optional, depending on the test. + +The test phases are: + * pre_test + * build + * deploy + * run + * fetch + * processing + * post_test + +Each of these are described below the diagram. + +.. image:: ../images/fuego-test-phases.png + :width: 600 + +pre_test +============ + +The pre_test phase consists of making sure the target is alive, and +preparing the workspace for the test. In this phase test directories +are created, and the firmware (a string describing the software on the +target) are collected. + +The 'before' syslog is created, and filesystems are synced and buffer +caches dropped, in preparation for any filesystem tests. + +If the function "test_pre_check" is defined in the base script, it is +called to do any pre_test operations, including checking to see if +required variables are set, programs or features are available on the +host or target board, or whether any other test dependencies are met. + +build +========== + +During this phase, the test program source is installed on the host +(inside the container), and the software for the test is actually +built. The toolchain specified by PLATFORM is used to build the +software. + +This phase is split into multiple parts: + * pre_build - build workspace is created, a build lock is acquired + * and the tarball is unpacked + + * :ref:`unpack <unpack>` is called during pre_build + * test_build - this function, from the base script, is called + + * usually this consists of 'make', or 'configure ; make' + * post_build - (empty for now) + +deploy +========= + +The purpose of this phase is to copy the test programs, and any +required supporting files, to the target. + +This consists of 3 sub-phases: + * pre_deploy - cd's to the build directory + * test_deploy - the base script's 'test_deploy' function is called. + + * Usually this consists of tarring up needed files, copying them to + the target with 'put', and then extracting them there + * Items should be placed in the directory + $BOARD_TESTDIR/fuego.$TESTDIR/ directory on the target + * post_deploy - removes the build lock + +run +======= + +In this phase the test program on the target is actually executed. + +This executes the 'test_run' function defined in the base script for +the test, which can consist of anything. Usually, however, it runs +the test program with any needed parameters (as specified by the test +specs and test plans). + +The test execution is usually performed by calling the 'report' +function, which collects the standard out from the command execution +on the target, and saves that as the testlog for the test. Note that +the testlog is saved on the target, but not yet transferred to the +host, yet. + + +post_test +================== + +In this phase, the test log is retrieved (fetched) from the target and +stored on the host. Also in this phase, the board is "cleaned up", +which means that test directories and logs are removed on the target +board, and any leftover processes related to this test that are +running on the board are stopped. + +processing +================== + +In the processing phase of the test, the results from the test log are +evaluated. The test_processing function of the base test script is +called. + +For functional tests: + +Usually, this phase consists of one or more calls to 'log_compare', to +determine if a particular string occurs in the testlog. This phase +determines whether the test passed or failed, and the base test script +indicates this (via it's exit code) to the Jenkins interface. + +For benchmarking tests: + +This phase consists of parsing the testlog, using parser.py, and also +running dataload.py to save data for plot generation. + +Also, a final analysis is done on the system logs is done in this step +(to detect things like Kernel Oopses that occurred during the test). + + + +phase relation to base script functions +============================================================ + +Some of the phases are automatically performed by Fuego, and some end +up calling a routine in the base script (or use data from the base +script) to perform their actions. This table shows the relation +between the phases and the data and routines that should be defined +in the base script. + +It also shows the most common commands utilized by base script +functions for this phase. + + + +------------+-------------------------------+---------------------------+ + | phase | relationship to base script | common operations | + +============+===============================+===========================+ + | pre_test | calls 'test_pre_check' |assert_define,is_on_target | + | | |,check_process_is_running | + +------------+-------------------------------+---------------------------+ + | build | uses the 'tarfile' definition,|patch,configure,make | + | | calls'test_build' | | + +------------+-------------------------------+---------------------------+ + | deploy | Calls 'test_deploy' | put | + +------------+-------------------------------+---------------------------+ + | run | calls 'test_run' | cmd,report,report_append | + +------------+-------------------------------+---------------------------+ + |get_testlog |(none) | | + +------------+-------------------------------+---------------------------+ + |processing |calls 'test_processing' | log_compare | + +------------+-------------------------------+---------------------------+ + |post_test |calls 'test_cleanup' | kill procs | + +------------+-------------------------------+---------------------------+ + + +other scripts and programs +============================== + + * parser.py is used for benchmark tests + + * it is run against the test log, on the host + * it extracts the values from the test log and puts them in a + normalized format + * these values, called benchmark 'metrics', are compared against + pre-defined threshholds to determine test pass or failure + * the values are saved for use by plotting software + + +============== + Data Files +============== + +There are data files with definitions for several things in the system. + +The Jenkins interface needs to know about boards, running test +processes (slaves), test definitions, and test results. + +The fuego core needs to know about test definitions, boards, platforms +(SDKS), test plans, and test specs. + +The core executes the test script for a test, executing the test +phases in sequence: build the test program, bundle the test programs +for the target, deploy them, execute the test, then post-process the +test. + +The base shell script should: + * build the test program + * deploy the test bundle to the target + * execute the tests + * read the log data from the test + +The base shell script can handle host/target tests (because it runs on +the host). + +(That is, tests that involve actions on both the host and target. + +To add a new test, the user defines several files and puts them into +/fuego-core/engine/tests + +The 'ftc' command uses this directory and the directory +'/fuego-ro/boards' to show available boards and tests on the command +line. Then a user can populate the nodes and jobs in Jenkins, using +'ftc' commands. Each test has to have a front-end entry (a Jenkins job +definition) to allow Jenkins to execute it. This front-end entry +specifies the board, spec and base script for the test. + +======== + Roles +======== + +Human roles: + * test program author - person who creates a new standalone test + program + * test integrator - person who integrates a standalone test into + fuego + * fuego developer - person who modifies Fuego (including the fuego + system scripts or Jenkins) to support more test scenarios or + additional features + * tester - person who executes tests and evaluates results + +================= + Developer Notes +================= + +You can find additional notes about details of Fuego, Jenkins and +their interactions at: + + * :ref:`Fuego Developer Notes <Devref>` + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/rst_src/FAQ.rst b/docs/rst_src/FAQ.rst new file mode 100644 index 0000000..793dfee --- /dev/null +++ b/docs/rst_src/FAQ.rst @@ -0,0 +1,48 @@ + +##### +FAQ +##### + +Here is a list of Frequently Asked Questions and Answers about Fuego: + +=========================== +Languages and formats used +=========================== + +Q. Why does Fuego use shell scripting as the language for tests? +================================================================== + +There are other computer languages which have more advanced features +(such as data structions, object orientation, rich libraries, +concurrency, etc.) than shell scripting. It might seem odd that shell +scripting was chosen as the language for implementing the base scripts +for the tests in fuego, given the availability of these other +languages. + +The Fuego architecture is specifically geared toward host/target +testing. In particular, tests often perform a variety of operations +on the target in addition to the operations that are performed on the +host. When the base script for a test runs on the host machine, +portions of the test are invoked on the target. It is still true +today that the most common execution environment (besides native code) +that is available on almost every embedded Linux system is a +POSIX-compliant shell. Even devices with very tight memory +requirements usually have a busybox 'ash' shell available. + +In order to keep the base script consistent, Fuego uses shell +scripting on both the host and target systems. Shell operations are +performed on the target using 'cmd', 'report' and 'report_append' +functions provided by Fuego. + +Note that Fuego officially use 'bash' as the shell on the host, but +does not require a particular shell implementatio to be available on +the target. Therefore, it is important to use only POSIX-compatible +shell features for those aspects of the test that run on target. + + + + + + + + diff --git a/docs/rst_src/OSS_Test_Vision.rst b/docs/rst_src/OSS_Test_Vision.rst new file mode 100644 index 0000000..e57504d --- /dev/null +++ b/docs/rst_src/OSS_Test_Vision.rst @@ -0,0 +1,348 @@ +.. _oss: + +################# +OSS Test Vision +################# + +This page describes aspects of the Open Source Test vision for the +Fuego project, along with some ideas for implementing specific ideas +related to this vision. + +===================== +overview of concepts +===================== + + +Letter to ksummit discuss +========================== + +Here's an e-mail Tim sent to the ksummit-discuss list in October, +2016 :: + + I have some ideas on Open Source testing that I'd like to + throw out there for discussion. Some of these I have been + stewing on for a while, while some came to mind after talking + to people at recent conference events. + + Sorry - this is going to be long... + + First, it would be nice to increase the amount of testing we + do, by having more test automation. (ok, that's a no-brainer). + Recently there has been a trend towards more centralized + testing facilities, like the zero-day stuff or board farms + used by kernelci. That makes sense, as this requires + specialized hardware, setup, or skills to operate certain + kinds of test environments. As one example, an automated test + of kernel boot requires automated control of power to a board + or platform, which is not very common among kernel developers. + A centralized test facility has the expertise and hardware to + add new test nodes relatively cheaply. They can do this more + quickly and much less expensively than the first such node by + an individual new to testing. + + However, I think to make great strides in test quantity and + coverage, it's important to focus on ease of use for + individual test nodes. My vision would be to have tens of + thousands of individual test nodes running automated tests on + thousands of different hardware platforms and configurations + and workloads. + + The kernel selftest project is a step in the right direction + for this, because it allows any kernel developer to easily + (in theory) run automated unit tests for the kernel. However, + this is still a manual process. I'd like to see improved + standards and infrastructure for automating tests. + + It turns out there are lots of manual steps in the testing + and bug-fixing process with the kernel (and other + Linux-related software). It would be nice if a new system + allowed us to capture manual steps, and over time convert + them to automation. + + Here are some problems with the manual process that I think + need addressing: + + 1) How does an individual know what tests are valid for their + platform? Currently, this is a manual decision. In a world + with thousands or tens of thousands of tests, this will be + very difficult. We need to have automated mechanisms to + indicate which tests are relevant for a platform.Test + definitions should include a description of the hardware + they need,or the test setup they need. For example, it would + be nice to have tests indicate that they need to be run on a + node with USB gadget support, + or on a node with the gadget hardware from a particular vendor + (e.g. a particular SOC), or with a particular hardware phy + (e.g. Synopsis). As another example, if a test requires that + the hardware physically reboot,then that should be indicated + in the test. If a test requires that a particular button be + pressed (and that the button be available to be pressed), it + should be listed. Or if the test requires that an external + node be available to participate in the test (such as a wifi + endpoint, CANbus endpoint, or + i2C device) be present, that should be indicated. + There should be a way for the test nodes which provide those + hardware capabilities, setups, or external resources to + identify themselves. Standards should + be developed for how a test node and a test can express these + capabilities and requirements. Also, standards need to be + developed so that a test can control those external resources + to participate in tests.Right now each test framework handles + this in its own way (if it provides + support for it at all). + + I heard of a neat setup at one company where the video + output from a system was captured by another video system, + and the results analyzed automatically. This type of test + setup currently requires an enormous investment of + expertise, and possibly specialized hardware. Once such a + setup is performed in a few locations, it makes much more + sense to direct tests that need such facilities to those + locations, than it does to try to spread the expertise to + lots of different individuals (although that certainly has + value also). + + For a first pass, I think the kernel CONFIG variables needed + by a test should be indicated, and they could be compared + with the config for the device under test. This would be a + start on the expression of the dependencies between a test + and the features of the test node. + + 2) how do you connect people who are interested in a + particular test with a node that can perform that test? + + My proposal here is simple - for every subsystem of the + kernel, put a list of test nodes in the MAINTAINERS file, to + indicate nodes that are available to test that subsystem. + Tests can be scheduled to run on those nodes, either + whenever new patches are received for that sub-system, or + when a bug is encountered and developers for that subsystem + want to investigate it by writing a new test. Tests or data + collection instructions that are now provided manually would + be converted to formal test definitions, and added to a + growing body of tests. This should help people re-use test + operations that are common. Capturing test operations that + are done manually into a script would need to be very easy + (possibly itself automated), and it would need to be easy to + publish the new test for others to use. + + Basically, in the future, it would be nice if when a person + reported a bug, instead of the maintainer manually walking + someone through the steps to identify the bug and track down + the problem, they could point the user at an existing test + that the user could easily run. + + I imagine a kind of "test app store", where a tester can + select from thousands of tests according to their interest. + Also, people could rate the tests, and maintainers could + point people to tests that are helpful to solve specific + problems. + + 3) How does an individual know how to execute a test and how + to interpret the results? + + For many features or sub-systems, there are existing tools + (e.g bonnie for filesystem tests, netperf for networking + tests, or cyclictest for realtime), but these tools have a + variety of options for testing different aspects of a + problem or for dealing with different configurations or + setups. Online you can find tutorials for running each of + these, and for helping people interpret the results. A new + test system should take care of running these tools with the + proper command line arguments for different test aspects, + and for different test targets ('device-under-test's). + + For example, when someone figures out a set of useful + arguments to cyclictest for testing realtime on a beaglebone + board, they should be able to easily capture those arguments + to allow another developer using the same board to easily + re-use those test parameters, and interpret the cylictest + results, in an automated fashion. Basically we want to + automate the process of finding out "what options do I use + for this test on this board, and what the heck number am I + supposed to look at in this output, and what should its + value be?". + + Another issue is with interpretation of test results from + large test suites. One notorious example of this is LTP. + It produces thousands of results, and almost always produces + failures or results that can be safely ignored on a + particular board or in a particular environment. It requires + a large amount of manual evaluation and expertise to + determine which items to pay attention to from LTP. It + would be nice to be able to capture this evaluation, and + share it with others with either the same board, or the same + test environment, to allow them to avoid duplicating this + work. + + Of course, this should not be used to gloss over bugs in LTP + or bugs that LTP is reporting correctly and actually need to + be paid attention to. + + 4) How should this test collateral be expressed, and how + should it be collected, stored, shared and re-used? + + There are a multitude of test frameworks available. I am + proposing that as a community we develop standards for test + packaging which include this type of information (test + dependencies, test parameters, results interpretation). I + don't know all the details yet. For this reason I am coming + to the community see how others are solving these problems + and to get ideas for how to solve them in a way that would + be useful for multiple frameworks. I'm personally working + on the Fuego test framework - see http://fuegotest.org/wiki, + but I'd like to create something that could be used with any + test framework. + + 5) How to trust test collateral from other sources (tests, + interpretation) + + One issue which arises with this type of sharing (or with + any type of sharing) is how to trust the materials involved. + If a user puts up a node with their own hardware, and trusts + the test framework to automatically download and execute a + never-before-seen test, this creates a security and trust + issue. I believe this will require the same types of + authentication and trust mechanisms (e.g. signing, + validation and trust relationships) that we use to manage + code in the kernel. + + I think this is more important than it sounds. I think the + real value of this system will come when tens of thousands + of nodes are running tests where the system owners can + largely ignore the operation of the system, and instead the + test scheduling and priorities can be driven by the needs of + developers and maintainers who the test node owners have + never interacted with. + + Finally, 6) What is the motivation for someone to run a test + on their hardware? + + Well, there's an obvious benefit to executing a test if you + are personally interested in the result. However, I think + the benefit of running an enormous test system needs to be + de-coupled from that immediate direct benefit. I think we + should look at this the same way we look at other + crowd-sourced initiatives, like Wikipedia. While there is + some small benefit for someone producing an individual page + edit, we need to move beyond that to the benefit to the + community of the cumulative effort. + + I think that if we want tens of thousands of people to run + tests, then we need to increase the cost/benefit ratio for + the system. First, you need to reduce the cost so that it + is very cheap, in all of [time|money|expertise| ongoing + attention], to set up and maintain a test node. Second, + there needs to be a real benefit that people can measure + from the cumulative effect of participating in the system. + I think it would be valuable to report bugs found and fixed + by the system as a whole, and possibly to attribute positive + results to the output provided by individual nodes. (Maybe + you could 'game-ify' the operation of test nodes.) + + Well, if you are still reading by now, I appreciate it. I + have more ideas, including more details for how such a + system might work, and what types of things it could + accomplish. But I'll save that for smaller groups who might + be more directly interested in this topic. + + To get started, I will begin working on a prototype of a + test packaging system that includes some of the ideas + mentioned here: inclusion of test collateral, and package + validation. I would also like to schedule a "test summit" + of some kind (maybe associated with ELC or Linaro Connect, + or some other event), to discuss standards in the area I + propose. + + I welcome any response to these ideas. I plan to discuss + them at the upcoming test framework mini-jamboree in Tokyo + next week, and at Plumbers (particularly during the 'testing + and fuzzing' session) the week following. But feel free to + respond to this e-mail as well. + + Thanks. + -- Tim Bird + + +============================= +Ideas related to the vision +============================= + + +Capturing tests easily +======================== + + * should be easy to capture a command line sequence, and test the + results + * maybe do an automated capture and format into a clitest file that + can be used at a here document inside a fuego test script? + +================== +test collateral +================== + + * does it need to be board-specific + * elements of test collateral: + + * test dependencies: + + * kernel config values needed + * kernel features needed: + + * proc filesystem + * sys filesystem + * trace filesystem + * test hardware needed + * test node setup features + + * ability to reboot the board + * ability to soft-reset the board + * ability to install a new kernel + * presence of certain programs on target + + * bc + * top, ps, /bin/sh, bash? + * already have: + + * CAPABILITIES? + * pn and reference logs + * positive and negative result counts (specific to board) + * test specs indicate parameters for the test + * test plans indicate different profiles (method to match test to + test environment - e.g. filesystem test with type of filesystem hardware) + +================= +test app store +================= + + * need a repository where tests can be downloaded + + * like Jenkins plugin repository + * like debian package feed + + * need a client for browsing tests, installing tests, updating tests + * store a test in github, and just refer to different tests in + different git repositories? + * test ratings + * test metrics (how many bugs found) + +====================== +authenticating tests +====================== + + * need to prevent malicious tests + * packages should be signed by an authority, after review by someone + + * who? the Fuego maintainers? This would turn into a bottleneck + +====================== +test system metrics +====================== + + * number of bugs found and fixed in upstream software + * number of bugs found and fixed in test system + * bug categories (See :ref:`Metrics <metrics>`) + + + + -- 2.7.4