Re: [Fuego] [PATCH] docs: Convert pages categorized as "Feature reference"

> -----Original Message-----
> From: Pooja <[email protected]>
> 
> From: Pooja More <[email protected]>
> 
> List ofpages converted:
> Fuego_charting
> Log_files
> Jenkins
> Jenkins_User_Interface
> 
> Signed-off-by: Pooja More <[email protected]>
> ---
>  docs/rst_src/Fuego_charting.rst         |  19 +++
>  docs/rst_src/Jenkins.rst                |  47 ++++++
>  docs/rst_src/Jenkins_User_Interface.rst | 249 +++++++++++++++++++++++++++++++-
>  docs/rst_src/Log_files.rst              | 185 ++++++++++++++++++++++++
>  4 files changed, 498 insertions(+), 2 deletions(-)
>  create mode 100644 docs/rst_src/Fuego_charting.rst
>  create mode 100644 docs/rst_src/Jenkins.rst
>  create mode 100644 docs/rst_src/Log_files.rst
> 
> diff --git a/docs/rst_src/Fuego_charting.rst b/docs/rst_src/Fuego_charting.rst
> new file mode 100644
> index 0000000..3ff9380
> --- /dev/null
> +++ b/docs/rst_src/Fuego_charting.rst
> @@ -0,0 +1,19 @@
> +################
> +Fuego charting
> +################
> +
> +Fuego has the capability of creating charts (either plots or tables)
> +from results data.  These charts are used either in the Jenkins
> +interface, or in reports generation.
> +
> +This page has a few notes on Fuego's charting features, for users.
> +
> +Charts are produced during the "processing" phase of test execution.
> +This is done after the test results have been parsed from the test log
> +and the results placed in the unified output format (run.json) file.
> +
> +See :ref:`Jenkins Visualization` for information about charts that are
> +produced for the Jenkins interface.  See :ref:`process <parser func
> +process>`  and :ref:`chart_config.json <chart config.json>` for information about the variables
> +and files used for chart creation, and
> +how they can be customized.

Looks good.  I expanded this a bit with a followup commit.

(I just noticed I hadn't pushed the fuego-core branch.  Sorry about
that.  Recent commits should be pushed now.)


> diff --git a/docs/rst_src/Jenkins.rst b/docs/rst_src/Jenkins.rst
> new file mode 100644
> index 0000000..342ec7d
> --- /dev/null
> +++ b/docs/rst_src/Jenkins.rst
> @@ -0,0 +1,47 @@
> +############
> +Jenkins
> +############
> +
> +Jenkins is a continuous integration system with a lot of features and
> +a rich plugin ecosystem.  It is used as the default front end (user
> +interface) for the Fuego test framework.
> +
> +The Jenkins website is at: `<https://jenkins.io/>`_
> +
> +=================================
> +How Jenkins is used by Fuego
> +=================================
> +
> +Jenkins is used to organize the tests provided by Fuego, and present
> +the user interface for test selection, test execution, and information
> +about test history, logs, etc.
> +
> +Jenkins is also used to see the tests for a particular set of boards.
> +
> +Jenkins also provides the triggers for the test system.  That is,
> +Jenkins initiates jobs, as configured by the user, for starting tests
> +and collecting test results.
> +
> +
> +======================
> +Miscelaneous notes
> +======================
> +
> + * Fuego logs (test log, devlog, syslog and parsed log - see :ref:`Log
> +   files` are not available through the Jenkins interface (as of
> +   Aug-2016).  This seems like a big oversight. See :ref:`Issue 0010`
> +
> + * builds are marked by Jenkins as: successful, failed, stable, unstable.
> +
> +   * a freestyle build is successful if the return code from executing
> +     the code snippet is 0
> +   * a freestyle build is considered 'failed' if
> +     the return code from executing the code snippet is non-zero
> +   * a build is stable unless explicitly marked otherwise by some
> +     Jenkins action
> +
> +    * they are usually marked so in a post-test operation, by
> +      something like TextFinder, groovy, or some other thing
> +
> +       * see  `<http://stackoverflow.com/questions/8148122/
> +         how-to-mark-a-build-unstable-in-jenkins-when-running-shell-scripts>`_
> diff --git a/docs/rst_src/Jenkins_User_Interface.rst b/docs/rst_src/Jenkins_User_Interface.rst
> index 022dee7..b379796 100644
> --- a/docs/rst_src/Jenkins_User_Interface.rst
> +++ b/docs/rst_src/Jenkins_User_Interface.rst
> @@ -1,5 +1,250 @@
> -.. _jUsrinterface:
> -
>  #######################
>  Jenkins User interface
>  #######################
> +
> +By default, Fuego uses the Jenkins continuous integration system to
> +manage boards, tests, logs, and test results.
> +
> +The Jenkins user interface is web-based.  This page shows several
> +screenshots of different pages in the Jenkins interface.
> +
> +Through this interface, you can see the status of tests that have run,
> +review the logs for tests, and schedule new tests to run on target
> +boards.  You also use this interface to add new boards and new tests
> +to the system.
> +
> +Note that Jenkins objects are:
> +
> + * nodes
> + * jobs
> + * builds
> + * views
> +
> +These are different from the Fuego names for the same objects.  The
> +first three of these Jenkins objects correspond to the Fuego objects
> +of: **boards**, **tests** and **runs**, respectively.
> +
> +==================
> +Main dashboard
> +==================
> +
> +The main dashboard of Jenkins looks like the following:
> +
> +New Installation
> +====================
> +
> +When Fuego has just been installed, there is nothing in the Jenkins
> +interface (no nodes, jobs or views). The interface should look
> +something like this:
> +
> +.. image:: ../images/fuego-1.1-jenkins-dashboard-new.png
> +   :width: 900
> +
> +With a single node (board) added
> +===================================
> +
> +Here is the main dashboard of Jenkins, after a single node (called
> +'beaglebone' in this case) has been added.  Note the node (board)
> +appears in the left sidebar under "Build Executor Status":
> +
> +.. image:: ../images/fuego-1.1-jenkins-dashboard-beaglebone.png
> +   :width: 900
> +
> +
> +
> +With beaglebone node and jobs
> +=================================
> +
> +Here is the main dashboard of Jenkins, showing a single node
> +(beaglebone) and jobs for this board.
> +
> +.. image:: ../images/fuego-1.1-jenkins-dashboard-beaglebone-jobs.png
> +   :width: 900
> +
> +
> +Dashboard with jobs in Build Queue
> +===================================
> +
> +Here is the Jenkins dashboard with a lot of jobs in the Build Queue.
> +Note the list of jobs in the left side-bar, in the "Build Queue" pane.
> +
> +.. image:: ../images/fuego-1.1-jenkins-dashboard-batch-build-queue.png
> +   :width: 900
> +
> +==============
> +Node pages
> +==============
> +
> +If you click on the node in the **Build Executor Status** pane, then
> +you can see a list of the jobs associated with a node.
> +
> +
> +Node status page
> +=======================
> +
> +Here is the status for the beaglebone node.
> +
> +.. image:: ../images/fuego-1.1-jenkins-beaglebone-node.png
> +   :width: 900
> +
> +
> +==============
> +Job pages
> +==============
> +
> +If you click on a job in the Jenkins interface, you can see
> +information about an individual job. This page shows information about
> +the status of the job, including a Build History for the job (in the
> +left sidebar).
> +
> +You can start a job by clicking on the "Build Now"  button in the left
> +menu.
> +
> +Functional job status page
> +=================================
> +
> +Here is a page showing the status information for a Functional test
> +called 'hello_world'.  The main area of the screen has information
> +about the last successful and failed builds of the test.  Note the
> +left sidebar pane with "Build History", to see individual test
> +execution results.
> +
> +.. image:: ../images/fuego-1.1-jenkins-hello_world-job.png
> +   :width: 900
> +
> +
> +Benchmark job - starting a build
> +=====================================
> +
> +Here is a picture of a job build being started.  Note the progress bar
> +in the **Build History** pane in the left sidebar.
> +
> +.. image:: ../images/fuego-1.1-jenkins-dhrystone-start-build.png
> +   :width: 900
> +
> +
> +Benchmark job - before successful execution
> +================================================
> +
> +Before a Benchmark job completes it has no data to plot on it's chart,
> +and appears similar to a Functional Job status page:
> +
> +.. image:: ../images/fuego-1.1-jenkins-Dhrystone-job.png
> +   :width: 900
> +
> +
> +
> +Benchmark job - with plot of metrics
> +========================================
> +
> +Normally, a Benchmark page shows one or more plots showing the values
> +for data returned by this benchmark.
> +
> +.. image:: ../images/fuego-1.1-jenkins-dhrystone-job-plot.png
> +   :width: 900
> +
> +===============
> +Build pages
> +===============
> +
> +A build page shows the results of a single execution of a job (test)
> +on a board.  You can click on the build number in the Jenkins
> +interface to see this page.
> +
> +
> +Results from a job build
> +==============================
> +
> +Here are the results from the execution of the "hello world" job.
> +This was the results of running the Fuego test
> +"Functional.hello_world" on a beaglebone board.
> +
> +.. image:: ../images/fuego-1.1-jenkins-hello_world-build.png
> +   :width: 900
> +
> +Test log results
> +====================
> +
> +You can examine the different logs for each test.  Each test produces
> +a log from the program that ran on the board.  This is available by
> +following a link called "log" from the 'build' page for that test run.
> +You can see the console log, which shows the output of commands for
> +this test, by clicking on "console log" in the build interface (or the
> +build drop-down menu in the **Build History** list).
> +
> +Drhystone test log
> +--------------------------
> +
> +Here are results from a run of the Dhrystone test on a beaglebone
> +board:
> +
> +.. image:: ../images/fuego-1.1-jenkins-dhrystone-log.png
> +   :width: 900
> +
> +Jenkins Console log
> +---------------------
> +
> +Here is the console log for a test executed on the beaglebone:
> +
> +.. image:: ../images/fuego-1.1-jenkins-console-log.png
> +   :width: 900
> +
> +==============
> +View pages
> +==============
> +
> +A **view** is an area in the main dashboard that shows a collection
> +of jobs with a particular set of status columns for each job.  They
> +appear as tabs in the main dashboard view of Jenkins.  You can create
> +your own view to see a subset of the jobs that are available in
> +Jenkins
> +
> +Here are some screen shots showing how to add a new view to Jenkins.
> +
> +
> +Screen to add a new view
> +==============================
> +
> +Here is the screen to add a new view to Jenkins.
> +
> +.. image:: ../images/fuego-1.1-jenkins-add-view-beaglebone.png
> +   :width: 900
> +
> +
> +Screen to configure the view settings
> +===========================================
> +
> +Here is the screen to configure view settings.  Note the use of a
> +regular expression to control what jobs to see in this view.  You can
> +also control what status columns to display in the view.
> +
> +.. image:: ../images/fuego-1.1-jenkins-config-view-beaglebone.png
> +   :width: 900
> +
> +=======================
> +Other Jenkins pages
> +=======================
> +
> +
> +Build History
> +==================
> +
> +The global build history page is available by clicking on the **Build
> +History** link in the main dashboard page of Jenkins.  It shows the
> +execution time and status for a recent time period.
> +
> +.. image:: ../images/fuego-1.1-jenkins-build-history.png
> +   :width: 900
> +
> +
> +Jenkins management
> +=======================
> +
> +You can manage Jenkins using the **Manage Jenkins** page, available
> +from the top-level dashboard page in Jenkins.  From here you can
> +update Jenkins itself, install or remove plugins, and perform other
> +management operations for Jenkins.
> +
> +.. image:: ../images/fuego-1.1-jenkins-management.png
> +   :width: 900
> +

Looks good.

> diff --git a/docs/rst_src/Log_files.rst b/docs/rst_src/Log_files.rst
> new file mode 100644
> index 0000000..b9caf68
> --- /dev/null
> +++ b/docs/rst_src/Log_files.rst
> @@ -0,0 +1,185 @@
> +##################
> +Log files
> +##################
> +
> +
> +During test execution, Fuego collects several different logs.
> +
> +These represent different aspects of the system, and are used for
> +different purposes.
> +
> +FIXTHIS - finish documenting log files on the Log files page.
> +Specifically, document parsed logs better (describe how they are
> +generated in log_compare)
> +
> +Here are the main logs collected or generated during a test:
> +
> + * **console log**
> + * **devlog**
> + * **syslog**
> + * **test log**
> + * **parsed log**
> +
> +These are located in the 'run' directory (also know as the 'log'
> +directory), at:
> +``/fuego-rw/logs/<test_name>/<board>.<spec>.<build_id>.<build_number>``
> +
> +There are additional logs that may be defined for a test, which are
> +used in post-processing and checking of results for the test.
> +
> + * **pn log**
> + * **reference log**
> +
> +================
> +Results logs
> +================
> +
> +These logs are the results of test execution, and have output from
> +different parts of the system.
> +
> +console log
> +==================
> +
> +The console log is collected by Jenkins during the entire execution of
> +the test.  It is dominated by the invocation of the test base script,
> +which is executed as a shell script, with the 'set -x' parameter set.
> +
> +During execution of the base script, several "source" statements are
> +encountered, nesting the invocation of scripts multiple times.  The
> +number of '+' signs preceding a line in the console log indicates the
> +depth (or invocation nesting level) for that line, in the shell
> +execution.
> +
> +There are "phase" messages added to the log during the test, which can
> +help identify which part of the test a particular sequence is in.
> +This can help you debug what part of a test (pre_test, build, deploy,
> +run, post_test) is failing, if any.
> +
> +The second major part of the script is the post_test phase, which is
> +also executed as a shell script fragment, utilizing the prolog
> +generated previously, and documenting the gather of logs and
> +determination of final test result.
> +
> +The location of the console log for a test run is at:
> +``/var/lib/jenkins/jobs/<jobname>/builds/<build_number>/log``
> +
> +If a test was executed by ftc instead of Jenkins (that is, directly
> +from the command line), then the consolelog is in the Fuego log
> +direcotry and is called: ``consolelog.txt``
> +
> +
> +devlog
> +===========
> +
> +This is a summarized list of operations performed during the execution
> +of a test.  These entries are created when internal routines use the
> +function :ref:`report_devlog <function report devlog>` to add a line to
> +this log.
> +
> +The name of the devlog for a test run is ``devlog.txt``
> +
> +syslog
> +===========
> +
> +The syslog records the messages from the target system's log.  There
> +are two of these recorded, one during the pre_test (called the
> +"before" log), and one during the post_test (called the "after" log).
> +
> +This includes messages from the kernel (if a logger is transferring
> +the messages from the kernel to the system log), as well as messages
> +from programs running on the system (that output to their status to
> +the syslog).
> +
> +The names of the syslogs for a test are:
> +
> + * ``syslog.before.txt``
> + * ``syslog.after.txt``
> +
> +test log
> +============
> +
> +This is the output produced by the test program itself.  It is
> +collected with the *report* and *report_append* functions, which
> +save the standard output from the commands they run into a log file
> +which is retrieved from the target at the end of the test.
> +
> +The name of the test log for a test is: ``testlog.txt``
> +
> +parsed log
> +===============
> +
> +The 'parsed' version of the log, is one that has been grep'ed for a
> +regular expression using log_compare.  The parsed log should consist
> +of individual lines showing a pass, fail or skip for each sub-test in
> +a test suite.
> +
> +The names of the parsed logs for a test are:
> +
> + * *testlog.p.txt*
> + * *testlog.n.txt*
> +
> +==================
> +Reference logs
> +==================
> +
> +Fuego also uses reference logs during test results processing, to
> +compare against the logs that are generated during a test:
> +
> +pn log
> +============
> +
> +A test may provide a so-called 'pn log', which is a log file recording
> +the positive and negative results from a previous test run.
> +
> +If this file exists in the test directory, it is used during
> +post-processing (in the :ref:`log_compare <function log compare>`
> +function), to see if the positive or negative results from the current
> +log match those from the saved pn log.
> +
> +reference log
> +==================
> +
> +This file holds the operations and threshold values for a Benchmark
> +test.
> +
> +See :ref:`reference.log`
> +
> +============
> +summary
> +============
> +
> +In the Fuego version 1.1 (early 2017), the log directories are as
> +follows:
> +
> + * Fuego logs:
> +
> +   * ``/fuego-rw/logs/<testname>/<board>.<spec>.<build_id>.<build_number>``
> +
> +     * run.json - saved by test itself
> +     * devlog.txt - written by report_devlog
> +     * syslog.before.txt - saved by ov_rootfs_logread (dump_syslogs)
> +     * syslog.after.txt - saved by ov_rootfs_logread
> +     * testlog.txt - saved by get_testlog
> +     * testlog.p.txt - created by log_compare
> +     * consolelog.txt - created by ftc (or link to Jenkins console log)
> +     * res.json - created by bench_processing (parser.py for a Benchmark test)
> +     * fres.json - created by post_test (parser.py for a Functional test
> +
> + * jenkins files:
> +
> +   * ``/var/lib/jenkins/jobs/<jobname>/builds/buildnum/``
> +
> +     * build.xml
> +     * changelog.xml
> +     * log - console log created by Jenkins
> +
> + * per-test data files:
> +
> +   * ``/fuego-rw/logs/<testname>``
> +
> +     * plot.data
> +     * plot.png
> +     * metrics.json - list of metrics for this test
> +     * <testname>.<metric>.json - list of data values for a metric
> +     * <testname>.info.json - list of meta-data for each data line (device, firmware, platform data points)
> +
> --
> 2.7.4
> 
> 
> --

Looks good.  I updated some of the material with followup commits
to correct or remove obsolete information.

I have pushed to the master branch.

Thanks for the conversions!

 -- Tim