Re: [Fuego] [PATCH] Docs: Convert the fuego wiki pages categorized as "file reference"

Thanks.  See comments inline below.

> -----Original Message-----
> From: Pooja <[email protected]>
> 
> From: Pooja More <[email protected]>
> 
> Fuego pages converted:
> Fuego_configuration_file
> chart_config.json
> criteria.json
> parser.py
> 
> Signed-off-by: Pooja More <[email protected]>
> ---
>  docs/rst_src/Fuego_configuration_file.rst | 112 +++++++
>  docs/rst_src/chart_config.json.rst        | 116 +++++++
>  docs/rst_src/criteria.json.rst            | 492 ++++++++++++++++++++++++++++++
>  docs/rst_src/parser.py.rst                | 155 ++++++++++
>  4 files changed, 875 insertions(+)
>  create mode 100644 docs/rst_src/Fuego_configuration_file.rst
>  create mode 100644 docs/rst_src/chart_config.json.rst
>  create mode 100644 docs/rst_src/criteria.json.rst
>  create mode 100644 docs/rst_src/parser.py.rst
> 
> diff --git a/docs/rst_src/Fuego_configuration_file.rst b/docs/rst_src/Fuego_configuration_file.rst
> new file mode 100644
> index 0000000..aaea0c3
> --- /dev/null
> +++ b/docs/rst_src/Fuego_configuration_file.rst
> @@ -0,0 +1,112 @@
> +############################
> +Fuego configuration file
> +############################
> +
> +The Fuego configuration file contains configuration options and
> +settings for the Fuego test system.
> +
> +The following settings are supported:
> +
> +===============================
> +Fuego test network settings
> +===============================
> +
> +These settings are used when a host participates in a Fuego test
> +network.  This refers to distributed operations between Fuego labs,
> +and not to the test network within a single Fuego lab.
> +
> +host_name
> +================
> +
> +host_name is used to specify a name for this instance of Fuego, within
> +the worldwide Fuego test network (or within a company Fuego network).
> +The default value for host_name is 'fuegohost'.  If this site
> +participates in distributed testing, the host_name should be set to
> +something unique, which describes the host.  For example, my lab uses
> +a Fuego host_name of "timdesk".
> +
> +During test execution, this value appears in the environment variable
> +``FUEGO_HOST``.
> +
> +A host_name should only consist of letters, numbers and underscores.
> +No other punctuation is allowed.
> +
> +fuego_server
> +==================
> +
> +The fuego_server setting indicates the domain name or IP address where
> +a Fuego test server is running.  This is a site where fuego tests can
> +be uploaded and downloaded, and where run requests can be posted and
> +retrieved, and run data can be uploaded for sharing.
> +
> +By default, the value for this is 'fuegotest.org', which is the main
> +fuego web site.  If an organization is running its own Fuego server,
> +it should specify the IP address for their own server.

I added missing material here about squad server settings.

> +
> +=====================
> +Fuego directories
> +=====================
> +
> +Fuego data and programs are located in three directories, used for
> +read-only data, read-write data, and a core system directory.  These
> +variables are normally located within the Fuego repository, and at
> +hardcoded locations within the docker container.  However, they are
> +specified in the fuego configuration file so that the directories may
> +be placed elsewhere if this is convenient.
> +
> +Each of these directories is normally specified as a file path
> +relative to the directory where the ``fuego.conf`` file resides.  However,
> +absolute paths are allowed.  If absolute paths are used, then the bind
> +mounts for the docker container should be such at the same absolute
> +paths can used both inside and outside the docker container to access
> +these directories.  For example, you could re-configure Fuego to use
> +directories under ``/opt/fuego`` both inside the container and on the
> +host.
> +
> +fuego_ro_dir
> +==================
> +
> +This indicates the location of the Fuego read-only data directory.
> +The default value is "..", since the fuego.conf file usually resides
> +in the directory ``fuego-ro/conf``.
> +
> +Some important directories that reside in the fuego-ro directory are
> +the conf, boards and toolchains directory.
> +
> +fuego_rw_dir
> +==================
> +
> +``fuego_rw_dir`` indicates the location of the Fuego read-write data
> +directory.  The default value for fuego_rw_dir is ``../../fuego-rw``
> +
> +Some important directories that reside in the ``fuego-rw`` directory are
> +the buildzone and logs directories.
> +
> +fuego_core_dir
> +======================
> +
> +The fuego_core_dir indicates the location of the Fuego core directory,
> +which has the main scripts, programs, data and source code that make
> +up the Fuego test system.  The default value for fuego_core_dir is
> +../../fuego-core
``../../fuego-core``

> +
> +=============================
> +Configuration file syntax
> +=============================
> +
> +The configuration file uses a very simple name=value syntax.  Most
> +settings consist of a single line.  Lines starting with '#' are
> +ignored.  Empty lines are ignored, and lines starting with a '#' are
> +interpreted as comments and are ignored.
> +
> +If a setting value requires multiple lines, it is enclosed in
> +triple-double quotes, like a python multiline string.
> +
> +Example: ::
> +
> +  # this is a comment
> +  variable=value
> +  multi_line_variable="""foo
> +     bar
> +  baz   baf
> +  """
> diff --git a/docs/rst_src/chart_config.json.rst b/docs/rst_src/chart_config.json.rst
> new file mode 100644
> index 0000000..acdbb7d
> --- /dev/null
> +++ b/docs/rst_src/chart_config.json.rst
> @@ -0,0 +1,116 @@
> +#####################
> +chart config.json
> +#####################
> +
> +The ``file chart_config.json`` is defined for each test.  This file

file ``chart_config.json``

> +controls how charts are drawn, in the Jenkins interface for the test
> +it is associated with.
> +
> +==========
> +Schema
> +==========
> +
> +``chart_config.json`` holds a single object, with attribute pairs
> +describing values for the configuration of the charts for a test.  The
> +following pairs are supported:
> +
> + * **chart_type** - this indicates the type of chart to
> +   present in the Jenkins interface for a test job
> +
> +   * It's value must be one of:
> +
> +     * **measure_plot**
> +     * **testcase_table**
> +     * **testset_summary_table**
> +     * **measure_table**
> +
> + * **match_board** - if true, this indicates that a job page in Jenkins
> +   should only display the results for boards that match the job.
> +
> +   * By default, Fuego displays the results for all boards on a job page
> +     for a test.  This is to allow users to compare results between boards.
> +     However, it is often desirable to only show the data for a single
> +     board (the one that the job actually refers to).
> +
> +   * possible values are:
> +
> +     * **true**
> +     * **false**
> +
> + * **measures** - this is a list of measures for which plots will be drawn
> +
> +   * Each measure is specified by it's tguid, which must be
> +     specified in full (see :ref:`Fuego naming rules`)
> +
> +   * If no measures are listed in the ``chart_config.json`` file,
> +
> +all the measures produced by the test will be plotted.
> +
> +The purpose of the "measures" field is to limit the charting to only a
> +few important, or characteristic, measures.  Some Benchmark tests save
> +many results, and the user may want to focus on only a few measures
> +that they are specifically interested in.
> +
> +Here is an example, from the test Benchmark.cyclictest: ::
> +
> +  {
> +      "chart_type": "measure_plot",
> +      "measures": ["default.latencies.max_latency",
> +         "default.latencies.avg_latency"]
> +  }
> +
> +============
> +Defaults
> +============
> +
> +If a test has no chart_config.json, then default values are used, as follows:
> +
> + * For Benchmark tests, create a measure_plot for all measures found in
> +   any run of that test.
> +
> +   * Each measure_plot has one measure, shown with values
> +     for all boards that have had that test run.
> +   * The value and reference value (threshold) for each measure are
> +     plotted relative to each run of the test (by Jenkins build number).
> +
> + * For Functional tests, create a testcase table for each board.
> +
> +   * Each testcase table has the result status for each result, and a
> +     set of summary lines at the bottom of the table, for each run of
> +     the test (by Jenkins build number)
> +
> + * For ``Functional.LTP``, create a testset_summary_table for each board
> +
> +   * Summary counts of pass/fail/error/skip status are shown for each
> +     testset (collection of test cases) for each run.
> +
> +====================
> +Planned features
> +====================
> +
> +Additional features are planned for future releases of Fuego, including the
> +following:
> +
> + * Additional chart_types:
> +
> +   * testset_summary_plots - a plot of summary data by test set
> +
> + * Control over measure grouping:
> +
> +   * The ability to place multiple measures in the same plot
> +
> + * Control over board grouping:
> +
> +   * The ability to only show a single board, or specific
> +     groups of boards, in single plots
> +
> + * Control over header data
> +
> +   * The ability to customize the meta-data placed in table headers
> +
> +============
> +See also
> +============
> +
> + * See :ref:`Jenkins Visualization` for more information about the charts
> +   that are configured by this file.
> diff --git a/docs/rst_src/criteria.json.rst b/docs/rst_src/criteria.json.rst
> new file mode 100644
> index 0000000..efb0b5f
> --- /dev/null
> +++ b/docs/rst_src/criteria.json.rst
> @@ -0,0 +1,492 @@
> +##################
> +criteria.json
> +##################
> +
> +================
> +Introduction
> +================
> +
> +The ``criteria.json`` file is used to specify the criteria used to
> +determine whether a test has passed or failed.
> +
> +For the purpose of this explanation, I'll group tests into roughly 3
> +groups:
> +
> + * Simple Functional tests
> + * Complex Functional tests
> + * Benchmarks
> +
> +I'll come back to these definitions in a moment.
> +
> +The ``criteria.json`` file contains data that allows Fuego to
> +interpret the test results, and indicate overall PASS or FAIL status
> +for a test.
> +
> +For functional tests, this includes things like counting the number of
> +test_cases in the test that had "PASS" or "FAIL" status, as well as
> +ignoring some specific test results.
> +
> +For benchmark tests, this includes specifying threshold values for
> +measurements taken by the benchmark, as well as operations (e.g. 'less
> +than' or 'greater than'), to use to determine if a specific measure
> +passed or failed.
> +
> +Fuego uses the results of the test along with the criteria, to
> +determine the final result of the test.
> +
> +If no criteria.json file is provided, then a default is constructed
> +based on the test results, consisting of the following:
> +
> +::
> +
> +  {
> +   'tguid': <test_set_name>
> +   'max_fail': 0
> +  }
> +
> +Types of tests and pass criteria
> +======================================
> +
> +A simple functional test runs a short sequence of tests, and if any one of them
> +fails, then the test is reported as a failure.  Since this corresponds to the
> +default ``criteria.json``, then most simple Functional tests do not need to provide
> +a ``criteria.json`` file.
> +
> +A complex functional test (such as LTP or glib) has hundreds or possibly
> +thousands of individual test cases.  Such tests often have some number of
> +individual test cases that fail, but which may be safely ignored (either
> +temporarily or permamently).  For example, some test cases may fail
> +sporadically due to problems with the test infrastructure or environment. Other
> +tests may fail due to configuration choices for the software on the board. (For
> +example, a choice of kernel config may cause some tests to fail - but this is
> +expected and these fail results should be ignored).
> +
> +Functional tests that are complex require a ``criteria.json`` file, to avoid
> +failing the entire test because of individual test_cases that should be
> +ignored.
> +
> +Finally, a Benchmark test is one that produces one or more "measurements",
> +which are test results with numeric values.  In order to determine whether
> +a result indicates a PASS or a FAIL result, Fuego needs to compare the
> +numeric result with some threshold value.  The ``criteria.json`` file holds
> +the threshold value and operator used for making this comparison.
> +
> +Different boards, or boards with different software installations or
> +configurations, may require different pass criteria for the same tests.
> +Therefore, the pass criteria are broken out into a separate file that can be
> +adjusted at each test site, and for each board.  Ultimately, we would like
> +testers to be able to share their pass criteria, so that each Fuego user does
> +not have to determine these on their own.
> +
> +
> +=======================
> +Evaluation criteria
> +=======================
> +
> +The criteria file lists "pass criteria" for test suites, test sets, test cases
> +and measures.  A single file may list one or more pass criteria for the test.
> +
> +The criteria file may include count-based pass criteria, specific testcase
> +lists, and measure reference values (thresholds).
> +
> +The criteria file specifies the pass criteria for one or more test element
> +results, by specifying the element's test id (or tguid), and the criterion
> +used to evaluate that element.  Some results elements, such as test sets,
> +are aggregates of other elements.  For these, the criteria specify
> +attributes of their child elements (like required counts, or listing
> +individual children that must pass or fail).
> +
> +The criteria file consists of a list of criterion objects (JSON objects),
> +each of which specifies the tguid for the result element of the test,
> +and additional data used to evaluate that element.  tguids are generated
> +by Fuego during the processing phase, and consist of statically defined
> +strings unique to each test.  You should look at a test's :ref:`run.json` file
> +to see the test element names for a test.
> +
> +Here are the different operations that can be used for criteria:
> +
> + * **max_fail** - specifies the maximum number of child elements that can
> +   fail, before causing this element to fail
> +
> +   * by default, every aggregate element must have all it's children pass,
> +     in order for it to pass (corresponding to a 'max_fail' of 0)
> +
> + * **min_pass** - specifies the minimum number of child elements that must pass,
> +   in order for this element to pass
> + * **must_pass_list** - specifies a list of child elements, by name, that must pass
> +   for this element to pass
> + * **fail_ok_list** - specifies a list of child elements, by name, that may fail,
> +   without causing this element to fail
> + * **reference** - specifies a reference value used as a threshold to evaluate where a
> +   number value for this element represents pass or fail.
> +
> +   * the reference object has two sub-attributes:
> +
> +     * **value** - the reference value (threshold)
> +     * **operator** - the test between the result and the reference value
> +
> +The operator can be one of the following strings:
> +
> + * **gt** - result must be greater than the reference value
> + * **ge** - result must be greater than or equal to the reference value
> + * **lt** - result must be less than the reference value
> + * **le** - result must be less than or equal to the reference value
> + * **eq** - result must equal the reference value
> + * **ne** - result must not equal the reference value
> + * **bt** - result is between two reference values (or equal to one of them)
> +
> +In case the reference object has an operator of 'bt', the 'value' field should
> +have a string consisting of two numbers separated by a ','.  For example, to
> +indicate that the result value should be between 4 and 5, the 'value' field
> +should have the string "4,5".  Note that the comparison for 'between' also
> +succeeds for equality.  So in the example case of a reference value of "4,5",
> +the test would pass if the test result was exactly 4, or exactly 5, or any
> +number between 4 and 5.
> +
> +.. note::
> +   The equality and inequality operators ('eq' and 'ne') are less likely
> +   to be useful for numerical evaluations of most benchmark measures, but are
> +   provided for completeness.  These are useful if a test reports numerical results
> +   from within a small set of numbers (like 0 and 1).
> +
> +
> +==================================================
> +Customizing the criteria.json file for a board
> +==================================================
> +
> +A Fuego user can customize the pass criteria for a board, by making a copy of
> +the ``criteria.json`` file, manually editing the contents, and putting it in a
> +specific directory with a specific filename, so Fuego can find it.
> +
> +Using an environment variable
> +===================================
> +
> +A Fuego user can specify their own path to the criteria file to use for a test
> +using the environment variable ``FUEGO_CRITERIA_JSON_PATH``.  This can be set in
> +the environment variables block in the Jenkins job for a test, if running the
> +Fuego test from Jenkins, or in the shell environment prior to running a Fuego
> +test using 'ftc'.
> +
> +For example, the user could do the following:
> +
> + * $ export FUEGO_CRITERIA_JSON_PATH=/tmp/my-criteria.json
> + * $ ftc run-test -b board1 -t Functional.foo
> +
> +
> +Using a board-specific directory
> +=====================================
> +
> +More commonly, a user can specify a board-specific criteria file, by placing
> +the file under either ``/fuego-rw/boards`` or ``/fuego-ro/boards``
> +
> +When Fuego does test evaluation, it searches for the the criteria file to
> +use, by looking for the following files in the indicated order:
> +
> + * $FUEGO_CRITERIA_JSON_PATH
> + * /fuego-ro/boards/{board}-{testname}-criteria.json
> + * /fuego-rw/boards/{board}-{testname}-criteria.json
> + * /fuego-core/engine/tests/{testname}/criteria.json
> +
> +As an example, a user could customize the criteria file as follows:
> +
> + * $ cp /fuego-core/engine/tests/Benchmark.Dhrystone/criteria.json /fuego-rw/boards/board1-Benchmark.Dhrystone-criteria.json
> + * $ edit /fuego-rw/boards/board1-Benchmark.Dhrystone-criteria.json
> +
> +    * Alter the reference value for the tguid 'default.Dhrystone.Score' to reflect a value
> +      appropriate for their board ('board1' in this example)
> +
> + * (execute the job 'board1.default.Benchmark.Dhrystone' in Jenkins)
> +
> +   * Fuego will use the criteria file for board1 in ``/fuego-rw`` instead of the
> +     default ``criteria.json`` file in the test's home directory
> +
> +=============
> +Examples
> +=============
> +
> +Here are some example ``criteria.json`` files:
> +
> +Benchmark.dbench
> +===================
> +
> +::
> +
> +  {
> +      "schema_version":"1.0",
> +      "criteria":[
> +          {
> +              "tguid":"default.dbench.Throughput",
> +              "reference":{
> +                  "value":100,
> +                  "operator":"gt"
> +              }
> +          },
> +          {
> +              "tguid":"default.dbench",
> +              "min_pass":1
> +          }
> +      ]
> +  }
> +
> +
> +The interpretation of this criteria file is that the measured value of
> +dbench.Throughput (the result value) must have a value greater than 100.  Also,

I decided to use literal-quotes (double-accent quotes) for testcase (tguid) and
test set names.  So I literal-quoted a number of items in these example sections.


> +at least 1 measure under the 'default.dbench' test must pass, for the the
> +entire test to pass.
> +
> +Simple count
> +===================
> +
> +::
> +
> +  {
> +      "schema_version":"1.0",
> +      "criteria": [
> +      {
> +          "tguid": "default",
> +          "max_fail": 2
> +      },
> +  }
> +
> +
> +The interpretation of this criteria file is that the test may fail up to 2
> +individual test cases, under the 'default' test set, and still pass.
> +
> +Child results
> +===================
> +
> +::
> +
> +  {
> +      "schema_version":"1.0",
> +      "criteria": [
> +      {
> +          "tguid": "syscall",
> +          "min_pass": 1000,
> +          "max_fail": 5
> +      },
> +      {
> +          "tguid": "timers",
> +          "fail_ok_list": ["leapsec_timer"]
> +      },
> +      {
> +          "tguid": "pty",
> +          "must_pass_list": ["hangup01"]
> +      }
> +      ]
> +  }
> +
> +
> +The interpretation of this criteria file is that, within the 'syscall' test
> +set, a minimum of 1000 testcases must pass, and no more than 5 fail, in order
> +for that set to pass.  Also, in the test set 'timers', if the testcase
> +'leapsec_timer' fails, it will not cause the entire test to fail.  However, in
> +the test set 'pty', the testcase 'hangup01' must pass for the entire test to
> +pass.
> +
> +===========
> +Schema
> +===========
> +
> +The schema for the criteria.json file is contained in the ``fuego-core`` repository
> +at: ``engine/scripts/parser/fuego-criteria-schema.json``.
> +
> +Here it is (as of Fuego 1.2):
> +
> +::
> +
> +  {
> +      "$schema":"http://json-schema.org/schema#",
> +      "id":"http://www.fuegotest.org/download/fuego_criteria_schema_v1.0.json",
> +      "title":"criteria",
> +      "description":"Pass criteria for a test suite",
> +      "definitions":{
> +          "criterion":{
> +              "title":"criterion ",
> +              "description":"Criterion for deciding if a test (test_set, test_case or measure) passes",
> +              "type":"object",
> +              "properties":{
> +                  "tguid":{
> +                      "type":"string",
> +                      "description":"unique identifier of a test (e.g.: Sequential_Output.CPU)"
> +                  },
> +                  "min_pass":{
> +                      "type":"number",
> +                      "description":"Minimum number of tests that must pass"
> +                  },
> +                  "max_fail":{
> +                      "type":"number",
> +                      "description":"Maximum number of tests that can fail"
> +                  },
> +                  "must_pass_list":{
> +                      "type":"array",
> +                      "description":"Detailed list of tests that must pass",
> +                      "items":{
> +                          "type":"string"
> +                      }
> +                  },
> +                  "fail_ok_list":{
> +                      "type":"array",
> +                      "description":"Detailed list of tests that can fail",
> +                      "items":{
> +                          "type":"string"
> +                      }
> +                  },
> +                  "reference":{
> +                      "type":"object",
> +                      "description":"Reference measure that is compared to a result measure to decide the status",
> +                      "properties":{
> +                          "value":{
> +                              "type":[
> +                                  "string",
> +                                  "number",
> +                                  "integer"
> +                              ],
> +                              "description":"A value (often a threshold) to compare against.  May be two numbers separated by a comma for the 'bt'
> operator."
> +                          },
> +                          "operator":{
> +                              "type":"string",
> +                              "description":"Type of operation to compare against",
> +                              "enum":[
> +                                  "eq",
> +                                  "ne",
> +                                  "gt",
> +                                  "ge",
> +                                  "lt",
> +                                  "le",
> +                                  "bt"
> +                              ]
> +                          }
> +                      },
> +                      "required":[
> +                          "value",
> +                          "operator"
> +                      ]
> +                  }
> +              },
> +              "required":[
> +                  "tguid"
> +              ]
> +          }
> +      },
> +      "type":"object",
> +      "properties":{
> +          "schema_version":{
> +              "type":"string",
> +              "description":"The version number of this JSON schema",
> +              "enum":[
> +                  "1.0"
> +              ]
> +          },
> +          "criteria":{
> +              "type":"array",
> +              "description":"A list of criterion items",
> +              "items":{
> +                  "$ref":"#/definitions/criterion"
> +              }
> +          }
> +      },
> +      "required":[
> +          "schema_version",
> +          "criteria"
> +      ]
> +  }
> +
> +=============================================
> +Compatibility with previous Fuego versions
> +=============================================
> +
> +The criteria.json file replaces the **reference.log** file that was used in
``reference.log``

> +versions of Fuego prior to 1.2.  If a test is missing a criteria.json file, and
> +has a ``reference.log`` file, then Fuego will read the ``reference.log`` file and use
> +it's data as the the pass criteria for the test.
> +
> +Previously, Fuego (and it's predecessor JTA) supported pass criteria
> +functionality in two different ways:
> +
> + * Functional test pass/fail counts
> + * Benchmark measure evaluations
> +
> +Functional test pass/fail counts
> +======================================
> +
> +For functional tests counts of positive and negative results were either
> +hard-coded into the base scripts for the test, as arguments to the
> +log_compare() in each test's test_processing() function, or they were specified
> +as variables, read from the board file, and applied in the test_processing()
> +function.
> +
> +For example, the Functional.OpenSSL test used values of 176 pass
> +and 86 fails (see ``fuego-core/engine/tests/Functional.OpenSSL/OpenSSL.sh`` in
> +fuego-1.1) to evaluate the result of this test.
> +
> +::
> +
> +    log_compare "$TESTDIR" "176" "${P_CRIT}" "p"
> +    log_compare "$TESTDIR" "86" "${N_CRIT}" "n"
> +
> +But tests in JTA, such as ``Functional.LTP.Open_Posix`` expected
> +the variables ``LTP_OPEN_POSIX_SUBTEST_COUNT_POS`` and
> +``LTP_OPEN_POSIX_SUBTEST_COUNT_NEG`` to be defined in a the board
> +file for the device under test.
> +
> +For example, the board file might have lines like the following:
> +
> +::
> +
> +  LTP_OPEN_POSIX_SUBTEST_COUNT_POS="1232"
> +  LTP_OPEN_POSIX_SUBTEST_COUNT_NEG="158"
> +
> +
> +These were used in the log_compare function of the base script of the test
> +like so:
> +
> +::
> +
> +   log_compare "$TESTDIR" $LTP_OPEN_POSIX_SUBTEST_COUNT_POS "${P_CRIT}" "p"
> +   log_compare "$TESTDIR" $LTP_OPEN_POSIX_SUBTEST_COUNT_NEG "${N_CRIT}" "n"
> +
> +
> +Starting with Fuego version 1.2, these would be replaced with ``criteria.json``
> +files like the following:
> +
> +For Functional.OpenSSL:
> +
> +::
> +
> +  {
> +      "schema_version":"1.0",
> +      "criteria":[
> +          'tguid': 'OpenSSL',
> +          'min_pass': 176,
> +          'max_fail': 86
> +      ]
> +  }
> +
> +
> +For Functional.LTP.Open_Posix:
> +
> +::
> +
> +  {
> +      "schema_version":"1.0",
> +      "criteria":[
> +          'tguid': 'LTP.Open_Posix',
> +          'min_pass': 1232,
> +          'max_fail': 158
> +      ]
> +  }
> +
> +::
> +
> + FIXTHIS - should there be 'default' somewhere in the preceding tguids?
> +
> +
> +Benchmark measure evaluations
> +===================================
> +
> +For Benchmark programs, the pass criteria consists of one or more measurement
> +thresholds that are compared with the results produced by the Benchmark, along
> +with the operator to be used for the comparison.
> +
> +In JTA and Fuego 1.1 this data was contained in the :ref:`reference.log` file.


I had to re-word-wrap several of the paragraphs here.  In general, I'd like
paragraphs wrapped at 72 chars.  A lot of the paragraphs were wider than that.

> diff --git a/docs/rst_src/parser.py.rst b/docs/rst_src/parser.py.rst
> new file mode 100644
> index 0000000..4b74f8b
> --- /dev/null
> +++ b/docs/rst_src/parser.py.rst
> @@ -0,0 +1,155 @@
> +##################
> +parser.py
> +##################
> +
> +===========
> +PROGRAM
> +===========
> +
> +``parser.py``
> +
> +===============
> +DESCRIPTION
> +===============
> +
> +``parser.py`` is a Python program that is used by each test to parse the
> +test log for a test run, check the threshold(s) for success or
> +failure, and store the data used to generate charts.
> +
> +Each benchmark should include an executable file called 'parser.py' in
``parser.py``

> +the test directory (/fuego-core/engine/tests/Benchmark.<testname>).
directory name should be literal-quoted.

> +Functional tests may also provide a ``parser.py``, when they return more
> +than a single testcase result from the test.  However, this is
> +optional.  If a Functional test does not have a parser.py script, then
> +a generic one is used (called ``generic_parser.py``), that just sets the
> +result for the test based on the return code from the test program and
> +the single result from running executing
> +function_log_compare in the function_test_processing portion of the test

Function names should either be converted to references, or literal-quoted.
In general, the wiki added prefixes to function names, to avoid name collisions
in the wiki page name space.  So wiki pages describing core functions
had the name of the function, prefixed with "function_", as their wiki page
filename.  Wiki pages describing parser functions had the name the of the function,
prefixed with "parser_func_", as their wiki page filename.

I think it would be good to keep this convention for the conversion to rst files.
But let me know what you think.

Whether we keep the prefixes on the function-describing filenames or not, I would
like to keep the text shown to users to just the function names, without the prefix.
This is why you see a lot of markup like the following on the wiki pages:

[[function_log_compare|log_compare]]

In wiki markup this created a link to the wiki page 'function_log_compare', with the
cover text of 'log_compare'.

When we get into converting the function reference pages, there will be a LOT of these
types of links, and we should decide ahead of time how we want to convert those page
names and link references, so that we can be consistent throughout the rst docs.

> +script.
> +
> +The test log for the current run is parsed by parser.py, and one or
> +more testcase results (measures or pass/fail results) are extracted,
> +and then provided via a dictionary to the results processing engine.
> +Normally this is done by scanning the log using simple regular
> +expressions. However, since this is a python program, an arbitrarily
> +complex parser can be written to extract result data from the test
> +log.
> +
> +Outline
> +=============
> +
> +The program usually has the following steps:
> +
> + * Import the parser library
> + * Specify a search pattern for finding one or more measurements (or
> +   testcases) from the test log
> + * Call the parser_func_parse_log function, to get a list of

Given the description above, I wasn't sure what do to with 'parser_func_parse_log',
but in this instance I did away with the link, and just literal-quoted the function
name (which ended up being ``parse_log``)

> +   matches for the search pattern
> + * Build a dictionary of result values
> + * Call the parser_func_process function, to save the information
``process``

> +   to the aggregate results files, and to re-generate the chart data for the test
> +
> +   * The process() function evaluates the results from the test, and determines
``process``

> +     the overall pass/fail status of a test, based on a :ref:`criteria.json` file
> +
> +
> +Testcase and measure names
> +==============================
> +
> +The parser.py program provides the name for the measures and testcases
``parser.py``

> +read from the test log file.  It also provides the result values for
> +these items, and passes the parsed data values to the processing
> +routine.
> +
> +These test names must be consistent in the parser.py program,
> +reference.json file and the criteria.json file.
> +
> +Please see :ref:`Fuego naming rules` for rules and guidelines
> +for test names in the Fuego system.
> +
> +
> +===========
> +SAMPLES
> +===========
> +
> +Here is a sample ``parser.py`` that does simple processing of a single
> +metric.  This is for Benchmark.Dhrystone.
> +
> +Note the two calls to parser library functions: parse_log() and process().
literal-quote these.

> +
> +::
> +
> +  #!/usr/bin/python
> +
> +  import os, re, sys
> +
> +  sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
> +  import common as plib
> +
> +  regex_string = "^(Dhrystones.per.Second:)(\ *)([\d]{1,8}.?[\d]{1,3})(.*)$"
> +
> +  measurements = {}
> +  matches = plib.parse_log(regex_string)
> +
> +  if matches:
> +      measurements['default.Dhrystone'] = [{"name": "Score", "measure" : float(matches[0][2])}]
> +
> +  sys.exit(plib.process(measurements))
> +
> +
> +
> +=============================
> +ENVIRONMENT and ARGUMENTS
> +=============================
> +
> +``parser.py`` uses the following environment variable:
> +
> + * FUEGO_CORE
``FUEGO_CORE``

> +
> +This is used to add ``/fuego-core/engine/scripts/parser`` to the python
> +system path, for importing the ``common.py`` module (usually as
> +internal module name 'plib').
> +
> +The parser library expects the following environment variables to be set:
> +
> + * ``FUEGO_RW``
> + * ``FUEGO_RO``
> + * ``FUEGO_CORE``
> + * ``NODE_NAME``
> + * ``TESTDIR``
> + * ``TESTSPEC``
> + * ``BUILD_NUMBER``
> + * ``BUILD_ID``
> + * ``BUILD_TIMESTAMP``
> + * ``PLATFORM``
> + * ``FWVER``
> + * ``LOGDIR``
> + * ``FUEGO_START_TIME``
> + * ``FUEGO_HOST``
> + * ``Reboot``
> + * ``Rebuild``
> + * ``Target_PreCleanup``
> + * ``WORKSPACE``
> + * ``JOB_NAME``
> +
> +``parser.py`` is called with the following invocation, from
> +function_processing:
> +
> +::
> +
> +  run_python $PYTHON_ARGS $FUEGO_CORE/engine/tests/${TESTDIR}/parser.py
> +
> +
> +
> +============
> +SOURCE
> +============
> +
> +Located in ``fuego-core/engine/tests/$TESTDIR/parser.py``.
> +
> +=============
> +SEE ALSO
> +=============
> +
> + * parser_func_parse_log, parser_func_process
> + * function_processing, :ref:`Parser module API`, Benchmark_parser_notes.
> --
> 2.7.4
> 
> 
> --

Thanks.  I have made some corrections, and pushed the resulting
commits to master on bitbucket.

 -- Tim