Re: [Fuego] [PATCH] docs: .rst files for pages categorized as Explanation, Tutorials and How Tos.
"Bird, Tim" <[email protected]> Sun, 4 Oct 2020 04:34:20 +0000
| Newsgroups | dev.linux.lists.fuego |
|---|---|
| Message-ID | <CY4PR13MB117520482D14B0F34DC9776CFD0F0@CY4PR13MB1175.namprd13.prod.outlook.com> |
Continuing the review of the large patch.... See comments inline below Unless otherwise mentioned, I have already made the suggested change and checked it in. If you disagree with the change, please let me know and we can discuss it. > -----Original Message----- > From: Pooja More <[email protected]> > > convert the following pages from the fuegotest wiki > into rst format and add to the Fuegl rst documentation directory: > Adding_a_Board, Adding_a_new_test, Adding_a_toolchain, > Adding_or_Customizing_a_Distribution, Adding_test_jobs_to_Jenkins, > Adding_views_to_Jenkins,Architecture,Artwork, Building_Documentation, > FAQ.rst, FrontPage, Fuego_Quickstart_Guide, Fuego_naming_rules, > Installing_Fuego, License_And_Contribution_Policy > OSS_Test_Vision, Parser_module_API, Quick_Setup_Guide, > Raspberry_Pi_Fuego_Setup,Test_variables, > Using_Batch_tests, Using_the_qemuarm_target. > > Signed-off-by: Pooja More <[email protected]> > --- > docs/rst_src/Adding_a_Board.rst | 242 +++++++------ > docs/rst_src/Adding_a_new_test.rst | 230 ++++++++----- > docs/rst_src/Adding_a_toolchain.rst | 224 +++++++++++- > .../Adding_or_Customizing_a_Distribution.rst | 130 ++++--- > docs/rst_src/Adding_test_jobs_to_Jenkins.rst | 136 +++++++- > docs/rst_src/Adding_views_to_Jenkins.rst | 68 ++-- > docs/rst_src/Architecture.rst | 375 +++++++++++---------- > docs/rst_src/Artwork.rst | 7 +- > docs/rst_src/Building_Documentation.rst | 25 +- > docs/rst_src/FAQ.rst | 49 +++ > docs/rst_src/FrontPage.rst | 43 ++- > docs/rst_src/Fuego_Quickstart_Guide.rst | 255 ++++++++++++++ > docs/rst_src/Fuego_naming_rules.rst | 126 ++++--- > docs/rst_src/Installing_Fuego.rst | 235 +++++++------ > docs/rst_src/License_And_Contribution_Policy.rst | 137 ++++---- > docs/rst_src/OSS_Test_Vision.rst | 349 +++++++++++++++++++ > docs/rst_src/Parser_module_API.rst | 114 ++++--- > docs/rst_src/Quick_Setup_Guide.rst | 161 +++++++++ > docs/rst_src/Raspberry_Pi_Fuego_Setup.rst | 73 ++-- > docs/rst_src/Test_variables.rst | 206 ++++++----- > docs/rst_src/Using_Batch_tests.rst | 240 +++++++------ > docs/rst_src/Using_the_qemuarm_target.rst | 38 ++- > docs/rst_src/Working_with_remote_boards.rst | 63 ++-- > docs/rst_src/index.rst | 4 +- > docs/rst_src/integration_with_ttc.rst | 83 +++-- > 25 files changed, 2575 insertions(+), 1038 deletions(-) > create mode 100644 docs/rst_src/FAQ.rst > create mode 100644 docs/rst_src/Fuego_Quickstart_Guide.rst > create mode 100644 docs/rst_src/OSS_Test_Vision.rst > create mode 100644 docs/rst_src/Quick_Setup_Guide.rst > .... [lots of lines skipped] ... > diff --git a/docs/rst_src/License_And_Contribution_Policy.rst b/docs/rst_src/License_And_Contribution_Policy.rst > index eea8bb0..6a34887 100644 > --- a/docs/rst_src/License_And_Contribution_Policy.rst > +++ b/docs/rst_src/License_And_Contribution_Policy.rst > @@ -11,8 +11,8 @@ License > > Fuego has the following license policy. > > -Fuego consists of several parts, and includes source code from > -a number of different external test projects. > +Fuego consists of several parts, and includes source code from a > +number of different external test projects. > > Default license > ================== > @@ -22,15 +22,15 @@ indicated in the LICENSE file at the top of the 'fuego' and > 'fuego-core' source repositories. > > If a file does not have an explicit license, or license indicator > -(such as SPDX identifier) in the file, than that file is > -covered by the default license for the project, with the exceptions > -noted below for "external test materials". > +(such as SPDX identifier) in the file, than that file is covered by > +the default license for the project, with the exceptions noted below > +for "external test materials". > > When making contributions, if you do NOT indicate an alternative > -license for your contribution, the contribution will be assigned > -the license of the file to which the contribution applies (which > -may be the default license, if the file contains no existing > -license indicator). > +license for your contribution, the contribution will be assigned the > +license of the file to which the contribution applies (which may be > +the default license, if the file contains no existing license > +indicator). > > Although we may allow for other licenses within the Fuego project > in order to accommodate external software added to our system, our > @@ -45,25 +45,26 @@ engine/tests/<test_name> (which is known as the test home directory), > and may include two types of materials: > > * 1) Fuego-specific files > - * 2) files obtained from external sources, which have their own license. > + * 2) files obtained from external sources, which have their own > + license. Remove bullet from numbered lists > > -The Fuego-specific materials consist of files such as: > -fuego_test.sh, spec.json, reference.json, test.yaml, chart_config.json, > -and possibly others as created for use in the Fuego project. > -External test materials may consist of tar files, helper scripts > -and patches against the source in the tar files. > +The Fuego-specific materials consist of files such as: fuego_test.sh, > +spec.json, reference.json, test.yaml, chart_config.json, and possibly Should make file references double-accent quotes (verbatim), like so: ``fuego_test.sh``, ``spec.json``, etc. > +others as created for use in the Fuego project. External test > +materials may consist of tar files, helper scripts and patches against > +the source in the tar files. > > Unless otherwise indicated, the Fuego-specific materials are > licensed under the Fuego default license, and the external test > materials are licensed under their own individual project > license - as indicated in the test source. > > -In some cases, there is no external source code, but only source > -that is originally written for Fuego and stored in the test > -home directory. This commonly includes tests based on a single > -shell script, that is written to be deployed to the Device Under > -Test by fuego_test.sh. Unless otherwise indicated, these files > -(source and scripts) are licensed under the Fuego default license. > +In some cases, there is no external source code, but only source that > +is originally written for Fuego and stored in the test home directory. > +This commonly includes tests based on a single shell script, that is > +written to be deployed to the Device Under Test by fuego_test.sh. > +Unless otherwise indicated, these files (source and scripts) are > +licensed under the Fuego default license. > > If there is any ambiguity in the category of a particular file > (external or Fuego-specific), please designate the intended license > @@ -72,20 +73,19 @@ clearly in the file itself, when making a contribution. > Copyright statements > ====================== > > -Copyrights for individual contributions should be added to > -individual files, when the contributions warrant copyright > -assignment. Some trivial fixes to existing code may not need > -to have copyright assignment, and thus not every change to a file > -needs to include a copyright notice for the contributor. > +Copyrights for individual contributions should be added to individual > +files, when the contributions warrant copyright assignment. Some > +trivial fixes to existing code may not need to have copyright > +assignment, and thus not every change to a file needs to include a > +copyright notice for the contributor. > > License tags > ============= > > -Our preference is to use SPDX license identifier, rather than a license > -notice, to indicate the license > -of any materials in Fuego. Such identifiers and notices are only > -desired if the materials are not contributed under the default Fuego > -license of "BSD-3-Clause". > +Our preference is to use SPDX license identifier, rather than a > +license notice, to indicate the license of any materials in Fuego. > +Such identifiers and notices are only desired if the materials are not > +contributed under the default Fuego license of "BSD-3-Clause". > > In a test.yaml, please indicate the license of the upstream > test program. If there is no upstream test program (ie, the > @@ -110,30 +110,45 @@ indicates agreement to the following: :: > > By making a contribution to this project, I certify that: > > - (a) The contribution was created in whole or in part by me and I > - have the right to submit it under the open source license > - indicated in the file; or > - > - (b) The contribution is based upon previous work that, to the best > - of my knowledge, is covered under an appropriate open source > - license and I have the right under that license to submit that > - work with modifications, whether created in whole or in part > - by me, under the same open source license (unless I am > - permitted to submit under a different license), as indicated > + (a) The contribution was created in whole or in > + part by me and I have the right to submit it > + under the open source license indicated in > + the file; or > + > + (b) The contribution is based upon previous work > + that, to the best of my knowledge, is > + covered under an appropriate open source > + license and I have the right under that > + license to submit that work with > + modifications, whether created in whole or > + in part by me, under the same open source > + license (unless I am permitted to submit > + under a different license), as indicated > in the file; or > > - (c) The contribution was provided directly to me by some other > - person who certified (a), (b) or (c) and I have not modified > - it. > - > - (d) I understand and agree that this project and the contribution > - are public and that a record of the contribution (including all > - personal information I submit with it, including my sign-off) is > - maintained indefinitely and may be redistributed consistent with > - this project or the open source license(s) involved. > - > - > -*Note: Please note that an "official" DCO at the web site* `<https://developercertificate.org/>`_ *has additional text (an LF copyright, > address, and statement of non-copyability).All of these are either nonsense or problematical in some legalsense. The above is a quote of a > portion of the document found in the Linuxkernel guide for submitting patches. See* > `<https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/submitting-patches.rst>`_ *(copied in > March, 2018).* > + (c) The contribution was provided directly to me > + by some other person who certified (a), (b) > + or (c) and I have not modified it. > + > + (d) I understand and agree that this project and > + the contribution are public and that a > + record of the contribution (including all > + personal information I submit with it, > + including my sign-off) is maintained > + indefinitely and may be redistributed > + consistent with this project or the open > + source license(s) involved. > + These lines were quoted verbatim from an external source, and their line wrapping should have been left exactly as is. > + > +*Note*: Please note that an "official" DCO at the web site Should be a .. note:: > +`<https://developercertificate.org/>`_ has additional text > +(an LF copyright, address, and statement of non-copyability).All of > +these are either nonsense or problematical in some legalsense. missing a space between legal and sense > +The above is a quote of a portion of the document found in the > +Linuxkernel guide for submitting patches. See missing a space between Linux and kernel > +`<https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/ > +tree/Documentation/process/submitting-patches.rst>`_ > +(copied in March, 2018). > > Each commit must include a DCO which looks like this :: > > @@ -156,13 +171,15 @@ Submitting contributions > ========================== > > Please format contributions as a patch, and send the patch to the > -`Fuego mailing list <https://lists.linuxfoundation.org/mailman/listinfo/fuego>`_ > +`Fuego mailing list <https://lists.linuxfoundation.org/mailman/ > +listinfo/fuego>`_ > > -Before making the patch, please verify that you have followed our preferred > -:ref:`Coding style <coding_style>`. > +Before making the patch, please verify that you have followed our > +preferred :ref:`Coding style <coding_style>`. > > -We follow the style of patches used by the Linux kernel, which is described > -here: `<https://www.kernel.org/doc/html/latest/process/submitting-patches.html>`_ > +We follow the style of patches used by the Linux kernel, which is > +described here: `<https://www.kernel.org/doc/html/latest/process/ > +submitting-patches.html>`_ > > Not everything described there applies, but please do the following: > - used a Signed-off-by line > @@ -176,7 +193,9 @@ Not everything described there applies, but please do the following: > > - the test name can be the short name, if it is unambiguous > > - - That is, please don't use the 'Functional' or 'Benchmark' prefix unless there are both types of tests with the same short name > + - That is, please don't use the 'Functional' or 'Benchmark' > + prefix unless there are both types of tests with the same > + short name > > - describe your changes in the commit message body > > diff --git a/docs/rst_src/OSS_Test_Vision.rst b/docs/rst_src/OSS_Test_Vision.rst > new file mode 100644 > index 0000000..bc8c317 > --- /dev/null > +++ b/docs/rst_src/OSS_Test_Vision.rst > @@ -0,0 +1,349 @@ > +.. _oss: > + > +################# > +OSS Test Vision > +################# > + > +This page describes aspects of the Open Source Test vision for the > +Fuego project, along with some ideas for implementing specific ideas > +related to this vision. > + > +===================== > +overview of concepts > +===================== > + > + > +Letter to ksummit discuss > +========================== > + > +Here's an e-mail Tim sent to the ksummit-discuss list in October, > +2016 :: > + > + I have some ideas on Open Source testing that I'd like to this whole section has leading tabs. In general I'd like for all tabs to be converted to spaces, throughout the documentation. tabs are notorious for introducing spacing problems (some people have tabs at 4 columns and some at 8). This is especially a problem for preformatted text, which all of this section is. I converted this whole section to leading spaces. > + throw out there for discussion. Some of these I have been > + stewing on for a while, while some came to mind after talking > + to people at recent conference events. > + > + Sorry - this is going to be long... > + > + First, it would be nice to increase the amount of testing we > + do, by having more test automation. (ok, that's a no-brainer). > + Recently there has been a trend towards more centralized > + testing facilities, like the zero-day stuff or board farms > + used by kernelci. That makes sense, as this requires > + specialized hardware, setup, or skills to operate certain > + kinds of test environments. As one example, an automated test > + of kernel boot requires automated control of power to a board > + or platform, which is not very common among kernel developers. > + A centralized test facility has the expertise and hardware to > + add new test nodes relatively cheaply. They can do this more > + quickly and much less expensively than the first such node by > + an individual new to testing. > + > + However, I think to make great strides in test quantity and > + coverage, it's important to focus on ease of use for > + individual test nodes. My vision would be to have tens of > + thousands of individual test nodes running automated tests on > + thousands of different hardware platforms and configurations > + and workloads. > + > + The kernel selftest project is a step in the right direction > + for this, because it allows any kernel developer to easily > + (in theory) run automated unit tests for the kernel. However, > + this is still a manual process. I'd like to see improved > + standards and infrastructure for automating tests. > + > + It turns out there are lots of manual steps in the testing > + and bug-fixing process with the kernel (and other > + Linux-related software). It would be nice if a new system > + allowed us to capture manual steps, and over time convert > + them to automation. > + > + Here are some problems with the manual process that I think > + need addressing: > + > + 1) How does an individual know what tests are valid for their > + platform? Currently, this is a manual decision. In a world > + with thousands or tens of thousands of tests, this will be > + very difficult. We need to have automated mechanisms to > + indicate which tests are relevant for a platform.Test > + definitions should include a description of the hardware > + they need,or the test setup they need. For example, it would > + be nice to have tests indicate that they need to be run on a > + node with USB gadget support, > + or on a node with the gadget hardware from a particular vendor > + (e.g. a particular SOC), or with a particular hardware phy > + (e.g. Synopsis). As another example, if a test requires that > + the hardware physically reboot,then that should be indicated > + in the test. If a test requires that a particular button be > + pressed (and that the button be available to be pressed), it > + should be listed. Or if the test requires that an external > + node be available to participate in the test (such as a wifi > + endpoint, CANbus endpoint, or > + i2C device) be present, that should be indicated. > + There should be a way for the test nodes which provide those > + hardware capabilities, setups, or external resources to > + identify themselves. Standards should > + be developed for how a test node and a test can express these > + capabilities and requirements. Also, standards need to be > + developed so that a test can control those external resources > + to participate in tests.Right now each test framework handles > + this in its own way (if it provides > + support for it at all). > + > + I heard of a neat setup at one company where the video > + output from a system was captured by another video system, > + and the results analyzed automatically. This type of test > + setup currently requires an enormous investment of > + expertise, and possibly specialized hardware. Once such a > + setup is performed in a few locations, it makes much more > + sense to direct tests that need such facilities to those > + locations, than it does to try to spread the expertise to > + lots of different individuals (although that certainly has > + value also). > + > + For a first pass, I think the kernel CONFIG variables needed > + by a test should be indicated, and they could be compared > + with the config for the device under test. This would be a > + start on the expression of the dependencies between a test > + and the features of the test node. > + > + 2) how do you connect people who are interested in a > + particular test with a node that can perform that test? > + > + My proposal here is simple - for every subsystem of the > + kernel, put a list of test nodes in the MAINTAINERS file, to > + indicate nodes that are available to test that subsystem. > + Tests can be scheduled to run on those nodes, either > + whenever new patches are received for that sub-system, or > + when a bug is encountered and developers for that subsystem > + want to investigate it by writing a new test. Tests or data > + collection instructions that are now provided manually would > + be converted to formal test definitions, and added to a > + growing body of tests. This should help people re-use test > + operations that are common. Capturing test operations that > + are done manually into a script would need to be very easy > + (possibly itself automated), and it would need to be easy to > + publish the new test for others to use. > + > + Basically, in the future, it would be nice if when a person > + reported a bug, instead of the maintainer manually walking > + someone through the steps to identify the bug and track down > + the problem, they could point the user at an existing test > + that the user could easily run. > + > + I imagine a kind of "test app store", where a tester can > + select from thousands of tests according to their interest. > + Also, people could rate the tests, and maintainers could > + point people to tests that are helpful to solve specific > + problems. > + > + 3) How does an individual know how to execute a test and how > + to interpret the results? > + > + For many features or sub-systems, there are existing tools > + (e.g bonnie for filesystem tests, netperf for networking > + tests, or cyclictest for realtime), but these tools have a > + variety of options for testing different aspects of a > + problem or for dealing with different configurations or > + setups. Online you can find tutorials for running each of > + these, and for helping people interpret the results. A new > + test system should take care of running these tools with the > + proper command line arguments for different test aspects, > + and for different test targets ('device-under-test's). > + > + For example, when someone figures out a set of useful > + arguments to cyclictest for testing realtime on a beaglebone > + board, they should be able to easily capture those arguments > + to allow another developer using the same board to easily > + re-use those test parameters, and interpret the cylictest > + results, in an automated fashion. Basically we want to > + automate the process of finding out "what options do I use > + for this test on this board, and what the heck number am I > + supposed to look at in this output, and what should its > + value be?". > + > + Another issue is with interpretation of test results from > + large test suites. One notorious example of this is LTP. > + It produces thousands of results, and almost always produces > + failures or results that can be safely ignored on a > + particular board or in a particular environment. It requires > + a large amount of manual evaluation and expertise to > + determine which items to pay attention to from LTP. It > + would be nice to be able to capture this evaluation, and > + share it with others with either the same board, or the same > + test environment, to allow them to avoid duplicating this > + work. > + > + Of course, this should not be used to gloss over bugs in LTP > + or bugs that LTP is reporting correctly and actually need to > + be paid attention to. > + > + 4) How should this test collateral be expressed, and how > + should it be collected, stored, shared and re-used? > + > + There are a multitude of test frameworks available. I am > + proposing that as a community we develop standards for test > + packaging which include this type of information (test > + dependencies, test parameters, results interpretation). I > + don't know all the details yet. For this reason I am coming > + to the community see how others are solving these problems > + and to get ideas for how to solve them in a way that would > + be useful for multiple frameworks. I'm personally working > + on the Fuego test framework - see http://fuegotest.org/wiki, > + but I'd like to create something that could be used with any > + test framework. > + > + 5) How to trust test collateral from other sources (tests, > + interpretation) > + > + One issue which arises with this type of sharing (or with > + any type of sharing) is how to trust the materials involved. > + If a user puts up a node with their own hardware, and trusts > + the test framework to automatically download and execute a > + never-before-seen test, this creates a security and trust > + issue. I believe this will require the same types of > + authentication and trust mechanisms (e.g. signing, > + validation and trust relationships) that we use to manage > + code in the kernel. > + > + I think this is more important than it sounds. I think the > + real value of this system will come when tens of thousands > + of nodes are running tests where the system owners can > + largely ignore the operation of the system, and instead the > + test scheduling and priorities can be driven by the needs of > + developers and maintainers who the test node owners have > + never interacted with. > + > + Finally, 6) What is the motivation for someone to run a test > + on their hardware? > + > + Well, there's an obvious benefit to executing a test if you > + are personally interested in the result. However, I think > + the benefit of running an enormous test system needs to be > + de-coupled from that immediate direct benefit. I think we > + should look at this the same way we look at other > + crowd-sourced initiatives, like Wikipedia. While there is > + some small benefit for someone producing an individual page > + edit, we need to move beyond that to the benefit to the > + community of the cumulative effort. > + > + I think that if we want tens of thousands of people to run > + tests, then we need to increase the cost/benefit ratio for > + the system. First, you need to reduce the cost so that it > + is very cheap, in all of [time|money|expertise| ongoing > + attention], to set up and maintain a test node. Second, > + there needs to be a real benefit that people can measure > + from the cumulative effect of participating in the system. > + I think it would be valuable to report bugs found and fixed > + by the system as a whole, and possibly to attribute positive > + results to the output provided by individual nodes. (Maybe > + you could 'game-ify' the operation of test nodes.) > + > + Well, if you are still reading by now, I appreciate it. I > + have more ideas, including more details for how such a > + system might work, and what types of things it could > + accomplish. But I'll save that for smaller groups who might > + be more directly interested in this topic. > + > + To get started, I will begin working on a prototype of a > + test packaging system that includes some of the ideas > + mentioned here: inclusion of test collateral, and package > + validation. I would also like to schedule a "test summit" > + of some kind (maybe associated with ELC or Linaro Connect, > + or some other event), to discuss standards in the area I > + propose. > + > + I welcome any response to these ideas. I plan to discuss > + them at the upcoming test framework mini-jamboree in Tokyo > + next week, and at Plumbers (particularly during the 'testing > + and fuzzing' session) the week following. But feel free to > + respond to this e-mail as well. > + > + Thanks. > + -- Tim Bird > + > + > +============================= > +Ideas related to the vision > +============================= > + > + > +Capturing tests easily > +======================== > + > + * should be easy to capture a command line sequence, and test the > + results > + * maybe do an automated capture and format into a clitest file that > + can be used at a here document inside a fuego test script? > + > +================== > +test collateral > +================== Heading should usually be capitalized. > + > + * does it need to be board-specific > + * elements of test collateral: > + > + * test dependencies: > + > + * kernel config values needed > + * kernel features needed: > + > + * proc filesystem > + * sys filesystem > + * trace filesystem > + * test hardware needed > + * test node setup features > + > + * ability to reboot the board > + * ability to soft-reset the board > + * ability to install a new kernel > + * presence of certain programs on target > + > + * bc > + * top, ps, /bin/sh, bash? > + * already have: > + > + * CAPABILITIES? > + * pn and reference logs > + * positive and negative result counts (specific to board) > + * test specs indicate parameters for the test > + * test plans indicate different profiles (method to match test to > + test environment - e.g. filesystem test with type of filesystem > + hardware) > + > +================= > +test app store > +================= > + > + * need a repository where tests can be downloaded > + > + * like Jenkins plugin repository > + * like debian package feed > + > + * need a client for browsing tests, installing tests, updating tests > + * store a test in github, and just refer to different tests in > + different git repositories? > + * test ratings > + * test metrics (how many bugs found) > + > +====================== > +authenticating tests > +====================== > + > + * need to prevent malicious tests > + * packages should be signed by an authority, after review by someone > + > + * who? the Fuego maintainers? This would turn into a bottleneck > + > +====================== > +test system metrics > +====================== > + > + * number of bugs found and fixed in upstream software > + * number of bugs found and fixed in test system > + * bug categories (See :ref:`Metrics <metrics>`) > + > + > + > + > diff --git a/docs/rst_src/Parser_module_API.rst b/docs/rst_src/Parser_module_API.rst > index 617c0ff..e7f81e7 100644 > --- a/docs/rst_src/Parser_module_API.rst > +++ b/docs/rst_src/Parser_module_API.rst > @@ -7,45 +7,60 @@ Parser module API > > > > -The file common.py is the python module for performing benchmark log file processing, and results processing and aggregation. > +The file common.py is the python module for performing benchmark log > +file processing, and results processing and aggregation. > > It is used by the parser.py program from the test directory, to process > the log after each test run. The data from a test run is processed to: > > - * check numeric values for pass/fail result(by checking against a reference threshold values) > - * determine the overall result of the test, based on potentially complex results criteria > + * check numeric values for pass/fail result(by checking against a > + reference threshold values) > + * determine the overall result of the test, based on potentially > + complex results criteria > * save the data for use in history and comparison charts Usually want to capitalize first word of bulleted items. > > =============== > Parser API > =============== > > -The following are functions used during log processing, by a test's parser.py program. > +The following are functions used during log processing, by a test's > +parser.py program. > > - * :ref:`parse_log() <parser_func_parse_log>` - parse the data from a test log > + * :ref:`parse_log() <parser_func_parse_log>` - parse the data from a > + test log > > - * this routine takes a regular expression, with one or more groups, and results a list of tuples for lines that matched the expression > - * the tuples consist of the strings from the matching line corresponding to the regex groups > + * this routine takes a regular expression, with one or more > + groups, and results a list of tuples for lines that matched the > + expression > + * the tuples consist of the strings from the matching line > + corresponding to the regex groups > > * :ref:`process() <parser_func_process>` - process results from a test > > - * this routine taks a dictionary of test results, and does 3 things: > + * this routine taks a dictionary of test results, and does 3 > + things: > > * formats them into the run.json file (run results file) > * detects pass or fail by using the specified pass criteria > * formats the data into charts (plots and tables) > > - * :ref:`split_output_per_testcase() <parser_func_split_output_per_testcase>` - split testlog into chunks accessible from the Jenkins user > interface (one per testcase) > + * :ref:`split_output_per_testcase() > + <parser_func_split_output_per_testcase>` > > -In general, a parser module will normally call **parse_log()**, then take > -the resulting list of matching groups to construct a dictionary to pass > -to the **process()** routine. > + - split testlog into chunks accessible from the Jenkins user > + interface (one per testcase) > + > +In general, a parser module will normally call **parse_log()**, then I switched the quoting to double-accent (verbatim), for function names. Usually, I try to have function names use the reference (as shown above) Since the function reference pages haven't been converted yet, I'm not sure what the link will look like. I don't want the function names to look too different from each other, depending on whether they are a link or not. Using verbatim does this. Do you know of a way to make the cover text for a link show up as a verbatim item? > +take the resulting list of matching groups to construct a dictionary > +to pass to the **process()** routine. > > If the log file format is amendable, the parser module may also call > -split_output_per_testcase() to generate a set of files from the testlog, > -that can be referenced from the charts generated by the charting module. > +split_output_per_testcase() to generate a set of files from the > +testlog, that can be referenced from the charts generated by the > +charting module. > > -Please see :ref:`parser.py <parser>` for more details and examples of use of the API. > +Please see :ref:`parser.py <parser>` for more details and examples of > +use of the API. > > > > @@ -54,11 +69,12 @@ Please see :ref:`parser.py <parser>` for more details and examples of use of the > Deprecated API > =================== > > -*Note: The following information is for historical purposes only.Although the API is still present in Fuego, these APIs are deprecated.* > +*Note: The following information is for historical purposes only.* This should be a .. note:: > +*Although the API is still present in Fuego, these APIs are deprecated.* > > In Fuego version 1.1 and prior, the following functions were used. > -These are still available for backwards compatibility with tests written > -for these versions of Fuego. > +These are still available for backwards compatibility with tests > +written for these versions of Fuego. > > * parse() > * process_data() > @@ -74,26 +90,28 @@ parse() > > * output: > > - * list of regular expression matches for each line matching the specified pattern > + * list of regular expression matches for each line matching the > + specified pattern > > -This routine scans the current log file, using a regular expression. It > -returns an re match object for each line of the log file that matches the > -expression. > +This routine scans the current log file, using a regular expression. > +It returns an re match object for each line of the log file that > +matches the expression. > > -This list is used to populate a dictionary of metric/value pairs that can > -be passed to the process_data function. > +This list is used to populate a dictionary of metric/value pairs that > +can be passed to the process_data function. > > process_data > ============= > > -This is the main routine of the module. It processes the list of metrics, > -and populates various output files for test. > +This is the main routine of the module. It processes the list of > +metrics, and populates various output files for test. > > * input: > > * ref_section_pat - regular expression used to read reference.log > * cur_dict - dictionary of metric/value pairs > - * m - indicates the size of the plot. It should be one of: 's', 'm', 'l', 'xl' > + * m - indicates the size of the plot. It should be one of: 's', > + 'm', 'l', 'xl' > > * if 'm', 'l', or 'xl' are used, then a multiplot is created > > @@ -116,13 +134,17 @@ functions in common.py > ======================== > > * hls - print a big warning or error message > - * parse_log(regex_str) - specify a regular expression string to use to parse lines in the log > + * parse_log(regex_str) - specify a regular expression string to use > + to parse lines in the log > > - * this is a helper function that returns a list of matches (with groups) that the parser.py can use to populate its dictionary of > measurements > + * this is a helper function that returns a list of matches > + (with groups) that the parser.py can use to populate its > + dictionary of measurements > > * parse(regex_compiled_object) > > - * similar to parse_log, but it takes a compiled regular expression object, and returns a list of matches (with groups) > + * similar to parse_log, but it takes a compiled regular expression > + object, and returns a list of matches (with groups) > > * this is deprecated, but left to support legacy tests > > @@ -149,7 +171,8 @@ functions in common.py > > * key=test_case_id (not including measure name) > > - * for a functional test, the test_case_id is usually "default.<test_name>" > + * for a functional test, the test_case_id is usually > + "default.<test_name>" > > * value=list of measures (for a benchmark) > * or value=string (PASS|FAIL|SKIP) (for a functional test) > @@ -191,35 +214,44 @@ call trees > miscellaneous notes > ======================== > > - * create_default_ref_tim (for docker.hello-fail.Functional.hello_world) > + * create_default_ref_tim > + (for docker.hello-fail.Functional.hello_world) > > - * ref={'test_sets': [{'test_cases': [{'measurements': [{'status': 'FAIL', 'name': 'Functional'}], 'name': 'default'}], 'name': 'default'}]} > + * ref={'test_sets': [{'test_cases': [{'measurements': > + [{'status': 'FAIL', 'name': 'Functional'}], 'name': 'default'}], > + 'name': 'default'}]} > > * create_default_ref > > - * ref={'test_sets': [{'test_cases': [{'status': 'FAIL', 'name': 'default'}], 'name': 'default'}]} > + * ref={'test_sets': [{'test_cases': [{'status': 'FAIL', > + 'name': 'default'}], 'name': 'default'}]} > > data format and tguid rules > ==================================== > > -The current API and the old parser API take different data and allow different > -test identifiers. This sections explains the difference: > +The current API and the old parser API take different data and allow > +different test identifiers. This sections explains the difference: > > Data format for benchmark test with new API > > - * measurements[test_case_id] = [{"name": measure_name, "measure": value}] > + * measurements[test_case_id] = [{"name": measure_name, > + "measure": value}] > > Data format for benchmark test with old API: > > * in reference.log > > - * if tguid is a single word, then use that word as the measure name and "default" as the test_case. > + * if tguid is a single word, then use that word as the > + measure name and "default" as the test_case. > > - * e.g. for benchmark.arm, the reference.log has "short". This becomes the fully-qualified tguid: arm.default.arm.short: > + * e.g. for benchmark.arm, the reference.log has "short". > + This becomes the fully-qualified tguid: arm.default.arm.short: > > - * test_name = arm, test_case = default, test_case_id = arm, measure = short > + * test_name = arm, test_case = default, test_case_id = arm, > + measure = short > > -Data format for functional tests with new API and the old API is the same: > +Data format for functional tests with new API and the old API is the > +same: > > * e.g. measurements["status"] = "PASS|FAIL" > > diff --git a/docs/rst_src/Quick_Setup_Guide.rst b/docs/rst_src/Quick_Setup_Guide.rst This page was converted to Install_and_First_Test.rst, and reworded to be a tutorial. ... [patch body omitted] ... > diff --git a/docs/rst_src/Raspberry_Pi_Fuego_Setup.rst b/docs/rst_src/Raspberry_Pi_Fuego_Setup.rst > index be8516e..7ce2a43 100644 > --- a/docs/rst_src/Raspberry_Pi_Fuego_Setup.rst > +++ b/docs/rst_src/Raspberry_Pi_Fuego_Setup.rst > @@ -4,14 +4,14 @@ > Raspberry Pi Fuego Setup > ######################### > > -This is a list of instructions for setting up a Raspberry Pi board > -for use with Fuego. These instructions will help you set up the > -ssh server, used by Fuego to communicate with the board, and the > -test directory on the machine, that Fuego will use to store programs > -and files during a test. > +This is a list of instructions for setting up a Raspberry Pi board for > +use with Fuego. These instructions will help you set up the ssh > +server, used by Fuego to communicate with the board, and the test > +directory on the machine, that Fuego will use to store programs and > +files during a test. > > -These instructions and the screen shots are for a Raspberry Pi Model 3 B, > -running "Raspbian 9 (stretch)". > +These instructions and the screen shots are for a Raspberry Pi Model 3 > +B, running "Raspbian 9 (stretch)". > > This assumes that the Raspberry Pi is already installed, and that > networking is already configured and running. > @@ -20,21 +20,22 @@ networking is already configured and running. > Obtain your network address > ============================== > > -First, determine what your Pi's network address is. You can see this by using > -the command 'ifconfig' in a terminal window, and checking for the 'inet' address. > +First, determine what your Pi's network address is. You can see this > +by using the command 'ifconfig' in a terminal window, and checking for > +the 'inet' address. > > -Or, move your mouse cursor over the network icon in the desktop panel bar. > -If you leave the mouse there for a second or two, a box will appear showing > -information about your current network connection. > +Or, move your mouse cursor over the network icon in the desktop panel > +bar. If you leave the mouse there for a second or two, a box will > +appear showing information about your current network connection. > > -This is what the network information box looks like (in the upper right corner > -of this screen shot): > +This is what the network information box looks like (in the upper > +right corner of this screen shot): > > .. image:: ../images/rpi-network-address.png > :height: 400 > > -In this case, my network address is 10.0.1.103. > -Your address might start with 192.168, which is common for home or local networks. > +In this case, my network address is 10.0.1.103. Your address might > +start with 192.168, which is common for home or local networks. > > Note this address for use later. > > @@ -46,14 +47,14 @@ Configure the SSH server > In order for other machines to access the Pi remotely, you need to > enable the ssh server. > > -This is done by enabling the SSH interface in the Raspberry Pi Configuration > -dialog. > +This is done by enabling the SSH interface in the Raspberry Pi > +Configuration dialog. > > -To access this dialog, click on the raspberry logo in the upper right corner > -of the main desktop window. Then click on "Preferences", then on > -"Raspberry Pi Configuration". In the dialog that appears, click on the > -"Interfaces" tab, and on the list of interfaces click on the "Enable" > -radio button for the SSH interface. > +To access this dialog, click on the raspberry logo in the upper right > +corner of the main desktop window. Then click on "Preferences", then > +on "Raspberry Pi Configuration". In the dialog that appears, click on > +the "Interfaces" tab, and on the list of interfaces click on the > +"Enable" radio button for the SSH interface. > > Here is the menu: > > @@ -69,8 +70,8 @@ The configuration dialog looks something like this: > Try connecting > ================ > > -Now, close this dialog, and make sure you can access the Pi using > -SSH from your host machine. > +Now, close this dialog, and make sure you can access the Pi using SSH > +from your host machine. > > > Try the following command, from your host machine: > @@ -91,8 +92,8 @@ This is not recommended on machines that are in production, as it is > a significant security risk. However, for test machines it may be > acceptable to allow root access over ssh. > > -To do this, on the Raspberry Pi, with root permissions, edit the file /etc/ssh/sshd_config > -and add the following line: :: > +To do this, on the Raspberry Pi, with root permissions, edit the file > +/etc/ssh/sshd_config and add the following line: :: > > PermitRootLogin yes > > @@ -126,9 +127,9 @@ information. > > ``$ adduser fuego`` > > -Answer the questions, including setting the password for this > -account. Remember the password you select, and use that in the board > -file when configuring Fuego to access this board. > +Answer the questions, including setting the password for this account. > +Remember the password you select, and use that in the board file when > +configuring Fuego to access this board. > > This will create the directory ``/home/fuego``. > > @@ -196,7 +197,8 @@ Inside the Fuego container, run: :: > $ ftc add-job -b rpi -t Functional.fuego_board_check > > > -An easy way to populate Jenkins with a set of tests is to install a batch test. > +An easy way to populate Jenkins with a set of tests is to install a > +batch test. > > Install the "smoketest" batch test, as follows: > > @@ -211,11 +213,12 @@ Run a board check > To see if everything is set up correctly, execute the test: > Functional.fuego_board_check. > > -In the Jenkins interface, select "rpi.default.Functional.fuego_board_check" > -and select the menu item "Build Now" on the left hand side of the screen. > +In the Jenkins interface, select > +"rpi.default.Functional.fuego_board_check" and select the menu item > +"Build Now" on the left hand side of the screen. > > -Wait a few moments for the test to complete. when the test completes, check > -the log for the test by clicking on the link to the 'testlog'. > +Wait a few moments for the test to complete. when the test completes, > +check the log for the test by clicking on the link to the 'testlog'. > > > > diff --git a/docs/rst_src/Test_variables.rst b/docs/rst_src/Test_variables.rst > index 51717c0..a20c262 100644 > --- a/docs/rst_src/Test_variables.rst > +++ b/docs/rst_src/Test_variables.rst > @@ -12,12 +12,13 @@ When Fuego executes a test, shell environment variables are used to > provide information about the test environment, test execution > parameters, communications methods and parameters, and other items. > > -These pieces of information are originate from numerous different places. > -An initial set of test variables comes in the shell environment from > -either Jenkins or from the shell in which ftc is executed (depending > -on which one is used to invoke the test). > +These pieces of information are originate from numerous different > +places. An initial set of test variables comes in the shell > +environment from either Jenkins or from the shell in which ftc is > +executed (depending on which one is used to invoke the test). > > -The information about the board being tested comes primarily from two sources: > +The information about the board being tested comes primarily from two > +sources: > > * the board file > * the stored board variables file Usually good to capitalize bulleted items > @@ -37,41 +38,57 @@ test execution. > Board file > ============== > > -The board file contains static information about a board. It is processed > -by the overlay system, and the values inside it appear as variables > -in the environment of a test, during test execution. > +The board file contains static information about a board. It is > +processed by the overlay system, and the values inside it appear as > +variables in the environment of a test, during test execution. > > The board file resides in: > > * /fuego-ro/boards/$BOARD.board We need a convention for the variable portion of a command or file. The above notation is intended to convey that the first part of the board filename is dependent on the boards name (in the $BOARD variable). I'm not sure that the syntax makes this obvious. Another alternative is: * /fuego-ro/boards/{board_name}.board Or, maybe we should make it italic: * /fuego-ro/boards/*board_name*.board the latter one prevents us from verbatim-quoting the path: * ``/fuego-ro/boards/{board_name}.board`` What do you think? I'm leaning towards the last form, but I'm not sure. > > -There are a number of variables which are used by the Fuego system itself, > -and there may also be variables that are used by individual tests. > +There are a number of variables which are used by the Fuego system > +itself, and there may also be variables that are used by individual > +tests. > > Common board variables > ========================= > + > Here is a list of the variables which might be found in a board file: > + > * ARCHITECTURE - specifies the architecture of the board > * BAUD - baud rate for serial device (if using 'serial' transport) > * BOARD_TESTDIR - directory on board where tests are executed > - * BOARD_CONTROL - the mechanism used to control board hardware (e.g. hardware reboot) > - * DISTRIB - filename of distribution overlay file (if not the default) > - * IO_TIME_SERIAL - serial port delay parameter (if using 'serial' transport) > + * BOARD_CONTROL - the mechanism used to control board hardware > + (e.g. hardware reboot) > + * DISTRIB - filename of distribution overlay file > + (if not the default) > + * IO_TIME_SERIAL - serial port delay parameter > + (if using 'serial' transport) > * IPADDR - network address of the board > * LOGIN - specifies the user account to use for Fuego operations > - * PASSWORD - specifies the password for the user account on the board used by Fuego > + * PASSWORD - specifies the password for the user account on the board > + used by Fuego > * PLATFORM - specifies the toolchain to use for the platform > - * SATA_DEV - specifies a filesystem device node (on the board) for SATA filesystem tests > - * SATA_MP - specifies a filesystem mount point (on the board) for SATA filesystem tests > - * SERIAL - serial device on host for board's serial console (if using 'serial' transport) > - * SRV_IP - network address of server endpoint, for networking tests (if not the same as the host) > - * SSH_KEY - the absolute path to key file with ssh key for password-less ssh operations (e.g. "/fuego-ro/board/myboard_id_rsa") > - * SSH_PORT - network port of ssh daemon on board (if using ssh transport) > + * SATA_DEV - specifies a filesystem device node (on the board) for > + SATA filesystem tests > + * SATA_MP - specifies a filesystem mount point (on the board) > + for SATA filesystem tests > + * SERIAL - serial device on host for board's serial console > + (if using 'serial' transport) > + * SRV_IP - network address of server endpoint, for networking tests > + (if not the same as the host) > + * SSH_KEY - the absolute path to key file with ssh key for > + password-less ssh operations (e.g. "/fuego-ro/board/myboard_id_rsa") > + * SSH_PORT - network port of ssh daemon on board (if using > + ssh transport) > * TRANSPORT - this specifies the transport to use with the target > - * USB_DEV - specifies a filesystem device node (on the board) for USB filesystem tests > - * USB_MP - specifies a filesystem mount point (on the board) for USB filesystem tests > + * USB_DEV - specifies a filesystem device node (on the board) for > + USB filesystem tests > + * USB_MP - specifies a filesystem mount point (on the board) for > + USB filesystem tests Should this list be a definition list in RST? I'm not sure. I think a definition list would make every definition 4 lines long, which is not as compact as this. I might just want to make each variable name verbatim (double-accent quoted) to have the variable name stand out in this list. So something like this: * ``USB_MP`` - specifies a filesystem ... Do we have a rule about making variable names verbatim? If so, this would be consistent. > > -See :ref:`Adding a Board <adding_board>` for more details about these variables. > +See :ref:`Adding a board <adding_board>` for more details about these > +variables. > > A board may also have additional variables, including variables that > are used for results evaluation for specific tests. > @@ -102,25 +119,30 @@ The overlay system is described in greater detail here: > Stored variables > ======================= > > -Stored board variables are test variables that are defined on a per-board > -basis, and can be modified and managed under program control. > +Stored board variables are test variables that are defined on a > +per-board basis, and can be modified and managed under program > +control. > > Stored variables allow the Fuego system, a test, or a user to store > -information that can be used by tests. This essentially > -creates an information cache about the board, that can be both > -manually and programmatically generated and managed. > - > -The information that needs to be held for a particular board depends on the tests that are installed in the system. Thus the system needs > to support > -ad-hoc collections of variables. Just putting everything into the static board > -file would not scale, as the number of tests increases. > - > -''Note: the LAVA test framework has a similar concept called a *board dictionary*.'' > - > -One use case for this to have a "board setup" test, that scans for lots of > -different items, and populates the stored variables with values that > -are used by other tests. Some items that are useful to know about a board > -take time to discover (using e.g. 'find' on the target board), and using > -a board dynamic variable can help reduce the time required to check these items. > +information that can be used by tests. This essentially creates an > +information cache about the board, that can be both manually and > +programmatically generated and managed. > + > +The information that needs to be held for a particular board depends > +on the tests that are installed in the system. Thus the system needs > +to support ad-hoc collections of variables. Just putting everything > +into the static board file would not scale, as the number of tests > +increases. > + > +*Note: the LAVA test framework has a similar concept called* > +*a board dictionary.* should be a .. note:: > + > +One use case for this to have a "board setup" test, that scans for > +lots of different items, and populates the stored variables with > +values that are used by other tests. Some items that are useful to > +know about a board take time to discover (using e.g. 'find' on the should be ``find`` > +target board), and using a board dynamic variable can help reduce the > +time required to check these items. > > The board stored variables are kept in the file: > * /fuego-rw/boards/$BOARD.vars > @@ -130,21 +152,27 @@ These variables are included in the test by the overlay generator. > Commands for interacting with stored variables > ==================================================== > > -A user or a test can manipulate a board stored variable using the ftc command. > -The following commands can be used to set, query and delete variables: > +A user or a test can manipulate a board stored variable using the ftc > +command.The following commands can be used to set, query and delete > +variables: > > - * **tc query-board** - to see test variables (both regular board variables and stored variables) > + * **tc query-board** - to see test variables (both regular board > + variables and stored variables) > * **ftc set-var** - to add or update a stored variable > * **ftc delete-var** - to delete a stored variable These commands should be converted to verbatim quotes: ``tc query-board`` - ... > > ftc query-board > ------------------ > > -'ftc query-board' is used to view the variables associated with a Fuego board. > -You can use the command to see all the variables, or just a single variable. > +'ftc query-board' is used to view the variables associated with a Commands should be verbatim-quoted: ``ftc query-board`` > +Fuego board. You can use the command to see all the variables, or > +just a single variable. > > -Note that 'ftc query-board' shows the variables for a test that come from both the board file and board stored variables file (that is, both > 'static' board > -variables and stored variables). It does not show variables which come from testplans or spec files, as those are specific to a test. > +Note that 'ftc query-board' shows the variables for a test that come > +from both the board file and board stored variables file (that is, > +both 'static' board variables and stored variables). It does not show > +variables which come from testplans or spec files, as those are > +specific to a test. > > The usage is: > * ftc query-board <board> [-n <VARIABLE>] > @@ -154,18 +182,20 @@ Examples: > $ ftc query-board myboard -n PROGRAM_BC > > The first example would show all board variables, including functions. > -The second example would show only the variable PROGRAM_BC, if it existed, for board 'myboard'. > +The second example would show only the variable PROGRAM_BC, if it > +existed, for board 'myboard'. > > ftc set-var > ------------ > > -'ftc set-var' allows setting or updating the value of a board stored variable. > +'ftc set-var' allows setting or updating the value of a board stored > +variable. > > The usage is: > * ftc set-var <board> <VARIABLE>=<value> > > -By convention, variable names are all uppercase, and function names are > -lowercase, with words separated by underscores. > +By convention, variable names are all uppercase, and function names > +are lowercase, with words separated by underscores. > > Example: > $ ftc set-var PROGRAM_BC=/usr/bin/bc This example should be put in an offset box, using Example: :: In general, any examples with actual lines of commands should be in an offset box. > @@ -181,40 +211,43 @@ Example: > Example usage > ============== > > -Functional.fuego_board_check could detect the path for the 'foo' binary, > -(e.g. is_on_target foo PROGRAM_FOO) > -and call 'ftc set-var $NODE_NAME PROGRAM_FOO=$PROGRAM_FOO'. > -This would stay persistently defined as a test variable, so other > -tests could use $PROGRAM_FOO (with assert_defines, or in > -'report' or 'cmd' function calls.) > +Functional.fuego_board_check could detect the path for the 'foo' > +binary, (e.g. is_on_target foo PROGRAM_FOO) and call 'ftc set-var > +$NODE_NAME PROGRAM_FOO=$PROGRAM_FOO'. This would stay persistently > +defined as a test variable, so other tests could use $PROGRAM_FOO > +(with assert_defines, or in 'report' or 'cmd' function calls.) > > > Example Stored variables > ========================= > > -Here are some examples of variables that can be kept as stored variables, > -rather than static variables from the board file: > +Here are some examples of variables that can be kept as stored > +variables, rather than static variables from the board file: > > * SATA_DEV = Linux device node for SATA file system tests > * SATA_MP = Linux mount point for SATA file system tests > - * LTP_OPEN_POSIX_SUBTEST_COUNT_POS = expected number of pass results for LTP OpenPosix test > - * LTP_OPEN_POSIX_SUBTEST_COUNT_NEG = expected number of fail results for LTP OpenPosix test > + * LTP_OPEN_POSIX_SUBTEST_COUNT_POS = expected number of pass results > + for LTP OpenPosix test > + * LTP_OPEN_POSIX_SUBTEST_COUNT_NEG = expected number of fail results > + for LTP OpenPosix test > * PROGRAM_BC = path to 'bc' program on the target board > - * MAX_REBOOT_RETRIES = number of retries to use when rebooting a board > + * MAX_REBOOT_RETRIES = number of retries to use when rebooting a > + board verbatim quote the variables in this list as above. > > =================== > Spec variables > =================== > -A test spec can define one or more variables to be used with a test. These are commonly > -used to control test variations, and are specified in a spec.json file. > +A test spec can define one or more variables to be used with a test. > +These are commonly used to control test variations, and are specified > +in a spec.json file. > > When a spec file defines a variable associated with a named test spec, > -the variable is read by the overlay generator on test execution, and the > -variable name is prefixed with the name of the test, and converted to > -all upper case. > +the variable is read by the overlay generator on test execution, and > +the variable name is prefixed with the name of the test, and converted > +to all upper case. > > -For example, support a test called "Functional.foo" had a test spec that > -defined the variable 'args' with a line > +For example, support a test called "Functional.foo" had a test spec > +that defined the variable 'args' with a line > like the following in its spec.json file: :: > > "default": { > @@ -223,9 +256,11 @@ like the following in its spec.json file: :: > > > When the test was run with this spec (the "default" spec), then the > -variable FUNCTIONAL_FOO_ARGS would be defined, with the value "-v -p2". > +variable FUNCTIONAL_FOO_ARGS would be defined, with the value "-v > +-p2". ``FUNCTIONAL_FOO_ARGS`` > > -See :ref:`Test_Specs_and_Plans <test_specs_and_plans>` for more information about specs and plans. > +See :ref:`Test_Specs_and_Plans <test_specs_and_plans>` for more > +information about specs and plans. > > Note that spec variables are overridden by > > @@ -233,21 +268,22 @@ Note that spec variables are overridden by > Dynamic variables > ========================= > > -Another category of variables used during testing are dynamic variables. > -These variables are defined on the command line of 'ftc run-test' using > -the '--dynamic-vars' option. > +Another category of variables used during testing are dynamic > +variables. These variables are defined on the command line of 'ftc > +run-test' using the '--dynamic-vars' option. > > The purpose of these variables is to allow scripted variations when > -running 'ftc run-test' The scripted variables are processed and presented > -the same way as Spec variables, which is to say that the variable name > -is prefixed with the test name, and converted to all upper case. > +running 'ftc run-test' The scripted variables are processed and > +presented the same way as Spec variables, which is to say that the > +variable name is prefixed with the test name, and converted to all > +upper case. Commands options, and paths should be verbatim-quoted. > > For example, if the following command was issued: > > * ftc run-test -b beaglebone -t Functional.foo --dynamic_vars *ARGS=-p* > > -then during test execution the variable *FUNCTIONAL_FOO_ARGS* would be defined > -with the value *-p*. > +then during test execution the variable *FUNCTIONAL_FOO_ARGS* would be > +defined with the value *-p*. > > See :ref:`Dynamic Variables <dynamic_variables>` for more information. > > @@ -255,10 +291,12 @@ See :ref:`Dynamic Variables <dynamic_variables>` for more information. > Variable precedence > ======================== > > -Here is the precedence of variable definition for Fuego, during test execution: > +Here is the precedence of variable definition for Fuego, during test > +execution: > > (from lowest to highest) > - * environment variable (from Jenkins or shell where 'ftc run-test' is invoked) > + * environment variable (from Jenkins or shell where 'ftc run-test' is > + invoked) > * board variable (from fuego-ro/boards/$BOARD.board file) > * stored variable (from fuego-rw/boards/$BOARD.vars file) > * spec variable (from spec.json file) > @@ -266,9 +304,9 @@ Here is the precedence of variable definition for Fuego, during test execution: > * core variable (from Fuego scripts) > * fuego_test variable (from fuego_test.sh) > > -Spec and dynamic variables are prefixed with the test name, and converted > -to upper case. That tends to keep them in a separate name space from the > -rest of the test variables. > +Spec and dynamic variables are prefixed with the test name, and > +converted to upper case. That tends to keep them in a separate name > +space from the rest of the test variables. > > > Empty lines should be removed at the end of the file (and at the end of sections) > diff --git a/docs/rst_src/Using_Batch_tests.rst b/docs/rst_src/Using_Batch_tests.rst > index 985221c..176dfd4 100644 > --- a/docs/rst_src/Using_Batch_tests.rst > +++ b/docs/rst_src/Using_Batch_tests.rst > @@ -5,28 +5,31 @@ Using Batch Tests > ########################## > > > -A "batch test" in Fuego is a Fuego test that runs a series of other tests > -as a group. The results of the individual tests are consolidated into > -a list of testcase results for the batch test. > +A "batch test" in Fuego is a Fuego test that runs a series of other > +tests as a group. The results of the individual tests are > +consolidated into a list of testcase results for the batch test. > > -Prior to Fuego version 1.5, there was a different feature, called "testplans", > -which allowed users to compose sets of tests into logical groups, and run > -them together. The batch test system, introduced in Fuego version 1.5 > -replaces the testplan system. > +Prior to Fuego version 1.5, there was a different feature, called > +"testplans", which allowed users to compose sets of tests into logical > +groups, and run them together. The batch test system, introduced in > +Fuego version 1.5 replaces the testplan system. > > ============================= > How to make a batch test > ============================= > > -A batch test consists of a Fuego test that runs other tests. A Fuego batch > -test is similar to other Fuego tests, in that the test definition lives > -in fuego-core/tests/<test-name>, and it consists of a fuego_test.sh file, > -a spec file, a parser.py, a test.yaml file and possibly other files. > +A batch test consists of a Fuego test that runs other tests. A Fuego > +batch test is similar to other Fuego tests, in that the test > +definition lives in fuego-core/tests/<test-name>, and it consists of a > +fuego_test.sh file, a spec file, a parser.py, a test.yaml file and > +possibly other files. should verbatim-quote files, paths, variables, and command line options. This applies throughout. I won't comment on all of them in this message. > > -The difference is that a Fuego batch test runs other Fuego tests, as a group. > -The batch test has a few elements that are different from other tests. > +The difference is that a Fuego batch test runs other Fuego tests, as a > +group. The batch test has a few elements that are different from > +other tests. > > -Inside the fuego_test.sh file, a batch test must define two main elements: > +Inside the fuego_test.sh file, a batch test must define two main > +elements: > > * the testplan element > * the test_run function, with commands to run other tests > @@ -34,17 +37,19 @@ Inside the fuego_test.sh file, a batch test must define two main elements: > Testplan element > ========================= > > -The testplan element consists of data assigned to the shell variable BATCH_TESTPLAN. This variable contains lines > -that specify, in machine-readable form, the tests that are part of the batch job. > -The testplan is specified in json format, and is used to specify the > -attributes (such as timeout, flags, and specs) for each test. > -The testplan element is used by 'ftc add-jobs' to create Jenkins jobs for > +The testplan element consists of data assigned to the shell variable > +BATCH_TESTPLAN. This variable contains lines that specify, in > +machine-readable form, the tests that are part of the batch job. The > +testplan is specified in json format, and is used to specify the > +attributes (such as timeout, flags, and specs) for each test. The > +testplan element is used by 'ftc add-jobs' to create Jenkins jobs for > each sub-test that is executed by this batch test. > > -The BATCH_TESTPLAN variable must be defined in the fuego_test.sh file. The > -definition must begin with a > -line starting with the string 'BATCH_TESTPLAN=' and end with a line starting with the string 'END_TESTPLAN'. > -By convention this is defined as a shell "here document", like this example: :: > +The BATCH_TESTPLAN variable must be defined in the fuego_test.sh file. > +The definition must begin with a line starting with the string > +'BATCH_TESTPLAN=' and end with a line starting with the string > +'END_TESTPLAN'. By convention this is defined as a shell "here > +document", like this example: :: > > > BATCH_TESTPLAN=$(cat <<END_TESTPLAN > @@ -59,13 +64,17 @@ By convention this is defined as a shell "here document", like this example: :: > ) > > > -The lines of the testplan follow the format described at :ref:`Testplan_Reference <testplan_reference>`. Please see that page for details > about the plan fields and structure (the schema for the testplan data). > +The lines of the testplan follow the format described at > +:ref:`Testplan_Reference <testplan_reference>`. Please see that page > +for details about the plan fields and structure (the schema for the > +testplan data). > > test_run function > ==================== > > -The other element in a batch test's fuego_test.sh is a test_run function. > -This function is used to actually execute the tests in the batch. > +The other element in a batch test's fuego_test.sh is a test_run > +function. This function is used to actually execute the tests in the > +batch. > > There are two functions that are available to help with this: > > @@ -76,14 +85,17 @@ The body of the test_run function for a batch test usually has a few > common elements: > > * setting of the FUEGO_BATCH_ID > - * execution of the sub-tests, using a call to the function :ref:`run_test <func_run_test>` for each one > + * execution of the sub-tests, using a call to the function > + :ref:`run_test <func_run_test>` for each one > > -Here are the commands in the test_run function for the test ``Functional.batch_hello``: :: > +Here are the commands in the test_run function for the test > +``Functional.batch_hello``: :: > > function test_run { > export TC_NUM=1 > DEFAULT_TIMEOUT=3m > - export FUEGO_BATCH_ID="hello-$(allocate_next_batch_id)" > + export FUEGO_BATCH_ID="hello- > + $(allocate_next_batch_id)" > > # don't stop on test errors > set +e > @@ -101,19 +113,22 @@ Here are the commands in the test_run function for the test ``Functional.batch_h > Setting the batch_id > ---------------------------- > > -Fuego uses a 'batch id' to indicate that a group of test runs are related. > -Since a single Fuego test can be run in many different ways (e.g. from > -the command line or from Jenkins, triggered manually or automatically, > -or as part of one batch test or another), it is helpful for the run data for > -a test to be assigned a batch_id that can be used to generate reports or visualize data for the group of tests that are part of the batch. > - > -A batch test should set the FUEGO_BATCH_ID for the run to a unique string > -for that run of the batch test. Each sub-test will store the batch id > -in its run.json file, and this can be used to filter run data in subsequent > -test operations. The Fuego system can provide a unique number, via the > -routine :ref:`allocate_next_batch_id <func_allocate_next_batch_id>`. By > -convention, the batch_id for a test is created by combining a test-specific > -prefix string with the number returned from ``allocate_next_batch_id``. > +Fuego uses a 'batch id' to indicate that a group of test runs are > +related. Since a single Fuego test can be run in many different ways > +(e.g. from the command line or from Jenkins, triggered manually or > +automatically, or as part of one batch test or another), it is helpful > +for the run data for a test to be assigned a batch_id that can be used > +to generate reports or visualize data for the group of tests that are > +part of the batch. > + > +A batch test should set the FUEGO_BATCH_ID for the run to a unique > +string for that run of the batch test. Each sub-test will store the > +batch id in its run.json file, and this can be used to filter run data > +in subsequent test operations. The Fuego system can provide a unique > +number, via the routine :ref:`allocate_next_batch_id > +<func_allocate_next_batch_id>`. By convention, the batch_id for a > +test is created by combining a test-specific prefix string with the > +number returned from ``allocate_next_batch_id``. > > In the example above, the prefix used is 'hello-', and this would be > followed by a number returned by allocate_next_batch_id. > @@ -121,74 +136,82 @@ followed by a number returned by allocate_next_batch_id. > Executing sub-tests > ---------------------- > > -The :ref:`run_test <func_run_test>` function is used to execute the sub-tests > -that are part of the batch. The other portions of the example above > -show setting various shell variables that are used by 'run_test', and > -turning off 'errexit' mode while the sub-tests are running. > +The :ref:`run_test <func_run_test>` function is used to execute the > +sub-tests that are part of the batch. The other portions of the > +example above show setting various shell variables that are used by > +'run_test', and turning off 'errexit' mode while the sub-tests are > +running. > > -In the example above, TC_NUM, TC_NAME, and DEFAULT_TIMEOUT are used for > -various effects. These variables are optional, and in most cases a > -batch test can be written without having to set them. Fuego will generate > -automatic strings or values for these variables if they are not defined > -by the batch test. > +In the example above, TC_NUM, TC_NAME, and DEFAULT_TIMEOUT are used > +for various effects. These variables are optional, and in most cases > +a batch test can be written without having to set them. Fuego will > +generate automatic strings or values for these variables if they are > +not defined by the batch test. > > -Please see the documentation for :ref:`run_test <func_run_test>` for details > -about the environment and arguments used when calling the function. > +Please see the documentation for :ref:`run_test <func_run_test>` for > +details about the environment and arguments used when calling the > +function. > > Avoiding stopping on errors > ---------------------------------------- > > -The example above shows use of 'set +e' and 'set -e' to control the shell's > -'errexit' mode. By default, Fuego runs tests with the shell errexit > -mode enabled. However, a batch test should anticipate that some of its > -sub-tests might fail. If you want all of the tests in the batch to run, > -even if some of them fail, they you should use 'set +e' to disable > -errexit mode, and 'set -e' to re-enable it when you are done. > - > -Of course, if you want the batch test to stop if one of the sub-tests fails, they you should control the errexit mode accordingly (for > example, leaving it > -set during all sub-test executions, or disabling it or enabling it > -only during the execution of particular sub-tests). > - > -Whether to manipulate the shell errexit mode or not depends on what the > -batch test is doing. If it is implementing a sequence of dependent test > -stages, the errexit mode should be left enabled. If a batch test is > -implementing a series of unrelated, independent tests, the errexit mode > -should be disabled and re-enabled as shown. > +The example above shows use of 'set +e' and 'set -e' to control the > +shell's 'errexit' mode. By default, Fuego runs tests with the shell > +errexit mode enabled. However, a batch test should anticipate that > +some of its sub-tests might fail. If you want all of the tests in the > +batch to run, even if some of them fail, they you should use 'set +e' > +to disable errexit mode, and 'set -e' to re-enable it when you are > +done. > + > +Of course, if you want the batch test to stop if one of the sub-tests > +fails, they you should control the errexit mode accordingly (for > +example, leaving it set during all sub-test executions, or disabling > +it or enabling it only during the execution of particular sub-tests). > + > +Whether to manipulate the shell errexit mode or not depends on what > +the batch test is doing. If it is implementing a sequence of > +dependent test stages, the errexit mode should be left enabled. If a > +batch test is implementing a series of unrelated, independent tests, > +the errexit mode should be disabled and re-enabled as shown. > > ================ > test output > ================ > > The run_test function logs test results in a format similar to TAP13. > -This consists > -of the test output, followed by a line starting with the batch id > -(inside double brackets), then "ok" or "not ok" to indicate the sub-test result, followed by the testcase number and testcase name. > +This consists of the test output, followed by a line starting with the > +batch id (inside double brackets), then "ok" or "not ok" to indicate > +the sub-test result, followed by the testcase number and testcase > +name. > > -A standard parser.py for this syntax is available and used by other > -batch tests in the system (See fuego-core/tests/Functional.batch_hello/parser.py) > +A standard parser.py for this syntax is available and used by other > +batch tests in the system (See > +fuego-core/tests/Functional.batch_hello/parser.py) > > ======================================== > Preparing the system for a batch job > ======================================== > > -In order to run a batch test from Jenkins, you need to define a Jenkins > -job for the batch test, and jobs for all of the sub-tests that are called > -by the batch test. > +In order to run a batch test from Jenkins, you need to define a > +Jenkins job for the batch test, and jobs for all of the sub-tests that > +are called by the batch test. > > -You can use 'ftc add-jobs' with the batch test, and Fuego will create the > -job for the batch test itself as well as jobs for all of its sub-tests. > +You can use 'ftc add-jobs' with the batch test, and Fuego will create > +the job for the batch test itself as well as jobs for all of its > +sub-tests. > > -It is possible to run a batch test from the command line using 'ftc run-test', without > -creating Jenkins jobs. However if you want to see the results of the test in > -the Jenkins interface, then the Jenkins test jobs need to be defined prior to > -running the batch test from the command line. > +It is possible to run a batch test from the command line using 'ftc > +run-test', without creating Jenkins jobs. However if you want to see > +the results of the test in the Jenkins interface, then the Jenkins > +test jobs need to be defined prior to running the batch test from the > +command line. > > =========================== > Executing a batch test > =========================== > > -A batch test is executed the same way as any other Fuego test. > -Once installed as a Jenkins job, you can execute it using the Jenkins > +A batch test is executed the same way as any other Fuego test. Once > +installed as a Jenkins job, you can execute it using the Jenkins > interface (manually), or use Jenkins features to cause it to trigger > automatically. Or, you can run the test from the command line using > 'ftc run-test'. > @@ -205,25 +228,29 @@ You can view results from a batch test in two ways: > Jenkins batch test results tables > ===================================== > > -Inside the Jenkins interface, a batch job will display the list of sub-tests, > -and the PASS/FAIL status of each one. In addition, if there is a Jenkins > -job associated with a particular sub-test, there will be a link in the > -table cell for that test run, that you can click to see that individual > -test's result and data in the Jenkins interface. > +Inside the Jenkins interface, a batch job will display the list of > +sub-tests, and the PASS/FAIL status of each one. In addition, if > +there is a Jenkins job associated with a particular sub-test, there > +will be a link in the table cell for that test run, that you can click > +to see that individual test's result and data in the Jenkins > +interface. > > > Generating a report > ====================== > > -You can view a report for a batch test, by specifying the batch_id with > -the 'ftc gen-report' command. > +You can view a report for a batch test, by specifying the batch_id > +with the 'ftc gen-report' command. > > -To determine the batch_id, look at the log for the batch test (testlog.txt file). Or, generate a report listing the batch_ids for the batch test, > like so: > +To determine the batch_id, look at the log for the batch test > +(testlog.txt file). Or, generate a report listing the batch_ids for > +the batch test, like so: > > - * $ ``ftc gen-report --where test=batch_<name> --fields timestamp,batch_id`` > + * $ ``ftc gen-report --where test=batch_<name> --fields > + timestamp,batch_id`` > > -Select an appropriate batch_id from the list that appears, and note it for > -use in the next command. > +Select an appropriate batch_id from the list that appears, and note it > +for use in the next command. > > Now, to see the results from the individual sub-tests in the batch, use > the desired batch_id as part of a ''where'' clause, like so: > @@ -239,20 +266,21 @@ Miscelaneous notes > Timeouts > ========== > > -The timeout for a batch test should be long enough for all sub-tests to complete. When a batch test is launched from Jenkins, the board > on which > -it will run is reserved and will be unavailable for tests until the entire > -batch is complete. Keep this in mind when executing batch tests that > -call sub-tests that have a long duration. > +The timeout for a batch test should be long enough for all sub-tests > +to complete. When a batch test is launched from Jenkins, the board on > +which it will run is reserved and will be unavailable for tests until > +the entire batch is complete. Keep this in mind when executing batch > +tests that call sub-tests that have a long duration. > > The timeout for individual sub-tests can be specified multiple ways. > First, the timeout listed in the testplan (embedded in fuego_test.sh > -as the BATCH_TESTPLAN variable) is the one assigned to the Jenkins > -job for the sub-test, when jobs are created during test installation into Jenkins. > -These take effect when a sub-test is run independently from the batch > -test. > +as the BATCH_TESTPLAN variable) is the one assigned to the Jenkins job > +for the sub-test, when jobs are created during test installation into > +Jenkins. These take effect when a sub-test is run independently from > +the batch test. > > -If you want to specify a non-default timeout for a test, then you > -must use a --timeout argument to the run_test function, for that sub-test. > +If you want to specify a non-default timeout for a test, then you must > +use a --timeout argument to the run_test function, for that sub-test. > > > Should remove trailing blank lines from the file. > diff --git a/docs/rst_src/Using_the_qemuarm_target.rst b/docs/rst_src/Using_the_qemuarm_target.rst > index 02cdcc6..0978199 100644 > --- a/docs/rst_src/Using_the_qemuarm_target.rst > +++ b/docs/rst_src/Using_the_qemuarm_target.rst > @@ -7,8 +7,8 @@ Using the qemuarm target > Here are some quick instructions for using the qemuarm target that is > preinstalled in fuego. > > -Fuego does not ship with a qemuarm image in the repository, but assumes > -that you have built one with the Yocto Project. > +Fuego does not ship with a qemuarm image in the repository, but > +assumes that you have built one with the Yocto Project. > > If you don't have one lying around, you will need to build one. Then > you should follow the other steps on this page to configure it to run > @@ -18,14 +18,18 @@ with Fuego. > Build a qemuarm image > ========================= > > -Here are some quick steps for building a qemuarm image using the Yocto Project: > -(See the `Project Quick Start <http://www.yoctoproject.org/docs/2.1/yocto-project-qs/yocto-project-qs.html|Yocto>`_, for more > information) > +Here are some quick steps for building a qemuarm image using the Yocto > +Project: (See the `Project Quick Start > +<http://www.yoctoproject.org/docs/2.1/yocto-project-qs/ > +yocto-project-qs.html|Yocto>`_, > +for more information) > > Note that these steps are for Ubuntu. > > * make sure you have required packages for building the software > > - * sudo apt-get install gawk wget git-core diffstat unzip texinfo gcc-multilib build-essential chrpath socat libsdl1.2-dev xterm > + * sudo apt-get install gawk wget git-core diffstat unzip texinfo > + gcc-multilib build-essential chrpath socat libsdl1.2-dev xterm I decided to put a lot of the steps on this page into offset blocks. This one included. > > * install the qemu software > > @@ -41,7 +45,8 @@ Note that these steps are for Ubuntu. > * source oe-init-build-env build-qemuarm build-qemuarm > * edit conf/local.conf > > - * Under the comment about "Machine Selection", uncomment the line 'MACHINE ?= "qemuarm"' > + * Under the comment about "Machine Selection", uncomment the line > + 'MACHINE ?= "qemuarm"' > > * build a minimal image (this will take a while) Usually should capitalize bulleted items. > > @@ -74,8 +79,9 @@ Of course, substitute the correct IP address in the commands above. > > Once you know that things are working, directly connecting from the > host to the qemuarm image, make sure the correct values are in the > -qemu-arm.board file. You can edit this file inside the fuego container > -at /fuego-ro/boards/qemu-arm.board, or on your host in fuego-ro/boards/qemu-arm.board > +qemu-arm.board file. You can edit this file inside the fuego > +container at /fuego-ro/boards/qemu-arm.board, or on your host in > +fuego-ro/boards/qemu-arm.board > > Here are the values you should set: > > @@ -88,21 +94,23 @@ Here are the values you should set: > Test building software > ========================== > > -It is important to be able to build the test software for the image you > -are using with qemu. > +It is important to be able to build the test software for the image > +you are using with qemu. > > The toolchain used to compile programs for a board is controlled via > the PLATFORM variable in the board file. Currently the qemu-arm.board > -file specifies PLATFORM="qemu-armv7hf". Unfortunately, in my own testing > -that toolchain won't produce a binary that runs with a core-image-minimal > -image from YP Poky. > +file specifies PLATFORM="qemu-armv7hf". Unfortunately, in my own > +testing that toolchain won't produce a binary that runs with a > +core-image-minimal image from YP Poky. > > You may need to install your Yocto Project SDK into fuego, in order to > successfully build programs for the platforms. > > -See :ref:`Adding a toolchain <addtoolchain>` for information about how to do that. > +See :ref:`Adding a toolchain <addtoolchain>` for information about how > +to do that. > > -Try building a simple program, like hello_world, as a test for the new system, and see what happens. > +Try building a simple program, like hello_world, as a test for the new > +system, and see what happens. Overall - verbatim-quote paths, programs, variables, etc. > > > And remove trailing blank lines. > diff --git a/docs/rst_src/Working_with_remote_boards.rst b/docs/rst_src/Working_with_remote_boards.rst > index 40f377b..7e4e325 100644 > --- a/docs/rst_src/Working_with_remote_boards.rst > +++ b/docs/rst_src/Working_with_remote_boards.rst > @@ -4,69 +4,78 @@ > Working with remote boards > ################################### > > -Here are some general tips for working with remote boards > -(that is, boards in remote labs) > +Here are some general tips for working with remote boards (that is, > +boards in remote labs) > > ========================== > using a jump server > ========================== > > -If you have an SSH jump server, then you can access > -machine directly in another lab, using the ssh ProxyCommand > -in the host settings for a board. > +If you have an SSH jump server, then you can access machine directly > +in another lab, using the ssh ProxyCommand in the host settings for a > +board. > > I found this page to be helpful: > `<https://www.tecmint.com/access-linux-server-using-a-jump-host/>`_ > > -You should try to make each leg of the jump (from local machine to jump server, > -and from jump server to remote machine) password-less. > +You should try to make each leg of the jump (from local machine to > +jump server, and from jump server to remote machine) password-less. > > -I found that if my local machine's public key was in the remote machine's > -authorized keys file, then I could log in without a password, even if > -the jump server's public key was not in the remote machine's authorized keys > -file. > +I found that if my local machine's public key was in the remote > +machine's authorized keys file, then I could log in without a > +password, even if the jump server's public key was not in the remote > +machine's authorized keys file. > > ================================== > Using ttc transport remotely > ================================== > > -If you have a server that already has ttc configured for a bunch of board, > -you can accomplish a lot just by referencing ttc commands on that server. > +If you have a server that already has ttc configured for a bunch of > +board, you can accomplish a lot just by referencing ttc commands on > +that server. > > For example, in your local ttc.conf, you can put: :: > > PASSWORD=foo > USER=myuser > - SSH_ARGS=-o UserKnownHostsFile=/dev/null -o StrictHostKeychecking=no -o LogLevel=QUIET > + SSH_ARGS=-o UserKnownHostsFile=/dev/null -o > + StrictHostKeychecking=no -o LogLevel=QUIET > > pos_cmd=ssh timdesk ttc %%(target)s pos OK - here's a new one. Anything converted from the wiki pages that has double-percent signs (%%), should be converted to single-percent signs. The reason for this, is that the wiki software uses percent-sign to escape a percent-sign. In any event, all these double-percents should be changed to single-percents. I've added that guideline here: http://fuegotest.org/wiki/rst_docs#some_guidelines > off_cmd=ssh timdesk ttc %%(target)s off > on_cmd=ssh timdesk ttc %%(target)s on > reboot_cmd=ssh timdesk ttc %%(target)s reboot > > - login_cmd=sshpass -p %%(PASSWORD)s ssh %%(SSH_ARGS)s -x %%(USER)s@%%(target)s > - run_cmd=sshpass -p %%(PASSWORD)s ssh %%(SSH_ARGS)s -x %%(USER)s@%%(target)s "$COMMAND" > - copy_to_cmd=sshpass -p %%(PASSWORD)s scp %%(SSH_ARGS)s $src %%(USER)s@%%(target)s:/$dest > - copy_from_cmd=sshpass -p %%(PASSWORD)s scp %%(SSH_ARGS)s %%(USER)s@%%(target)s:/$src $dest > + login_cmd=sshpass -p %%(PASSWORD)s ssh %%(SSH_ARGS)s -x > + %%(USER)s@%%(target)s > + run_cmd=sshpass -p %%(PASSWORD)s ssh %%(SSH_ARGS)s -x > + %%(USER)s@%%(target)s "$COMMAND" > + copy_to_cmd=sshpass -p %%(PASSWORD)s scp %%(SSH_ARGS)s > + $src %%(USER)s@%%(target)s:/$dest > + copy_from_cmd=sshpass -p %%(PASSWORD)s scp %%(SSH_ARGS)s > + %%(USER)s@%%(target)s:/$src $dest > > > -Please note that 'ttc status <remote-board>' does not work with ttc version 1.4.4. > -This is due to internal usage of %%(ip_addr)s in the function network_status(), > -which will not be correct for the remote-board. > +Please note that 'ttc status <remote-board>' does not work with ttc > +version 1.4.4. This is due to internal usage of %%(ip_addr)s in the > +function network_status(), which will not be correct for the > +remote-board. > > ============================================================= > setting up ssh ProxyCommand in the Fuego docker container > ============================================================= > > -Please note that tests in Fuego are executed inside the docker container as user 'jenkins'. > +Please note that tests in Fuego are executed inside the docker > +container as user 'jenkins'. > > -In order to set up password-less operation, or use of a jump server or ProxyCommand, > -you have to add appropriate items (config and keys) to: > +In order to set up password-less operation, or use of a jump server or > +ProxyCommand, you have to add appropriate items (config and keys) to: > /var/lib/jenkins/.ssh > > -Please note that this may make your docker container a security risk, as it may expose > -your private keys to tests. Please use caution when adding private keys or other > -sensitive security information to the docker container. > +Please note that this may make your docker container a security risk, > +as it may expose your private keys to tests. Please use caution when > +adding private keys or other sensitive security information to the > +docker container. The word 'docker' when used to refer to the Docker system, should be capitalized. I changed several on this page. The same is true for the word 'jenkins', when used to refer to the Jenkins system (as opposed to a directory name, or user name). > > > diff --git a/docs/rst_src/index.rst b/docs/rst_src/index.rst > index e359897..5a3dbff 100644 > --- a/docs/rst_src/index.rst > +++ b/docs/rst_src/index.rst > @@ -83,6 +83,7 @@ Index > :hidden: > > Fuego_Quickstart_Guide > + Quick_Setup_Guide.rst This is now Install_and_First_Test.rst, and is already listed elsewhere in the index.rst. > Raspberry_Pi_Fuego_Setup > Using_the_qemuarm_target > > @@ -169,7 +170,4 @@ Indices and tables > The following is to hide a warning. > FrontPage.rst is included rather than referenced using toctree > > -.. toctree:: > - :hidden: > > - FrontPage > diff --git a/docs/rst_src/integration_with_ttc.rst b/docs/rst_src/integration_with_ttc.rst > index 1d53784..1dc484b 100644 > --- a/docs/rst_src/integration_with_ttc.rst > +++ b/docs/rst_src/integration_with_ttc.rst Same stuff with this one as others. I converted command names, filenames, paths, variable names, and command line options to verbatim-quoted. > @@ -11,8 +11,8 @@ board farms, and for doing kernel development on multiple different > target boards at a time (including especially boards with varying > processors and architectures.) > > -This page describes how ttc and fuego can be integrated, so that the fuego > -test framework can use 'ttc' as it's transport mechanism. > +This page describes how ttc and fuego can be integrated, so that the > +fuego test framework can use 'ttc' as it's transport mechanism. > > You can find more information about 'ttc' on the linux wiki at: > http://elinux.org/Ttc_Program_Usage_Guide > @@ -23,39 +23,51 @@ Outline of supported functionality > > Here is a rough outline of the support for 'ttc' in fuego: > > - * Integration for the tool and helper utilities in the container build > + * Integration for the tool and helper utilities in the container > + build > > - * When the docker container is built, ttc is downloaded from github and installed into the docker image. > - * During this process, the path to the ttc.conf file is changed from /etc/ttc.conf to /fuego-ro/conf/ttc.conf > + * When the docker container is built, ttc is downloaded from github > + and installed into the docker image. > + * During this process, the path to the ttc.conf file is changed > + from /etc/ttc.conf to /fuego-ro/conf/ttc.conf > > * 'ttc' is now a valid transport option > > - * You can specify ttc as the 'transport' for a board, instead of ssh > + * You can specify ttc as the 'transport' for a board, instead of > + ssh > > * ttc now supports -r as an option to the 'ttc cp' command > > - * this is required since fuego uses -r extensively to do recursive directory copies (See :ref:`Transport_notes <transport_notes>` for > details) > + * this is required since fuego uses -r extensively to do recursive > + directory copies (See :ref:`Transport_notes <transport_notes>` > + for details) > > - * fuego-core has been modified to avoid using wildcards on 'get' operations > + * fuego-core has been modified to avoid using wildcards on 'get' > + operations > > * a new test called Functional.fuego_transport has been added > > - * this tests use of wildcards, multiple files and directories and directory recursion with the 'put' command. > - * it also indirectly tests the 'get' command, because logs are obtained during the test. > + * this tests use of wildcards, multiple files and directories and > + directory recursion with the 'put' command. > + * it also indirectly tests the 'get' command, because logs are > + obtained during the test. > > > ========================== > Supported operations > ========================== > > -ttc has several sub-commands. Fuego currently only uses the following ttc sub-commands: > +ttc has several sub-commands. Fuego currently only uses the following > +ttc sub-commands: > > * 'ttc run' - to run a command on the target > - * 'ttc cp' - to get a file from the target, and to put files to the target > + * 'ttc cp' - to get a file from the target, and to put files to the > + target > > -Note that some other commands, such as 'ttc reboot' are not used, in spite of there > -being similar functionality provided in fuego (see > -:ref:`function target reboot <func_target_reboot>` and :ref:`function ov rootfs reboot <func_ov_rootfs_reboot>`. > +Note that some other commands, such as 'ttc reboot' are not used, in > +spite of there being similar functionality provided in fuego (see > +:ref:`function target reboot <func_target_reboot>` and :ref:`function > +ov rootfs reboot <func_ov_rootfs_reboot>`). > > Finally, other commands, such as 'ttc get_kernel', 'ttc get_config', > 'ttc kbuild' and 'ttc kinstall' are not used currently. These may be > @@ -80,22 +92,32 @@ Steps to use ttc with a target board > > Here is a list of steps to set up a target board to use ttc. > These steps assume you have already added a board to fuego > -following the steps described in :ref:`Adding a Board <adding_board>`. > +following the steps described in :ref:`Adding a board <adding_board>`. > > - * If needed, create your docker container using 'docker-create-usb-privileged-container.sh > + * If needed, create your docker container using 'docker-create-usb- > + privileged-container.sh' > > - * This may be needed if you are using ttc with board controls that require access to USB devices (such as the Sony debug board) > - * substitute this command in place of 'docker-create-container.sh' in the `Fuego Quickstart Guide > <http://fuegotest.org/wiki/Fuego_Quickstart_Guide#Download,_build,_start_and_access>`_. > + * This may be needed if you are using ttc with board controls that > + require access to USB devices (such as the Sony debug board) > + * substitute this command in place of 'docker-create-container.sh' > + in the `Fuego Quickstart Guide <http://fuegotest.org/wiki/Fuego_ > + Quickstart_Guide#Download,_build,_start_and_access>`_. > > - * Make sure that /userdata/conf/ttc.conf has the definitions required for your target board > + * Make sure that /userdata/conf/ttc.conf has the definitions required > + for your target board > > - * Validate this by doing 'ttc list' to see that the board is present, and 'ttc run' and 'ttc cp' commands, to test that these operations work > with the board, from inside the container. > + * Validate this by doing 'ttc list' to see that the board is > + present, and 'ttc run' and 'ttc cp' commands, to test that these > + operations work with the board, from inside the container. > > - * Edit the fuego board file (found in /userdata/conf/boards/<somthing>.board > + * Edit the fuego board file (found in /userdata/conf/boards > + /<somthing>.board) > > * Set the TRANSPORT to 'ttc' > - * Set the TTC_TARGET variable is set to the name for the target used by ttc > - * See the following example, for a definition for a target named 'bbb' (for my beaglebone black board):: > + * Set the TTC_TARGET variable is set to the name for the target > + used by ttc > + * See the following example, for a definition for a target named > + 'bbb' (for my beaglebone black board):: > > > TRANSPORT=ttc > @@ -105,10 +127,11 @@ following the steps described in :ref:`Adding a Board <adding_board>`. > modify your copy_to_cmd > =========================== > > -In your ttc.conf file, you may need to make changes to any copy_to_cmd definitions. Fuego allows programs to pass a '-r' argument to its > internal > -'put' command, which in turn invokes ttc's cp command, with the source as target > -and destination as the host. In other words, it ends up invokings ttc's > -'copy_from_cmd' for the indicated target. > +In your ttc.conf file, you may need to make changes to any copy_to_cmd > +definitions. Fuego allows programs to pass a '-r' argument to its > +internal 'put' command, which in turn invokes ttc's cp command, with > +the source as target and destination as the host. In other words, it > +ends up invokings ttc's 'copy_from_cmd' for the indicated target. > > All versions of copy_to_cmd should be modified to > reference a new environment variable $copy_args. > @@ -122,11 +145,7 @@ use to execute a copy_to_cmd. > See examples in ttc.conf.sample and ttc.conf.sample2 for usage examples. > > > -.. toctree:: > - :hidden: > > - Transport_notes > - > > > > -- > 2.7.4 > > > -- Thanks for all this work. Let me know if you have any questions. Please keep working on the conversion process. -- Tim