Re: [Fuego] LTP and POSIX test suite report
<[email protected]> Wed, 30 Jun 2021 22:32:35 +0000
| Newsgroups | dev.linux.lists.fuego |
|---|---|
| Message-ID | <BYAPR13MB2503DEAAF54455F941A04FA5FD019@BYAPR13MB2503.namprd13.prod.outlook.com> |
> -----Original Message----- > From: Balaji Kalidas <[email protected]> > > Hi Tim, > We pulled the latest LTP and ran on rpi.here are some observations. > > 1.In current scenario Functional.Ltp includes 2 main test cases one is syscalls and SEM on spec.json file.but it checks for 3 different > test cases(syscalls,pty and timers) in criteria.json file for threshold,but it is not included in spec.json file(pty and timers testcase). Hmmm. I'm not seeing this behavior. The default criteria.json file has these entries, which are related to individual sub-tests which might be executed by Fuego's Functional.LTP test: { "tguid": "pty", "must_pass_list": ["hangup01"] } This says, that if the 'pty' sub-test is run, then the individual testcase hangup01 must pass. However, it should not affect the results if the 'pty' sub-test is not present. If you are not running the pty test, this must_pass_list should not be processed. If it *is* running in your case, then that would be a significant bug in Fuego that we need to fix. This entry: { "tguid": "timers", "fail_ok_list": ["leapsec_timer"] }, Should not have caused any problems. This entry also should not have been processed if the "timers" sub-test was not run. Even so, since this is a "fail_ok_list", it should not have mattered that the "timers" sub-test was not run. At most, this construct should convert a FAIL to a PASS, it should never turn absence of a result into a FAIL. > > 2.since pty and timers test cases are not included in spec.json, those 2 test cases won't run. so syscalls and SEM passes also LTP > gives an error. I have run LTP using the "smoketest" spec from spec.json, with the default criteria.json file, and had the test pass. The smoketest spec does not run the timers or pty sub-tests, so if those were the problem in the criteria file, those criteria entries would presumably also cause that spec to fail. How did you determine that those entries (for 'timers' and 'pty') were causing the problem? You can see debugging information about how the criteria is processed by turning on debugging for that phase of the test. That is, you can set: criteria:debug in your FUEGO_LOGLEVELS, and get a large (actually horrifyingly huge) amount of criteria processing data in the console log for the test run. (Look for entries starting with "DEBUG(criteria)"). > There is a sample spec in LTP's spec.json that runs timers and pty. It is the 'somefail' spec, which has this definition: "somefail": { "tests": "timers pty", "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}, "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"} }, In general, the overall idea of criteria files is to specify exceptions for the "all individual testcases must pass" rule that is normally in effect for evaluating the overall result of a test. Having said this, I think that the combination of test specs (which selects different parameters to control test output and the set of individual testcases to run) and criteria files (which specifies the results analysis) is not as coordinated as it should be. Possibly, the criteria file should be specified as relating to a particular spec. That is, we would have a criteria for the 'default' spec, and this would be different from the criteria for the 'somefail' spec or the "selection" spec. > > 3.To make LTP success we have two option in front of us: > > Option 1: Add pty and timer test case to spec.json It's already there in the "somefail" spec. > Option 2: Remove pty and timers threshold checking in criteria.json file > > > We took the second option and removed those test cases(pty and timers) from the criteria.json file.and also changed the max_fail > threshold checking of the syscalls test cases on criteria.json file. Option 2 is the better choice, if their presence is the actual bug. I'm not sure how you customized the criteria file for the board. Please let me know. Did you overwrite the on in fuego-core/tests/Functional.LTP, or did you put one a board-specific one in the /fuego-rw/boards directory? The preferred method is to use a board-specific one, rather than change the global one. See the instructions in fuego/docs/rst_src/criterial.json.rst, section "Using a board-specific directory". > > After we ran the LTP test suite it ran successfully. OK - if the only change you made to the critiera file was to remove the timers and pty entries, and this affected the overall LTP pass results (when the same set of testcases failed), then this would be a serious bug in LTP criteria handling. Can you please do a few things: - check that the failed testcases was the same for the runs that succeeded and failed - check that the criteria file was not modified in some other way (such as by changing the syscalls 'max_fail' or 'min_pass' amount. - turn on debugging for the criteria handling, and compare the console output for a test that fails with the original criteria file, and succeeds for your modified criteria file. The point of this would be to find the reason for the difference in the top-level result, which should be printed in the debug statements of the criteria processing code. One feature which I have wanted to add, is a "reason" message for the results, that could flow up the testcase/test set/ test suite tree, and indicate at the top level what the problem was that was causing the top-level failure. Another feature I have wanted to add is the ability to alter the criteria file using ftc at the command line. I got quite a ways along towards implementing that before I got sidetracked, and unfortunately it has been left unfinished. I think either of these features would have helped resolve this problem (or illuminated the bug, if there is one). Thanks for working on this. Let discuss the results and next steps in our call this week. Regards, -- Tim > note:here attached the error report,results and console log. > > regards, > balaji.k > > On Thu, Jun 17, 2021 at 5:21 PM Balaji Kalidas <[email protected] <mailto:[email protected]> > wrote: > > > > Hi Tim, > > > queries about LTP report: > > > > Why did some of your tests not have libaio.so.1 issues, when others did (and more of my tests did)? > --While running these tests for the first time I got these libaio errors. > > --We resolved these errors by additionally downloading sources for libaio and cross-compiling them. > > > In the meantime we will run the LTP and get back to you soon. > > regards, > balaji.k > > On Sat, Jun 12, 2021 at 4:26 AM <[email protected] <mailto:[email protected]> > wrote: > > > OK - here is some feedback on these results. > > Before I get to discussing the issues below, I have a few comments. > > First, I found a couple of bugs that needed to be fixed in the LTP test > (or in Fuego itself). I have pushed some of these to fuego-core, and > others are still outstanding to resolve. > > Here are some of issues that would be good to address: > - autoskip processing takes a long time. There are some routines (such as is_on_target) > which take much longer than they should (doing multiple round trips to the target > that could be eliminated). So this could be improved. > - if the kernel config can't be found, then the test errors out (this should be fixed in Fuego) > - the LTP test console is mostly silent during the test. This makes it really hard to figure > out which test is causing problems in the case of a kernel panic or machine hang. > I started work on this, but found some bugs and had to roll back my changes. > - I had to fix some things related to the update to 20210524 version of LTP, but I'm not > confident I caught everything that needs to be fixed. > - One things is that LTP has a new test runner that should work much better with Fuego, > but I will have to re-write sections of fuego_test.sh and ltp_target_run.sh. If it affects > the test output, I will also have to rewrite the parser.py and ltp_process.py scripts, which > would be a major effort. I'm still thinking whether this is worth it or not. > > OK, now for some response and feedback. > > > -----Original Message----- > > From: Pavan Arun Deshpande <[email protected] <mailto:[email protected]> > > > > > Hi Tim, > > > > We analysed the previous test error report and sorted out some TCONF errors but still some errors are coming. > > We ran the latest ltp(20210524) test on a raspberry pi board and we have prepared the test error reports for both tests > based on the output > > we got . > > > > > > we have some observations: > > > > * tests which failed as TCONF test error , shows status as "pass" in run.json log file. > > > > for an example : fork14 which gives the error as - mmap() fails too many times,so we are almost impossibleto > get an > > vm_area_structsized 16TB. > > but in run.json shows status as pass . like below > > > > { > > "name": "fork14", > > "status": "PASS" > > } > > query is on what bases tests are considered as pass or fail in the ltp? > > According to the code in Functional.LTP/parser.py (line 128) TCONF is interpreted > as a "PASS". However, I think it would be better if it were interpreted as a SKIP. > > The way Fuego processes LTP results, it doesn't count any tests that were > auto-skipped. These are processed independently of the set of specified > runtest files, which is a bit confusing, IMHO, for users. It might be worth > re-thinking how skips are reported, and the mappings from LTP result codes > to Fuego result codes. (LTP has more than Fuego does, so the mapping > will lose some information somewhere.) > > In the case of fork14, I'm not sure what the test is saying about the machine > configuration. It looks like a runtime autodetection that the test cannot run > based on dynamic evaluation of fork mmap failures. This seems kind of > sketchy to me, but I'm OK with LTP interpreting that as TCONF. > > > > > * Most of the TCONF failed tests are due to absence of required configurations in the kernel config file.Tests which > failed as TCONF > > on raspberry pi runs successfully on x86 machine because required configurations are enabled in the config file. > > > > for example: fanotify test was failed due to CONFIG_FANOTIFY_ACCESS_PERMISSIONS not configured in the kernel > but same test > > passed on the x86 machine > > because CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y enabled in the x86 config file. > > > > So can we add all those required configs to the raspberry pi config file? > > That's something that would be a lab-specific policy. If you want less TCONFs, then you can do that. > In general, I don't worry about TCONFs. It depends on one's testing strategy and situation. > > In general, the autoskip code catches a bunch of these, and makes it so that tests that not relevant > to your processor, distro, or configuration are automatically avoided. This cuts down no noise during > testing. However, it does possibly create confusion because the set of tests that is run changes > based on kernel config or other factors (like whether you have a specific program installed (like 'at')) > > With product testing (ie, the kind I do for Sony), I do NOT like to alter the configuration of the kernel. > And in some cases, I can't change the kernel config. I want to run the tests on the same kernel (config) > that the end user will use on that product hardware. > > However, for testing for upstream developers, they may want to enable as much of the kernel > config as possible, so they can see all possible diagnostics, and avoid missing tests due to configuration. > > > > > I have attached a test error report and also attached result output documents(results.xlsx,run.json,consolelog.txt) > generated during test > > execution. > > Let me know your thoughts on this. > > Did you generate the PDF error report manually? It's a nice summary report. > > It would be nice if 'ftc put-run' worked well with LTP. If that were working well, you could just do > a put-run for any run that you wanted to share, and just provide a link (or a reference build number), > and I could get the data from fuegotest.org <http://fuegotest.org> . > > However, right now 'ftc put-run' misses several key pieces of data from LTP that would be useful for analysis. > It would be very nice to include the kernel config file (by putting in the log directory for a run). > That might be a nice thing to add to the machine snapshot function for this test. I'll think about that. > Also I think it would be good to include the 'results' directory in the run package. It's easier to manually > examine issues from the command line with the broken out result files (ie the result file per test program, > found under results/syscalls/outputs). > > I would like to fix the 'ftc package-run' code to capture these other files into the run package for an > LTP test. That's something to go on the to-do list. > > I found that I got different results for some of the tests that I ran. > > I summarized the differences in a table on this page: > http://fuegotest.org/wiki/LTP-results-analysis > > It would be nice to figure out why I got different results in some cases. > So far, I only analyzed failures, not TCONF, TBROK, or TWARN differences. > > Two things that stand out immediately are: > 1) why did you have problems with chown and setre* test, when I did not > 2) why did some of your tests not have libaio.so.1 issues, when others did (and more of my tests did)? > > If we could fix these two categories of errors, I think we would be very close to having > the same results. We could then dig into whether these were real bugs in the kernel or > were maybe problems with the LTP test, or possibly even problems with the RPI hardware. > In any case, we would want to report results to the appropriate upstream project. > > I think some good goals for this LTP would be: > - be able to replicate each other's results, or find explanations for any differences > - fix some of the obvious issues with Fuego's handling of LTP execution and results > > Let me know what you think. > > In the mean time, can you perform another test, and see if there are any differences in > results between the original run on June 9th, and the subsequent run. Sometimes > tests are flakey, and give different results depending on unobvious factors. > > Thanks. This is very helpful. > -- Tim > > > > On Thu, Jun 3, 2021 at 5:31 PM Balaji Kalidas <[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]> > > wrote: > > > > > > Hi Tim, > > > > We ran the latest ltp version(20210524) Functional LTP and POSIX tests on a raspberry pi board and we have > prepared the test > > error reports for both tests based on the output we got . > > > > > > I have attached both(LTP and POSIX) test error reports and also attached result output documents generated during > test execution. > > > > Let me know your thoughts on the same. > > > > > > Thanks and regards > > BALAJI.K > > > > > > On Thu, Jun 3, 2021 at 5:22 AM <[email protected] <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]> > > wrote: > > > > > > OK - I did some more analysis and testing today. I found a few different > > bugs in the Functional.LTP test. It works OK on some other boards in > > my lab, but on my raspberry pi it had problems. Specifically msgstress04 > > is taking quite a long time (but it does eventually complete). > > But the overall test hangs before completing. > > > > I updated Functional.LTP to the latest version (20210524). This has > > been updated in the fuego-core master git repository. You should > > be able to do a git pull of the new test. Can you please > > try this test again, and let me know what results you get? > > > > Thanks, > > -- Tim > > > > > > > -----Original Message----- > > > From: Pavan Arun Deshpande <[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]> > > > > > > > > Hi Tim, > > > > > > fuego has LTP test version - 20180118. > > > but the latest LTP repo has version - 20210524. > > > > > > > > > > > > We ran the LTP and POSIX tests on a raspberry pi board and we have prepared the test error reports for both > tests based > > on the output we > > > got . Both the tests have the common errors(test failures). > > > > > > > > > I have attached both(LTP and POSIX) test error reports and also attached result output documents generated > during test > > execution. > > > > > > Let me know your thoughts on the same. > > > > > > > > > Thanks and regards > > > Pavan Arun Deshpande > > > > > > > > > > > > This message contains confidential information and is intended only for the individual(s) named. If you are not > the > > intended recipient, you > > > are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and > attached > > file/s is strictly > > > prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission > cannot be > > guaranteed to be > > > secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, > or > > contain viruses. The > > > sender therefore does not accept liability for any errors or omissions in the contents of this message, which > arise as a > > result of e-mail > > > transmission. > > > > > > > > > > This message contains confidential information and is intended only for the individual(s) named. If you are not the > intended recipient, you > > are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached > file/s is strictly > > prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be > guaranteed to be > > secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or > contain viruses. The > > sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a > result of e-mail > > transmission. > > > > > This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient, you > are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly > prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be > secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The > sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail > transmission.