Re: [Fuego] LTP and POSIX test suite report

Hi Tim,

queries about LTP report:

Why did some of your tests not have libaio.so.1 issues, when others did
(and more of my tests did)?
     --While running these tests for the first time I got these libaio
errors.

     --We resolved these errors by additionally downloading sources for
libaio and cross-compiling them.

In the meantime we will run the LTP and get back to you soon.

regards,
balaji.k

On Sat, Jun 12, 2021 at 4:26 AM <[email protected]> wrote:

> OK - here is some feedback on these results.
>
> Before I get to discussing the issues below, I have a few comments.
>
> First, I found a couple of bugs that needed to be fixed in the LTP test
> (or in Fuego itself).  I have pushed some of these to fuego-core, and
> others are still outstanding to resolve.
>
> Here are some of issues that would be good to address:
>  - autoskip processing takes a long time.  There are some routines (such
> as is_on_target)
>     which take much longer than they should (doing multiple round trips to
> the target
>     that could be eliminated).  So this could be improved.
>  - if the kernel config can't be found, then the test errors out (this
> should be fixed in Fuego)
>  - the LTP test console is mostly silent during the test.  This makes it
> really hard to figure
>     out which test is causing problems in the case of a kernel panic or
> machine hang.
>     I started work on this, but found some bugs and had to roll back my
> changes.
>  - I had to fix some things related to the update to 20210524 version of
> LTP, but I'm not
>    confident I caught everything that needs to be fixed.
>     - One things is that LTP has a new test runner that should work much
> better with Fuego,
>       but I will have to re-write sections of fuego_test.sh and
> ltp_target_run.sh.  If it affects
>       the test output, I will also have to rewrite the parser.py and
> ltp_process.py scripts, which
>       would be a major effort.  I'm still thinking whether this is worth
> it or not.
>
> OK, now for some response and feedback.
>
> > -----Original Message-----
> > From: Pavan Arun Deshpande <[email protected]>
> >
> > Hi Tim,
> >
> > We analysed the previous test error report and sorted out some TCONF
> errors but still some errors are coming.
> > We ran the latest ltp(20210524) test  on a raspberry pi board and we
> have prepared the test error reports for both tests based on the output
> > we got .
> >
> >
> > we have some observations:
> >
> > *      tests which failed as TCONF test error , shows status as "pass"
> in run.json log file.
> >
> >                for an example :  fork14 which gives the  error as -
> mmap() fails too many times,so we are almost impossibleto get an
> > vm_area_structsized 16TB.
> >                     but in run.json shows status as pass . like  below
> >
> >                {
> >                     "name": "fork14",
> >                     "status": "PASS"
> >                 }
> >           query is on what bases tests are considered as pass or fail in
> the ltp?
>
> According to the code in Functional.LTP/parser.py (line 128) TCONF is
> interpreted
> as a "PASS".  However, I think it would be better if it were interpreted
> as a SKIP.
>
> The way Fuego processes LTP results, it doesn't count any tests that were
> auto-skipped.  These are processed independently of the set of specified
> runtest files, which is a bit confusing, IMHO, for users.  It might be
> worth
> re-thinking how skips are reported, and the mappings from LTP result codes
> to Fuego result codes.  (LTP has more than Fuego does, so the mapping
> will lose some information somewhere.)
>
> In the case of fork14, I'm not sure what the test is saying about the
> machine
> configuration.  It looks like a runtime autodetection that the test cannot
> run
> based on dynamic evaluation of fork mmap failures.  This seems kind of
> sketchy to me, but I'm OK with LTP interpreting that as TCONF.
>
> >
> > *     Most of the TCONF failed tests are due to absence of required
> configurations in the kernel config file.Tests which failed as TCONF
> > on raspberry pi runs successfully on x86 machine because required
> configurations are enabled in the config file.
> >
> >            for example: fanotify test was failed due to
> CONFIG_FANOTIFY_ACCESS_PERMISSIONS not configured in the kernel but same
> test
> > passed on the x86 machine
> >            because   CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y  enabled in
> the x86 config file.
> >
> >             So can we add all those required configs to the raspberry pi
> config file?
>
> That's something that would be a lab-specific policy.  If you want less
> TCONFs, then you can do that.
> In general, I don't worry about TCONFs.  It depends on one's testing
> strategy and situation.
>
> In general, the autoskip code catches a bunch of these, and makes it so
> that tests that not relevant
> to your processor, distro, or configuration are automatically avoided.
> This cuts down no noise during
> testing.  However, it does possibly create confusion because the set of
> tests that is run changes
> based on kernel config or other factors (like whether you have a specific
> program installed (like 'at'))
>
> With product testing (ie, the kind I do for Sony), I do NOT like to alter
> the configuration of the kernel.
> And in some cases, I can't change the kernel config.  I want to run the
> tests on the same kernel (config)
> that the end user will use on that product hardware.
>
> However, for testing for upstream developers, they may want to enable as
> much of the kernel
> config as possible, so they can see all possible diagnostics, and avoid
> missing tests due to configuration.
>
> >
> > I have attached a test error report and also attached result output
> documents(results.xlsx,run.json,consolelog.txt) generated during test
> > execution.
> > Let me know your thoughts on this.
>
> Did you generate the PDF error report manually?  It's a nice summary
> report.
>
> It would be nice if 'ftc put-run' worked well with LTP.  If that were
> working well, you could just do
> a put-run for any run that you wanted to share, and just provide a link
> (or a reference build number),
> and I could get the data from fuegotest.org.
>
> However, right now 'ftc put-run' misses several key pieces of data from
> LTP that would be useful for analysis.
> It would be very nice to include the kernel config file (by putting in the
> log directory for a run).
> That might be a nice thing to add to the machine snapshot function for
> this test.  I'll think about that.
> Also I think it would be good to include the 'results' directory in the
> run package.  It's easier to manually
> examine issues from the command line with the broken out result files (ie
> the result file per test program,
> found under results/syscalls/outputs).
>
> I would like to fix the 'ftc package-run' code to capture these other
> files into the run package for an
> LTP test.  That's something to go on the to-do list.
>
> I found that I got different results for some of the tests that I ran.
>
> I summarized the differences in a table on this page:
> http://fuegotest.org/wiki/LTP-results-analysis
>
> It would be nice to figure out why I got different results in some cases.
> So far, I only analyzed failures, not TCONF, TBROK, or TWARN differences.
>
> Two things that stand out immediately are:
>  1) why did you have problems with chown and setre* test, when I did not
>  2) why did some of your tests not have libaio.so.1 issues, when others
> did (and more of my tests did)?
>
> If we could fix these two categories of errors, I think we would be very
> close to having
> the same results.  We could then dig into whether these were real bugs in
> the kernel or
> were maybe problems with the LTP test, or possibly even problems with the
> RPI hardware.
> In any case, we would want to report results to the appropriate upstream
> project.
>
> I think some good goals for this LTP would be:
>  - be able to replicate each other's results, or find explanations for any
> differences
>  - fix some of the obvious issues with Fuego's handling of LTP execution
> and results
>
> Let me know what you think.
>
> In the mean time, can you perform another test, and see if there are any
> differences in
> results between the original run on June 9th, and the subsequent run.
> Sometimes
> tests are flakey, and give different results depending on unobvious
> factors.
>
> Thanks.  This is very helpful.
>  -- Tim
>
>
> > On Thu, Jun 3, 2021 at 5:31 PM Balaji Kalidas <
> [email protected] <mailto:[email protected]> >
> wrote:
> >
> >
> >       Hi Tim,
> >
> >             We ran the latest ltp version(20210524) Functional LTP and
> POSIX tests on a raspberry pi board and we have prepared the test
> > error reports for both tests based on the output we got .
> >
> >
> >       I have attached both(LTP and POSIX) test error reports and also
> attached result output  documents generated during test execution.
> >
> >       Let me know your thoughts on the same.
> >
> >
> >       Thanks and regards
> >       BALAJI.K
> >
> >
> >       On Thu, Jun 3, 2021 at 5:22 AM <[email protected] <mailto:
> [email protected]> > wrote:
> >
> >
> >               OK - I did some more analysis and testing today.  I found
> a few different
> >               bugs in the Functional.LTP test.  It works OK on some
> other boards in
> >               my lab, but on my raspberry pi it had problems.
> Specifically msgstress04
> >               is taking quite a long time (but it does eventually
> complete).
> >               But the overall test hangs before completing.
> >
> >               I updated Functional.LTP to the latest version
> (20210524).  This has
> >               been updated in the fuego-core master git repository. You
> should
> >               be able to do a git pull of the new test. Can you please
> >               try this test again, and let me know what results you get?
> >
> >               Thanks,
> >                -- Tim
> >
> >
> >               > -----Original Message-----
> >               > From: Pavan Arun Deshpande <[email protected]
> <mailto:[email protected]> >
> >               >
> >               > Hi Tim,
> >               >
> >               > fuego has LTP test version - 20180118.
> >               > but the latest LTP repo has version - 20210524.
> >               >
> >               >
> >               >
> >               > We ran the LTP and POSIX tests on a raspberry pi board
> and we have prepared the test error reports for both tests based
> > on the output we
> >               > got . Both the tests have the common errors(test
> failures).
> >               >
> >               >
> >               > I have attached both(LTP and POSIX) test error reports
> and also attached result output  documents generated during test
> > execution.
> >               >
> >               > Let me know your thoughts on the same.
> >               >
> >               >
> >               > Thanks and regards
> >               > Pavan Arun Deshpande
> >               >
> >               >
> >               >
> >               > This message contains confidential information and is
> intended only for the individual(s) named. If you are not the
> > intended recipient, you
> >               > are notified that disclosing, copying, distributing or
> taking any action in reliance on the contents of this mail and attached
> > file/s is strictly
> >               > prohibited. Please notify the sender immediately and
> delete this e-mail from your system. E-mail transmission cannot be
> > guaranteed to be
> >               > secured or error-free as information could be
> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> > contain viruses. The
> >               > sender therefore does not accept liability for any
> errors or omissions in the contents of this message, which arise as a
> > result of e-mail
> >               > transmission.
> >
> >
> >
> >
> > This message contains confidential information and is intended only for
> the individual(s) named. If you are not the intended recipient, you
> > are notified that disclosing, copying, distributing or taking any action
> in reliance on the contents of this mail and attached file/s is strictly
> > prohibited. Please notify the sender immediately and delete this e-mail
> from your system. E-mail transmission cannot be guaranteed to be
> > secured or error-free as information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or contain viruses. The
> > sender therefore does not accept liability for any errors or omissions
> in the contents of this message, which arise as a result of e-mail
> > transmission.
>
>

-- 






This
message contains confidential information and is intended only 
for the
individual(s) named. If you are not the intended
recipient, you are 
notified that disclosing, copying, distributing or taking any
action in 
reliance on the contents of this mail and attached file/s is strictly
prohibited. Please notify the
sender immediately and delete this e-mail 
from your system. E-mail transmission
cannot be guaranteed to be secured or 
error-free as information could be
intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain
viruses. The sender therefore does 
not accept liability for any errors or
omissions in the contents of this 
message, which arise as a result of e-mail
transmission.