Re: [RFC] Test catalog template

Hi David,

On Wed, Nov 20, 2024 at 3:16 AM David Gow <[email protected]> wrote:
>
> On Thu, 7 Nov 2024 at 01:01, Donald Zickus <[email protected]> wrote:
> >
> > Hi,
> >
> > Thanks for the feedback. I created a more realistic test.yaml file to
> > start (we can split it when more tests are added) and a parser.  I was
> > going to add patch support as input to mimic get_maintainers.pl
> > output, but that might take some time.  For now, you have to manually
> > select a subsystem.  I will try to find space on kernelci.org to grow
> > this work but you can find a git tree here[0].
> >
> > From the README.md
> > """
> > An attempt to map kernel subsystems to kernel tests that should be run
> > on patches or code by humans and CI systems.
> >
> > Examples:
> >
> > Find test info for a subsystem
> >
> > ./get_tests.py -s 'KUNIT TEST' --info
> >
> > Subsystem:    KUNIT TEST
> > Maintainer:
> >   David Gow <[email protected]>
> > Mailing List: None
> > Version:      None
> > Dependency:   ['python3-mypy']
> > Test:
> >   smoke:
> >     Url: None
> >     Working Directory: None
> >     Cmd: ./tools/testing/kunit/kunit.py
> >     Env: None
> >     Param: run --kunitconfig lib/kunit
> > Hardware:     arm64, x86_64
> >
> > Find copy-n-pastable tests for a subsystem
> >
> > ./get_tests.py -s 'KUNIT TEST'
> >
> > ./tools/testing/kunit/kunit.pyrun --kunitconfig lib/kunit
> > """
> >
> > Is this aligning with what people were expecting?
> >
>
>
> Awesome! I've been playing around a bit with this, and I think it's an
> excellent start.
>
> There are definitely some more features I'd want in an ideal world
> (e.g., configuration matrices, etc), but this works well enough.

Yeah, I was trying to nail down the usability angle first before
expanding with bells and whistles.  I would like to think the yaml
file is flexible enough to handle those features though??

>
> I've been playing around with a branch which adds the ability to
> actually run these tests, based on the 'run_checks.py' script we use
> for KUnit:
> https://github.com/sulix/test-catalog/tree/runtest-wip

Thanks!

>
> In particular, this adds a '-r' option which runs the tests for the
> subsystem in parallel. This largely matches what I was doing manually
> — for instance, the KUnit section in test.yaml now has three different
> tests, and running it gives me this result:
> ../test-catalog/get_tests.py -r -s 'KUNIT TEST'
> Waiting on 3 checks (kunit-tool-test, uml, x86_64)...
> kunit-tool-test: PASSED
> x86_64: PASSED
> uml: PASSED

Interesting.  Originally I was thinking this would be done serially.
I didn't think tests were safe enough to run in parallel.  I am
definitely open to this.  My python isn't the best, but I think your
PR looks reasonable.

>
> (Obviously, in the real world, I'd have more checks, including other
> architectures, checkpatch, etc, but this works as a proof-of-concept
> for me.)
>
> I think the most interesting questions will be:
> - How do we make this work with more complicated dependencies
> (containers, special hardware, etc)?

I was imagining a 'hw-requires' type line to handle the hardware
requests as that seemed natural for a lot of the driver work. Run a
quick check before running the test to see if the required hw is
present or not and bail if it isn't.  The containers piece is a little
trickier and ties into the test environment I think.  The script would
have to create an environment and inject the tests into the
environment and run them.  I would imagine some of this would have to
be static as the setup is complicated.  For example, a 'container'
label would execute custom code to setup a test environment inside a
container.  Open to ideas here.

> - How do we integrate it with CI systems — can we pull the subsystem
> name for a patch from MAINTAINERS and look it up here?

There are two thoughts.  First is yes.  As a developer you probably
want to run something like 'get_maintainers.sh <patch> | get_tests.py
-s -' or something to figure out what variety of tests you should run
before posting.  And a CI system could probably do something similar.

There is also another thought, you already know the subsystem you want
to test.  For example, a patch is usually written for a particular
subsystem that happens to touch code from other subsystems.  You
primarily want to run it against a specified subsystem.  I know Red
Hat's CKI will run against a known subsystem git-tree and would fall
into this category.  While it does leave a gap in other subsystem
testing, sometimes as a human you already know running those extra
tests is mostly a no-op because it doesn't really change anything.

> - What about things like checkpatch, or general defconfig build tests
> which aren't subsystem-specific?

My initial thought is that this is another category of testing.  A lot
of CI tests are workload testing and have predefined configs.  Whereas
a generic testing CI system (think 0-day) would focus on those types
of testing.  So I would lean away from those checks in this approach
or we could add a category 'general' too.  I do know checkpatch rules
vary from maintainer to maintainer.

> - How can we support more complicated configurations or groups of
> configurations?

Examples?

> - Do we add support for specific tools and/or parsing/combining output?

Examples?  I wasn't thinking of parsing test output, just providing
what to run as a good first step.  My initial thought was to help
nudge tests towards the KTAP output??

>
> But I'm content to keep playing around with this a bit more for now.

Thank you! Please do!

Cheers,
Don