Re: [PATCH 1/3] MAINTAINERS: Introduce V: field for required tests

On Tue, Nov 21, 2023 at 01:27:44PM +0000, Mark Brown wrote:
> > (I don't need to see all of the tests that passes; it's the test
> > failures or the test flakes that are significant.)
> 
> The listing of tests does get a bit more complex when you mix in running
> on different platforms.

Yeah, that's part of the aggregation reports problem.  Given a
particular test, say, generic/475, I'd love to see a summary of which
file system config (e.g., ext4/4k, ext4/1k) and which architectures a
particular test is failing or which is flaky.  Right now, I do this
manually using a combination of a mutt mail reader (the test summaries
are e-mailed to me), and emacs....

> I think if we get tooling in place so that people can just run a script,
> add a flag to their tools or whatever to ingest results from the
> standard testsuites the barrier to reporting becomes sufficiently low
> that it's more of a "why not?" type thing.

Sure, I'm happy to add something like that to my test runners:
kvm-xfstests, gce-xfstests, and android-xfstests.  Then anyone who
uses my test runner infrastructure would get uploading for free.  We
might need to debate whether I enable uploading as something which is
enabled by default or not (I can imagine some people won't wanting to
upload information to a public site, lest it leak information about an
upcoming mobile handset :-), but that's a minor point.

Personally, I'm not going to have time to look into this for a while,
but... patches welcome.  Or even something that takes a junit xml
file, and uploads it to the kcidb.  If someone can make something like
that available, I should be able to take it the rest of the way.

Cheers,

						- Ted