Re: [PATCH 1/3] MAINTAINERS: Introduce V: field for required tests

On 11/15/23 22:14, Mark Brown wrote:
> On Wed, Nov 15, 2023 at 07:43:49PM +0200, Nikolai Kondrashov wrote:
> 
>> Introduce a new tag, 'Tested-with:', documented in the
>> Documentation/process/submitting-patches.rst file. The tag is expected
>> to reference the documented test suites, similarly to the 'V:' field,
>> and to certify that the submitter executed the test suite on the change,
>> and that it passed.
> 
> This doesn't feel like it fits so well with git based flows - generally
> the tags end up in git one way or another so there'll be a strong
> tendency for this to end up getting added for one version and then
> carried forward to the next version.  The way the tooling is at present
> it doesn't really feel like there's a good point at which to insert the
> tag.
> 
> I'm not sure exactly what'd be better though.

Yeah, I agree that's a bit of a problem. One that only automated 
tools/testing/CI could fully solve. Cough, git forges, cough.

OTOH, once you managed to run an automated suite once, it's much easier to do 
it again, and most of the time developers *want* their code to work and pass 
the tests (it's much easier than manual testing after all). So it's likely 
they will keep running them for new revisions, even though they might not 
notice they simply reused the previously-added Tested-with: tag.

Still, one way to make this better could be requiring a URL pointing at test 
results to follow the test suite name in the Tested-with: tag. Then the 
maintainer could check that they're indeed fresh.

This would be however getting way ahead of ourselves and, with the current 
(average) state of testing infra, hard to do. Perhaps sometime later.

For now, I think this could already go a long way towards having more (and 
better) testing.

Nick