Re: [linux-safety] [ELISA Development Process WG] [PATCH] mm: vmscan: provide a change to the development-process group

On Thu, 17 Sep 2020, Paoloni, Gabriele wrote:

> 
> 
> > -----Original Message-----
> > From: [email protected] <development-
> > [email protected]> On Behalf Of Lukas Bulwahn
> > Sent: Thursday, September 17, 2020 5:44 PM
> > To: Paoloni, Gabriele <[email protected]>
> > Cc: Lukas Bulwahn <[email protected]>; linux-
> > [email protected]; [email protected]
> > Subject: Re: [linux-safety] [ELISA Development Process WG] [PATCH] mm:
> > vmscan: provide a change to the development-process group
> > 
> > 
> > 
> > On Thu, 17 Sep 2020, Paoloni, Gabriele wrote:
> > 
> > > > -----Original Message-----
> > > > From: [email protected] <development-
> > > > [email protected]> On Behalf Of Lukas Bulwahn
> > > > Sent: Thursday, September 17, 2020 10:44 AM
> > > > To: [email protected]
> > > > Cc: [email protected]; Lukas Bulwahn
> > > > <[email protected]>
> > > > Subject: [ELISA Development Process WG] [PATCH] mm: vmscan: provide
> > a
> > > > change to the development-process group
> > > >
> > > > I think this change is needed for safety, whatever that might mean to
> > you.
> > > >
> > > > I am unqualified to really make a change here, as I have no clue what this
> > > > code does, nor what my change does, but sure, the testing and
> > verification
> > > > reference process can now point out the required next steps in the
> > > > reference process to test this code and code change.
> > > >
> > > > Good luck :)
> > > >
> > > > Not intended for distribution to the general kernel mailing lists.
> > > >
> > > > Signed-off-by: Lukas Bulwahn <[email protected]>
> > > > ---
> > > > I would like to submit such a patch, what do I need to do according to
> > > > the expected testing and verification recommendations for safety-
> > related
> > > > systems?
> > >
> > > Probably the problem is not just limited to submitting patches, but I think it
> > > is a good starting point.
> > >
> > > >
> > > > Please help me. What do I need to compile, what test do I need to run,
> > > > which verification tool do I need to employ for this change?
> > >
> > > Right. I have tried to reformulate the problem as "ok I am looking at or I am
> > > trying to change one or more code lines, so I would like to understand if
> > such
> > > code lines are tested today and how".
> > > Right now if we had a s maintained structural coverage report we could tell
> > > if the impacted code lines are covered and by which tests...I could not find
> > it
> > > so I guess that so far we are missing it....am I wrong?
> > >
> > > As next step I went to look at the function wrapping the impacted code line
> > > that in this case is kswapd. I could not find it tested in the Kunit framework.
> > >
> > > As next steps then I would run the available tests in
> > > https://elixir.bootlin.com/linux/latest/source/tools/testing/selftests
> > > in conjunction with GCOV trying to figure out if that line is covered already
> > > somehow.
> > >
> > > There are probably smarter, better, faster ways to elaborate on this...so
> > > here I just put down what I would do in order to figure this out...
> > >
> > 
> > Gab, these are GREAT ideas and approaches. So, do we intend to provide
> > methods, tools and documentation to really do these things you suggest?
> > 
> > That makes the intensions more tangible to me; if we try to find out
> > how we and anyone else can get that information in some 'easy' way.
> > 
> > You can imagine that for a single-line change, you are suggesting probably
> > quite some few days of work of investigation to get this information,
> > right?
> 
> Unless together with the test frameworks in Linux we start maintaining a
> corresponding structural coverage report that would instantly map the
> affected code lines against the corresponding test.
> 
> Then reading the corresponding test we can figure out if the tests are still
> valid (i.e. - black box - have I changed any behavior with my patch compared
> to the expected result of the tests? or - white box - am I able to reach my
> new change using the current test suites?)
> 
> My feeling is that we just need to get started putting together a more
> structured test framework and report and them the problem as presented
> by this patch would be very straightforward
> 

Show me and I will believe you, Gab :)

If we can pull this off, we could actually impress some people in the 
kernel community.

This sounds like a plan that could serve as reference for the next 
patch to come...

Lukas