Re: Reduced build and test coverage

On 19/09/2023 18:43, Guillaume Charles Tucker wrote:
> On Tuesday, September 19, 2023 18:12 CEST, Mark Brown <[email protected]> wrote:
> 
>> On Tue, Sep 19, 2023 at 04:44:06PM +0200, Ricardo Cañuelo wrote:
>>> On mar, sep 19 2023 at 15:31:31, Mark Brown <[email protected]> wrote:
>>>> If we're doing this -next would be very helpful too.
>>
>>> Does it make sense to bisect -next, though? Considering that it's
>>> constantly rebased, so I don't think it has a coherent and "linear" log
>>> like mainline. That means that bisections aren't guaranteed to work on
>>> it. Is that right?
>>
>> When bisecting -next you should generally bisect it against the mainline
>> it was based on.
> 
> Yes that's how the KernelCI bisection works. It's the most useful branch to bisect automatically precisely because it's rebased every day with thousands of commits, so a bisection takes between 10 and 15 iterations and it's painful to do manually. And that's where most of the bugs are.  Then mainline, stable and stable-rc are the other critical trees to cover so it makes sense to enable them first.
> 
> We've discussed enabling bisections again with sysadmins and I think we can have this done in a week or two.  Other issues are about measuring costs empirically for different parts of the system and adding a filter in the legacy back end to avoid flooding Jenkins with jobs that won't get run (they still use up all the RAM).

We've now re-enabled the full functional test coverage which can
result in roughly twice the amount of test results coming from
LAVA labs.  We'll be monitoring the additional costs incurred as
a result this week and compare with previous weeks.

Then the plan is to prepare everything to have bisections enabled
again next Monday but with only mainline, linux-next, stable-rc
and stable as discussed previously.  These are the trees that
benefit the most from automated bisection.  We'll be able to
measure the added costs for that again in the following weeks
although we already have an estimate.

In October, the current plan is to enable some more builds such
as allmodconfig on a couple more trees such as mainline (right
now only linux-next has allmodconfig builds enabled) and continue
measuring the related costs.

This exercise aims at:

* knowing more accurately the cost of each area KernelCI's
  coverage to better secure its long-term funding

* streamlining the overall config to maximise the value of the
  results produced by KernelCI

* review the whole configuration in preparation for migrating to
  the new API with a fresh implementation

As mentioned before, if the current coverage is missing parts
that are critical to any use-case in the community then please
raise the issue here and we can address it.  For example, we're
looking into Android builds and bisections on more trees based on
feedback from users.

Thanks,
Guillaume