Re: Meeting proposal for KernelCI Bisection

On Thu, 12 Mar 2026 at 00:24, Mark Brown <[email protected]> wrote:
>
> On Wed, Mar 11, 2026 at 10:01:08PM +0100, Guillaume Tucker wrote:
>
> > Stage 1: the basics
> > -------------------
>
> > The first step is to implement the same kind of bisection as was
> > initially done in Jenkins but this time with Maestro:
>
> > * only for LAVA test regressions
> > * triggered once all the results for a particular revision are in
> > * removing duplicates with simple criteria
> > * optimising builder usage while waiting for runtime test results
>
> > The pipeline logic boils down to this:
>
> > 1. look for base 'good' commit with reference branch (i.e. mainline)
> > 2. check bad / good commits
> > 3. iterate with git bisect
> > 4. check found commit and revert in place to check it fixes the issue
> > 5. send email report in reply to the original patch thread
> >    (or alternatively build a list of recipients from the commit)
>
> > It would be wise to send reports to a fixed audience initially for
> > moderation until it's deemed good enough for automated replies.
>
> I think there's a bunch of stuff we can do a lot better here than the v1
> stuff did - the big one is sharing the builds and test runs between
> multiple bisections.  Especially with the builds it's relatively common
> to be able to share at least the first step or two of a set of
> bisections (eg, if two tests fail on the same build we can reuse the
> build), and it's not uncommon for multiple tests in the same test job
> fail at once with the same or closely related causes.  You can get
> something a whole lot more responsive that's still very simple in
> implementation.

Agreed, build sharing is key. We're currently integrating TuxMake into
KCI which gives us reproducible builds keyed by (commit, arch, defconfig,
toolchain) =E2=80=94 that gives us a natural cache key for sharing builds a=
cross
bisections and reusing builds that already exist from normal CI runs.

Also we should propose n-bisect: rather than testing one commit per round,
we pick N evenly-spaced commits from the range and build/test them all
in parallel. That combined with a build cache means many of those builds
may already exist and we skip straight to testing. For a range of 1000
commits, n=3D7 gets us down to ~4 rounds instead of 10 =E2=80=94 and any bu=
ild
that's already cached is essentially free.

> Doing builds and tests as Maestro jobs probably gets us a lot of the way
> there, what Arisu has implemented already in kci-dev but rearranged a
> bit with more of a server focus (eg, driving multiple bisects in
> parallel).

Yes, Arisu's kci-dev bisect already drives the full loop via Maestro's
/api/checkout endpoint with job and platform filtering. The main gap is
that it's client-side =E2=80=94 one bisection at a time on the developer's
machine. Moving the orchestration server-side so we can drive multiple
bisections in parallel and share builds between them is the natural
next step. We'll discuss whether that's a new pipeline service or an
extension of what's already there.

> > Stage 2: expanded coverage
> > --------------------------
>
> > * add primitive support for metrics-based results e.g. thresholds
>
> BTW throwing https://otava.apache.org/ out there in case people haven't
> seen it.  More relevantly there's also the fastpath bisection stuff.

Thanks for the pointer =E2=80=94 Otava looks useful for Stage 2 when we get=
 to
performance regression detection. It does a statistical change-point
detection on performance data from various backends which is exactly
the kind of thing we'd need for metrics-based bisection.

> > Stage 3: advanced features
> > --------------------------
>
> > Using 'git bisect' has its own limitations.  It's great during
> > development but automated systems have different use cases.  A more
>
> There's also stuff like mining history and throwing test results that we
> already have in the range at the bisect (eg, if we've tested commits on
> a different branch or as part of another bisect), not particularly smart
> in terms of the bisect algorithms or anything but making use of the
> data set that KernelCI is building up to try to improve their performance=
.

This is a good one. KCIDB already has results across trees and branches
=E2=80=94 querying it before each bisection step to see if we've already te=
sted
a commit could save us entire rounds. Even for n-bisect, if half the
points are already known good/bad from prior runs, we narrow the range
before we even start building.

> Another idea that's getting more into trying to be actually clever would
> be looking at the builds we've already got and testing those if they're
> somehow close enough to what we might have otherwise chosen rather than
> spending time doing another build.

Agreed =E2=80=94 with n-bisect we have some flexibility here. If we pick 7
evenly-spaced commits but one of them is 2 commits away from a build
we already have, we could shift the test point to use the existing
build. The range narrowing still works as long as we know which segment
each result falls in.

> > their tree (e.g. -rc2 tag).  Semi-automated bisections is also
> > something to explore, with a command line that 'git bisect' can run
> > on each iteration and the ability to trigger fully automated ones
> > manually.
>
> There's the bisect stuff Arisu did in kci-dev that I mentioned for the
> first bit.

Right, kci-dev bisect already supports this =E2=80=94 developers can run it
manually with --good/--bad and it drives the full loop. We also have
TuxMake + TuxRun for fully local bisection (git bisect run with
reproducible builds and QEMU testing). So semi-automated is essentially
working today; the meeting on the 19th is focused on getting from there
to a fully automated server-side bisection triggered by regression
detection.

> > So this is a bit of a brainstom or braindump exercise without
> > actually diving into anything in much detail.  It'll be interesting
> > to see how this combines with others' experience of automated
> > bisections in various environments.  Hope this helps!
>
> Yeah, likewise.

Great discussion, looking forward to continuing this next Thursday.

Ben