Re: Talk proposal: What 125K kernel bugs tell us about testing gaps

(resending in plain text instead of html)

Hi Jenny,


On Thu, Feb 5, 2026 at 4:44=E2=80=AFAM Jenny Qu <[email protected]> wrote=
:
>
> [resending to list - accidentally replied off-list]
>
> On Wed, Feb 04, 2026 at 11:00:00PM, Greg KH wrote:
> > I hate to say "your ai model could be replaced with a sql statement"
>
> Fair point on the descriptive statistics. I should have been clearer:
> the 125K bug analysis was training data, not the contribution. verhaal
> and the LWN employer reports (Jonathan Corbet's per-release stats
> using the gitdm database) already cover the descriptive side well.
>
> The part SQL can't do is the predictive model. VulnBERT takes a raw
> git diff *before merge* and predicts whether it introduces a
> vulnerability. The evaluation is a strict temporal holdout: trained
> on commits with Fixes: tags from <=3D2023, tested on 2024 commits that
> later received Fixes: tags. 92% recall, 1.2% FPR on that split.
>
> To be direct about limitations: those numbers are on historical data
> where we know ground truth. The model catches patterns it's seen
> before (unbalanced refcounts, missing NULL checks, lock/unlock
> mismatches). It will miss novel bug classes it hasn't been trained on.
> It's a triage tool and not yet an oracle.
>
> And it's not ready for production use yet. I'm reworking the
> architecture. The current approach uses CodeBERT embeddings with
> handcrafted features, and I think incorporating LLM reasoning traces
> over diffs will do substantially better. I don't want to hand anyone
> a tool that generates false confidence.
>
> On employer attribution: you're right, email domain mapping
> undercounts significantly. Developers using personal emails,
> acquisitions (Mellanox -> NVIDIA), and consultants all break the
> heuristic.
>
> > how do you feel this information can help with our project? What
> > would you like us to do based on what you have found here?
>
> Honestly, I'd rather hear from the KernelCI community what would
> actually be useful than prescribe solutions. But two directions I
> think are worth discussing:
>
> 1. Subsystem-level test prioritization. The lifetime gap between
>    CAN bus (4.2 years) and gpu/i915 (1.4 years) almost certainly
>    reflects testing coverage differences. i915 has dedicated
>    fuzzing infrastructure and active reviewers like Chris Wilson
>    and Ville Syrjala. KernelCI could use lifetime data as a signal
>    for where to invest in test enablement. This is actionable now,
>    no ML required.
>
> 2. Longer-term: commit-level risk scoring to allocate CI resources.
>    Flag high-risk commits for extra sanitizer runs, longer fuzzing
>    passes. Low-risk commits get the standard pipeline. But this
>    needs a model I trust enough to deploy, and I'm not there yet.
>
> I'm speaking at BugBash 2026 in April and looking at LPC for a more
> technical deep-dive.

Thanks for this.  As a board member of KernelCI, most of the efforts
we have funded or try to support are ones that have been adopted by
the community.  The work we try to sponsor needs to provide value to
the community but the kernel community can be tricky to navigate as
you can see by Greg's comments.

I would recommend those conferences but also try attaching your work
as replies to various patches.  Try to show off the value of your work
on mailing lists and let that start conversations on how to steer it
towards something that could be considered useful.  That journey will
lead to overlap of existing technologies that Greg mentioned, but more
importantly it will lead to conversations on how to collaborate around
those technologies to make something valuable to the community.  The
end result being that it becomes a no-brainer to add to kernelci.

A current example we are working with is Thorsten's regzbot[0].  A
difficult social problem around regression tracking that the community
helped him navigate towards something of value and now makes sense for
kernelci to sponsor.

Cheers,
Don

[0] - https://linux-regtracking.leemhuis.info/about/


>
> kindly,
> Jenny
> [email protected]
>
>
> On Wed, Feb 4, 2026 at 11:00=E2=80=AFPM Greg KH <[email protected]=
rg> wrote:
> >
> > On Wed, Feb 04, 2026 at 06:49:57PM -0800, Jenny Qu wrote:
> > > Hi,
> > >
> > > I'm a security researcher working on automated kernel vulnerability
> > > detection. I'd love to present at an upcoming Thursday call if there'=
s
> > > interest.
> >
> > Cool, but isn't this a better subject for a conference talk?
> >
> > > I analyzed every Fixes: tag in the kernel's 20-year git history (125K
> > > bug-fix pairs) and built a model to catch vulnerabilities at commit
> > > time. Some findings that might be relevant to KernelCI's testing
> > > strategy:
> > >
> > > - Security bugs hide for 2.1 years on average; race conditions persis=
t 5.0 years
> > > - 117 "super-reviewers" (including Dan Carpenter, who invented the
> > > Fixes: tag) catch bugs 47% faster
> > > - Subsystems like CAN bus (4.2 years) and SCTP (4.0 years) have
> > > dramatically longer bug lifetimes than gpu/i915 (1.4 years)
> > > - Weekend commits are 8% less likely to introduce bugs, but take 45%
> > > longer to fix (review coverage effect)
> > >
> > > The model (VulnBERT) achieves 92% recall at 1.2% false positive rate
> > > on held-out 2024 data. I'm also working on SmartKuang, an RL-based
> > > system that has reproduced CVE-2022-34918 autonomously.
> >
> > I hate to say "your ai model could be replaced with a sql statement",
> > but really, we do have tools that show this today that give all of this
> > data in a sqlite database that people can use to mine for the same info=
.
> > It's what the kernel CVE team uses to track bug fixes over time for
> > their work:
> >         https://git.sr.ht/~gregkh/verhaal
> > and is part of the vulns.git repo on git.kernel.org
> >
> > Also for the tracking of employer to people and who is doing the work,
> > see the reports on lwn.net for the past few decades that have been
> > documenting this.  The tool for that is also public (but part of the
> > database of employer mapping is not for obvious reasons, sorry).  I
> > think you undercounted people's employers a lot as you can not always
> > rely on email addresses to convey this.
> >
> > Anyway, I liked your reports as I'm always interested in more people
> > mining our public data for stuff like this, it's great to see.  But wit=
h
> > regards to kernelci, how do you feel this information can help with our
> > project?  What would you like us to do based on what you have found
> > here?
> >
> > thanks,
> >
> > greg k-h
>