Re: Talk proposal: What 125K kernel bugs tell us about testing gaps

[resending to list - accidentally replied off-list]

On Wed, Feb 04, 2026 at 11:00:00PM, Greg KH wrote:
> I hate to say "your ai model could be replaced with a sql statement"

Fair point on the descriptive statistics. I should have been clearer:
the 125K bug analysis was training data, not the contribution. verhaal
and the LWN employer reports (Jonathan Corbet's per-release stats
using the gitdm database) already cover the descriptive side well.

The part SQL can't do is the predictive model. VulnBERT takes a raw
git diff *before merge* and predicts whether it introduces a
vulnerability. The evaluation is a strict temporal holdout: trained
on commits with Fixes: tags from <=3D2023, tested on 2024 commits that
later received Fixes: tags. 92% recall, 1.2% FPR on that split.

To be direct about limitations: those numbers are on historical data
where we know ground truth. The model catches patterns it's seen
before (unbalanced refcounts, missing NULL checks, lock/unlock
mismatches). It will miss novel bug classes it hasn't been trained on.
It's a triage tool and not yet an oracle.

And it's not ready for production use yet. I'm reworking the
architecture. The current approach uses CodeBERT embeddings with
handcrafted features, and I think incorporating LLM reasoning traces
over diffs will do substantially better. I don't want to hand anyone
a tool that generates false confidence.

On employer attribution: you're right, email domain mapping
undercounts significantly. Developers using personal emails,
acquisitions (Mellanox -> NVIDIA), and consultants all break the
heuristic.

> how do you feel this information can help with our project? What
> would you like us to do based on what you have found here?

Honestly, I'd rather hear from the KernelCI community what would
actually be useful than prescribe solutions. But two directions I
think are worth discussing:

1. Subsystem-level test prioritization. The lifetime gap between
   CAN bus (4.2 years) and gpu/i915 (1.4 years) almost certainly
   reflects testing coverage differences. i915 has dedicated
   fuzzing infrastructure and active reviewers like Chris Wilson
   and Ville Syrjala. KernelCI could use lifetime data as a signal
   for where to invest in test enablement. This is actionable now,
   no ML required.

2. Longer-term: commit-level risk scoring to allocate CI resources.
   Flag high-risk commits for extra sanitizer runs, longer fuzzing
   passes. Low-risk commits get the standard pipeline. But this
   needs a model I trust enough to deploy, and I'm not there yet.

I'm speaking at BugBash 2026 in April and looking at LPC for a more
technical deep-dive.

kindly,
Jenny
[email protected]


On Wed, Feb 4, 2026 at 11:00=E2=80=AFPM Greg KH <[email protected]=
> wrote:
>
> On Wed, Feb 04, 2026 at 06:49:57PM -0800, Jenny Qu wrote:
> > Hi,
> >
> > I'm a security researcher working on automated kernel vulnerability
> > detection. I'd love to present at an upcoming Thursday call if there's
> > interest.
>
> Cool, but isn't this a better subject for a conference talk?
>
> > I analyzed every Fixes: tag in the kernel's 20-year git history (125K
> > bug-fix pairs) and built a model to catch vulnerabilities at commit
> > time. Some findings that might be relevant to KernelCI's testing
> > strategy:
> >
> > - Security bugs hide for 2.1 years on average; race conditions persist =
5.0 years
> > - 117 "super-reviewers" (including Dan Carpenter, who invented the
> > Fixes: tag) catch bugs 47% faster
> > - Subsystems like CAN bus (4.2 years) and SCTP (4.0 years) have
> > dramatically longer bug lifetimes than gpu/i915 (1.4 years)
> > - Weekend commits are 8% less likely to introduce bugs, but take 45%
> > longer to fix (review coverage effect)
> >
> > The model (VulnBERT) achieves 92% recall at 1.2% false positive rate
> > on held-out 2024 data. I'm also working on SmartKuang, an RL-based
> > system that has reproduced CVE-2022-34918 autonomously.
>
> I hate to say "your ai model could be replaced with a sql statement",
> but really, we do have tools that show this today that give all of this
> data in a sqlite database that people can use to mine for the same info.
> It's what the kernel CVE team uses to track bug fixes over time for
> their work:
>         https://git.sr.ht/~gregkh/verhaal
> and is part of the vulns.git repo on git.kernel.org
>
> Also for the tracking of employer to people and who is doing the work,
> see the reports on lwn.net for the past few decades that have been
> documenting this.  The tool for that is also public (but part of the
> database of employer mapping is not for obvious reasons, sorry).  I
> think you undercounted people's employers a lot as you can not always
> rely on email addresses to convey this.
>
> Anyway, I liked your reports as I'm always interested in more people
> mining our public data for stuff like this, it's great to see.  But with
> regards to kernelci, how do you feel this information can help with our
> project?  What would you like us to do based on what you have found
> here?
>
> thanks,
>
> greg k-h