Re: Talk proposal: What 125K kernel bugs tell us about testing gaps

Thanks Greg and Don, this is exactly the guidance I needed.

I'll dig into AUTOSEL, the eBPF AI review workflow, and Coccinelle.
The path makes sense: prove value by running on real patches and
engaging on-list, iterate based on feedback, let adoption happen
organically.

Will report back when I have something worth showing.

Kindly,
Jenny

On Thu, Feb 5, 2026 at 10:24=E2=80=AFAM Donald Zickus <[email protected]> =
wrote:
>
> Hi Jenny,
>
> On Thu, Feb 5, 2026 at 4:44=E2=80=AFAM Jenny Qu <[email protected]> wro=
te:
>>
>> [resending to list - accidentally replied off-list]
>>
>> On Wed, Feb 04, 2026 at 11:00:00PM, Greg KH wrote:
>> > I hate to say "your ai model could be replaced with a sql statement"
>>
>> Fair point on the descriptive statistics. I should have been clearer:
>> the 125K bug analysis was training data, not the contribution. verhaal
>> and the LWN employer reports (Jonathan Corbet's per-release stats
>> using the gitdm database) already cover the descriptive side well.
>>
>> The part SQL can't do is the predictive model. VulnBERT takes a raw
>> git diff *before merge* and predicts whether it introduces a
>> vulnerability. The evaluation is a strict temporal holdout: trained
>> on commits with Fixes: tags from <=3D2023, tested on 2024 commits that
>> later received Fixes: tags. 92% recall, 1.2% FPR on that split.
>>
>> To be direct about limitations: those numbers are on historical data
>> where we know ground truth. The model catches patterns it's seen
>> before (unbalanced refcounts, missing NULL checks, lock/unlock
>> mismatches). It will miss novel bug classes it hasn't been trained on.
>> It's a triage tool and not yet an oracle.
>>
>> And it's not ready for production use yet. I'm reworking the
>> architecture. The current approach uses CodeBERT embeddings with
>> handcrafted features, and I think incorporating LLM reasoning traces
>> over diffs will do substantially better. I don't want to hand anyone
>> a tool that generates false confidence.
>>
>> On employer attribution: you're right, email domain mapping
>> undercounts significantly. Developers using personal emails,
>> acquisitions (Mellanox -> NVIDIA), and consultants all break the
>> heuristic.
>>
>> > how do you feel this information can help with our project? What
>> > would you like us to do based on what you have found here?
>>
>> Honestly, I'd rather hear from the KernelCI community what would
>> actually be useful than prescribe solutions. But two directions I
>> think are worth discussing:
>>
>> 1. Subsystem-level test prioritization. The lifetime gap between
>>    CAN bus (4.2 years) and gpu/i915 (1.4 years) almost certainly
>>    reflects testing coverage differences. i915 has dedicated
>>    fuzzing infrastructure and active reviewers like Chris Wilson
>>    and Ville Syrjala. KernelCI could use lifetime data as a signal
>>    for where to invest in test enablement. This is actionable now,
>>    no ML required.
>>
>> 2. Longer-term: commit-level risk scoring to allocate CI resources.
>>    Flag high-risk commits for extra sanitizer runs, longer fuzzing
>>    passes. Low-risk commits get the standard pipeline. But this
>>    needs a model I trust enough to deploy, and I'm not there yet.
>>
>> I'm speaking at BugBash 2026 in April and looking at LPC for a more
>> technical deep-dive.
>
>
> Thanks for this.  As a board member of KernelCI, most of the efforts we h=
ave funded or try to support are ones that have been adopted by the communi=
ty.  The work we try to sponsor needs to provide value to the community but=
 the kernel community can be tricky to navigate as you can see by Greg's co=
mments.
>
> I would recommend those conferences but also try attaching your work as r=
eplies to various patches.  Try to show off the value of your work on maili=
ng lists and let that start conversations on how to steer it towards someth=
ing that could be considered useful.  That journey will lead to overlap of =
existing technologies that Greg mentioned, but more importantly it will lea=
d to conversations on how to collaborate around those technologies to make =
something valuable to the community.  The end result being that it becomes =
a no-brainer to add to kernelci.
>
> A current example we are working with is Thorsten's regzbot[0].  A diffic=
ult social problem around regression tracking that the community helped him=
 navigate towards something of value and now makes sense for kernelci to sp=
onsor.
>
> Cheers,
> Don
>
> [0] - https://linux-regtracking.leemhuis.info/about/
>
>>
>> kindly,
>> Jenny
>> [email protected]
>>
>>
>> On Wed, Feb 4, 2026 at 11:00=E2=80=AFPM Greg KH <gregkh@linuxfoundation.=
org> wrote:
>> >
>> > On Wed, Feb 04, 2026 at 06:49:57PM -0800, Jenny Qu wrote:
>> > > Hi,
>> > >
>> > > I'm a security researcher working on automated kernel vulnerability
>> > > detection. I'd love to present at an upcoming Thursday call if there=
's
>> > > interest.
>> >
>> > Cool, but isn't this a better subject for a conference talk?
>> >
>> > > I analyzed every Fixes: tag in the kernel's 20-year git history (125=
K
>> > > bug-fix pairs) and built a model to catch vulnerabilities at commit
>> > > time. Some findings that might be relevant to KernelCI's testing
>> > > strategy:
>> > >
>> > > - Security bugs hide for 2.1 years on average; race conditions persi=
st 5.0 years
>> > > - 117 "super-reviewers" (including Dan Carpenter, who invented the
>> > > Fixes: tag) catch bugs 47% faster
>> > > - Subsystems like CAN bus (4.2 years) and SCTP (4.0 years) have
>> > > dramatically longer bug lifetimes than gpu/i915 (1.4 years)
>> > > - Weekend commits are 8% less likely to introduce bugs, but take 45%
>> > > longer to fix (review coverage effect)
>> > >
>> > > The model (VulnBERT) achieves 92% recall at 1.2% false positive rate
>> > > on held-out 2024 data. I'm also working on SmartKuang, an RL-based
>> > > system that has reproduced CVE-2022-34918 autonomously.
>> >
>> > I hate to say "your ai model could be replaced with a sql statement",
>> > but really, we do have tools that show this today that give all of thi=
s
>> > data in a sqlite database that people can use to mine for the same inf=
o.
>> > It's what the kernel CVE team uses to track bug fixes over time for
>> > their work:
>> >         https://git.sr.ht/~gregkh/verhaal
>> > and is part of the vulns.git repo on git.kernel.org
>> >
>> > Also for the tracking of employer to people and who is doing the work,
>> > see the reports on lwn.net for the past few decades that have been
>> > documenting this.  The tool for that is also public (but part of the
>> > database of employer mapping is not for obvious reasons, sorry).  I
>> > think you undercounted people's employers a lot as you can not always
>> > rely on email addresses to convey this.
>> >
>> > Anyway, I liked your reports as I'm always interested in more people
>> > mining our public data for stuff like this, it's great to see.  But wi=
th
>> > regards to kernelci, how do you feel this information can help with ou=
r
>> > project?  What would you like us to do based on what you have found
>> > here?
>> >
>> > thanks,
>> >
>> > greg k-h
>>