Re: request for help: LLM-based quality assurance

Hello,

On Tue, Jun 02, 2026 at 07:55:58PM +0200, Bruno Haible via Gnulib discussion list wrote:
> Anyone out here who is familiar with LLMs (or wants to get familiar with
> LLMs): How about using it not for coding, but for checking commits that
> went into gnulib master?
> 
> Since 2026-01-01, at least 17 gnulib commits contained regressions, that
> had to be fixed subsequently. We often detect regressions by code review
> or by a CI run. The problems:
>   - Not all commits gets reviewed from a different developer than the
>     committer. (Like many free software projects, Gnulib lacks good reviewers.)
>   - The CI runs possibly a week later. (We can't increase the frequency,
>     because some CI runs fail due to network problems or other noise,
>     and this noise needs to be filtered out.)
> 
> As a complement to these QA techniques, Paul Eggert suggests to use an
> LLM to analyze the commits that have been pushed into gnulib master.
> 
> This should be promising, because I read recently that LLMs outperform
> all classical static analysis tools, when it comes to analyzing source code.
> 
> I can't do this myself, because I'm already quite loaded with the existing
> QA techniques and with my work on other GNU packages.
> 
> Therefore, if you volunteer, please step up!

I can run the analysis manually as a first try. Since I don't know which
commits are these 17 that contained regressions and I don't follow
gnulib development that closely, I think I am a good candidate to
perform a double blind experiment. Perhaps you could start by providing
some commits to analyze (some of them that contain known regressions,
some of them that don't - without indicating which one is which) to have
a smaller set than e.g. all commits since 2026-01-01?

Regards, Pavel