Re: request for help: LLM-based quality assurance

Bruno Haible via Gnulib discussion list <[email protected]> writes:

> Hi,
>
> Anyone out here who is familiar with LLMs (or wants to get familiar with
> LLMs): How about using it not for coding, but for checking commits that
> went into gnulib master?
>
> Since 2026-01-01, at least 17 gnulib commits contained regressions, that
> had to be fixed subsequently. We often detect regressions by code review
> or by a CI run. The problems:
>   - Not all commits gets reviewed from a different developer than the
>     committer. (Like many free software projects, Gnulib lacks good reviewers.)
>   - The CI runs possibly a week later. (We can't increase the frequency,
>     because some CI runs fail due to network problems or other noise,
>     and this noise needs to be filtered out.)
>
> As a complement to these QA techniques, Paul Eggert suggests to use an
> LLM to analyze the commits that have been pushed into gnulib master.
>
> This should be promising, because I read recently that LLMs outperform
> all classical static analysis tools, when it comes to analyzing source code.
>
> I can't do this myself, because I'm already quite loaded with the existing
> QA techniques and with my work on other GNU packages.
>
> Therefore, if you volunteer, please step up!

I'd imagine paying for it would be quite expensive. That is why I never
bothered looking into it.

The LLM companies have 6 month free (as in monetary) offers for Open
Source [1] [2]. But that reads to me like they believe they will profit
off you in 6 months. Also, they use stupid metrics like GitHub stars, so
we would likely not qualify. It is especially ironic since their models
suggest using our commands frequently...

Collin

[1] https://claude.com/contact-sales/claude-for-oss
[2] https://developers.openai.com/community/codex-for-oss