Re: request for help: LLM-based quality assurance

Hi Jeffrey,

> However, there are downsides to LLMs, and I would look
> into fixing the current processes while using the LLMs as a complement
> to existing practices.

That's a reasonable approach too.

> First, code should not be merged into Master until the CI tests have
> successfully run.  Commits that don't pass the CI test don't pass
> through the security gate.  The new code stays in a development branch
> or testing fork until they pass the CI tests.  This is how many (all?)
> organizations with a mature SDLC operate.

There are three reasons why we don't do it this way:

  - Gnulib commits can introduce many platform specific issues, and
    therefore the "CI tests" would need to be the multi-platform CI.
    This multi-platform CI is expensive: currently it takes more than
    11 hours of CPU time [1]. Even if GitHub use if free for us, there
    is some environmental cost: Assume 100 W per CPU, it consumes 1.1 kWh
    for a single run, that is ca. 0.21 $ in USA or 0.40 € in Europe.

  - GitHub not being Free Software, we cannot make its use mandatory for
    all Gnulib contributors.

  - It is demotivating to have to struggle with such a gate. We all know
    that the "fun factor" is higher if a developer can complete a piece
    of work without going through robots and gates.

> Second, LLMs have at least three costs.  First is the deskilling that
> happens when relying on them. [1,2].  Relying too much on LLMs will
> have a negative effect on the talent contributing to the project.

I agree: When Collin or I review a commit from Paul, we can 1) learn from
him and 2) learn about Gnulib's internals. This skills build-up is lost
if an LLM takes over this review.

> Second is the power consumed powering the algorithms, and its effect
> on renewable energy.[3,4]  The environmental and societal cost seems
> to be high.

I tend to agree: 15 minutes for an LLM-generated pretranslation of a
PO file seems adequate (because it would replace 1/2 hour of work of
a translator), whereas 15 minutes for an LLM-generated patch review
seems excessive (because a developer would do it in 1-2 minutes).

But things tend to get more efficient over time. Therefore if it's
not reasonable for cost reasons today, maybe it will be in two years?

Bruno

[1] https://github.com/gnu-gnulib/ci-testdir-check/actions/runs/27127350805