Re: request for help: LLM-based quality assurance

Tim Rühsen <[email protected]> Tue, 9 Jun 2026 11:00:38 +0200
Newsgroups	gmane.comp.lib.gnulib.bugs
Message-ID	<[email protected]>
Hi Bruno

>> Another experiment with Claude code (closed source agent) connected to
>> the same local LLM as before found the issue in commit
>> 17dc60e624cd6fc3491f9cb002f760d60e66ce8b:
>>
>> "Important caveat: The original commit 17dc60e624 introduced a bug in
>> mbrtowc.c — it replaced MBRTOWC_EMPTY_INPUT_BUG with
>> MBRTOC32_EMPTY_INPUT_BUG (wrong macro name) and added _GL_SMALL_WCHAR_T
>> (mbrtoc32-specific). This was fixed by follow-up commit 2ca51a77e6. Both
>> commits should be evaluated together."
> 
> Well, if it knows about the follow-up commit, we don't know whether it
> found the regression independently of that.

Good point, it likely like you say. To confirm, I git reset --hard to 
the commit to be reviewed and Claude + Qwen3.6-35B didn't report this 
bug any more. I'll (slowly) continue my testing and let you know if I 
find a usable OSS/open-weight agent+model combination.

>> It produced a longish result/summary explaining the issue in details.
> 
> Too much details are not advisable. Because what is cheaper: Reviewing
> a 20-lines patch or reading a 50-lines analysis report?

This is a matter of what you ask for. LLMs even allow you to generate 
machine-readable output (as JSON).

> In general, we should strive for the locally running open-weights models.

I agree.

> If it's not locally running, it's SaaS, and what people report is that
> the quality slowly gets worse over time, until a new version of the LLM
> is released, at which moment the quality rises again - but with unpredictable
> effects on the particular workflow.

New Newer models get better in general, but that older models getting 
worse over time doesn't match my experienced. They are tuned sometimes, 
and depending on your tasks, you may perceive that as better or worse. 
Even changes to the underlying hardware may change the results. That's 
also one of the reasons why you can't get reproducible results for SaaS 
models.

> Occasionally it happens that the closed-source alternative is more usable
> than the open-source one. For instance, I did not get a good experience
> with GitLab CI (open-source), because I could not find out how to store
> large log files in the case of failed builds. Whereas in GitHub the handling
> of the log files is not perfect either, but at least reasonable.

For LLMs, I'd say that SaaS models in general are much better than 
(small) open-weight models. It's a matter of compute power / resources.

Regarding Gitlab, did you try to tee the build log into a file and 
upload it as job artifact? The SaaS limit is 5GB. Well, I definitely 
missed the discussion/decision, do you have a link at hand?


> But here, in the LLM space, the situation is different: If you stick
> to an SaaS model, you are forced to update the prompt or AGENTS.md file
> every two months. Which may be unreasonably costly in the long run.

The AGENTS.md should be relatively vendor/model agnostic. It should 
describe the directory/file layout, or in general it should contain 
project-specific information that helps LLMs to reduce token costs (and 
thus give you answers faster and with higher quality).

The vendor/model specific prompts are contained in the system prompt of 
the agent. For example Claude Code works best with Anthropic models - 
because the system prompt is fine-tuned. The gemini agent works best 
with Google's Gemini models. Etc.
Still, you can use these agents with any model, also with local models.

They all read the same AGENTS.md file.

Tim
OpenPGP_signature.asc (application/pgp-signature, 840 B)