Re: request for help: LLM-based quality assurance

Hi Bruno, Pavel

> Thanks for explaining your setup and showing their results.

I think for some kind of decision we need some kind of reproducibility 
(using the loosely here as results from LLMs rarely reproduce exactly).

 > The fact that your setup did not flag this commit as a regression shows
 > that Pavel's setup is more useful.

Absolutely. That's why I am very curious about the setup.

Another experiment with Claude code (closed source agent) connected to 
the same local LLM as before found the issue in commit 
17dc60e624cd6fc3491f9cb002f760d60e66ce8b:

"Important caveat: The original commit 17dc60e624 introduced a bug in 
mbrtowc.c — it replaced MBRTOWC_EMPTY_INPUT_BUG with 
MBRTOC32_EMPTY_INPUT_BUG (wrong macro name) and added _GL_SMALL_WCHAR_T 
(mbrtoc32-specific). This was fixed by follow-up commit 2ca51a77e6. Both 
commits should be evaluated together."

It produced a longish result/summary explaining the issue in details.
What makes Claude code different from the pi agent (OSS) is it's very 
verbose system prompt, which alone is >20.000 tokens. I did not use a 
skill this time. Claude took 15mins vs. pi taking 3mins in the previous 
experiment.

This tells me, that we have to craft a proper system prompt and/or a 
review skill to get comparable results from an oss agent + open weight 
model.

Do we already know whether we tend to
   - best results using closed source, closed weights, energy intensive, 
costly
or
   - best effort using open source, open weights, 100x (1000x?) less 
energy intensive, minimal costs
?

Does it make sense to work on a gnulib specific AGENTS.md as well as on 
a system prompt that can be used for reviews?

> 15 lines of suggestions about a #if, without noticing that this #if condition
> is in fact buggy (incomplete): It needs to be enabled on all platforms which
> test pwc or n. Including the cases MBRTOWC_NUL_RETVAL_BUG and
> MBRTOWC_IN_C_LOCALE_MAYBE_EILSEQ. I'm fixing that through the patch below.

Thank you, Bruno!

Regards, Tim

-----BEGIN PGP SIGNATURE-----

wsF5BAABCAAjFiEEHLJ9vJhhSy1YQWRtCDAttqJnBCgFAmomkycFAwAAAAAACgkQCDAttqJnBCjf
lBAAmGrGOHtQ6UYXeKcDcKNveJ6W0mt9WNKMCsq/s2bz6dxYlHzw/IWQW9pI8E74rsK6sWCHV+Vz
iRaRcv/fiuSNb6PqcMejiybSBsiPm0f9u4hAkeoYvG2ax/RQGM9OldYtpmWPFiCRI0vghkKdIg6F
Sj1f76gwg3lwMeOe7neu2wmHDCOR8qFXcvd0z/sZRQgd9P5gNesCg+Th0PcY4FFooDKIccfVLdrB
66gQeRRyOszoxUOPAtSmKKB5Bak+c0+2BNzIRa8t73Ptqjxpmu6u6QK8SaM8CWgFQzrTyIs/yd09
d9AT2UYCUYrbxQktS1os9DRa6FAPXJynDQTQ8iZQ/w18QRkI1mOy2aq/31hbR0oZ48RchOhcN0BF
yxhZsY228k0v0l8tn/r4u6sVLUAWm4P/P3/GksOgMztKZDe+jZURJbiksrtQ6X5Jghy4IXdVhOpS
zYlymVYFlxQVSQ1eruLZLyos1whkd23DWOr6l1qYTPzZc9kiLbHwqe4vtIlkGon05neEM99pDpbF
yS8fsnr7s+4y+8+ADUHsTGxT/5c298vLiHyChx3QvIDdu86aBzYLe6fD9s8X7EiPWduW+GDHK7pQ
mYn1rdKCUbCpqzy80LJkykYDGskxadw5jV/XKnM4+SIsTw9Ts+WuJqlx5S2jz/dQTFZdp9aQrzLg
dX8=
=sDD9
-----END PGP SIGNATURE-----