Re: Policy on use of LLM tools and bug fixes

Giacomo Tesio <[email protected]> writes:

> Hi Florian,
>
> On Fri, 24 Apr 2026 12:48:42 +0200 Florian Weimer wrote:
>
>> Contributions created using machine learning techniques go back to the
>> 2000s at least, I think.
>
> That's why I didn't wrote wither "AI" or "Machine Learning".
> Large Language Models are a pretty well defined class of software these
> days. Albert mentioned Claude and Copilot, for example.
>
>> Could you define what, exactly, “LLM-generated” means?
>
> Any sequence of bytes computed by a local or remote large language
> model that have been included in GCC repository.
>
> I care most about those sequence of bytes that influence the
> compilation process and its output binaries (code, .po etc..)
>
> You know, trusting trust and so forth...
>
>> Tools for vibe-coding test cases are well-established within the GCC
>> community.
>
> Good!
> They are well-established in our company too.
>
> Actually, since xz-utils based attack, we consider open source
> test suites among the attack vectors. As for our own generated tests,
> we have policies about how to commit them, in dedicated commits,
> with detailed tool info and full prompt.

We don't generally have binaries in the GCC testsuite and we try to keep
committed testcases quite small.

I don't think it would be useful to share full cvise interestingness
tests for how they were reduced. I often do some hand reduction at the
start and end too.

>
> While it might seem excessive to hobbists developers, in several
> occasions, reviewers found weird mismatches between prompt
> and generated (passing) tests. In a couple of cases though, only
> branch coverage analysis made us realize that use cases we considered
> properly tested were not because the test code diverged from the prompt.
>

Yes, this is a concern I have in general, though mostly for vacuous unit tests.

>
> In any case, my question to Albert (and others LLM users contributing
> to GCC) is just related to LLM output that was included in GCC codebase
> and influence build output and build process.
>
> I hope GCC policy will cover statistically generated tests too,
> and fwiw I can say that recording test-related prompts is useful in the
> long run, but that's up to the SC to decide.
>
>
> Giacomo

-----BEGIN PGP SIGNATURE-----

iQEBBAEWCgCpFiEEJaa7iN2bdkxrVUHCc4QJ9SDfkZAFAmnrelcbFIAAAAAABAAO
bWFudTIsMi41KzEuMTIsMiwyXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25z
Lm9wZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQyNUE2QkI4OEREOUI3NjRDNkI1NTQx
QzI3Mzg0MDlGNTIwREY5MTkwDxxzYW1AZ2VudG9vLm9yZwAKCRBzhAn1IN+RkIDD
AQDv8gC4EqLB6aR41OTZCs9m/fUpR5kxQiRFfS9KJGzlKQEA0iDtrIuCSCiUJcga
nXmOzUaFEUKw566NzFUUFwtTJQE=
=zQzb
-----END PGP SIGNATURE-----