bug#81520: Add pre-receive hook to Savannah to reject LLM-encumbered commits

Jim Porter [17/Aug  1:07pm -07] wrote:
> Subject: [PATCH 1/2] Simplify commit msg file hook
>
> * build-aux/git-hooks/commit-msg-files.awk (get_commit_changes): Use
> "git diff-tree" to simplify parsing of touched files.
>
> * build-aux/git-hooks/pre-push: Use a simpler invocation of "git
> rev-list" to get all commits to be pushed.
> ---
>  build-aux/git-hooks/commit-msg-files.awk | 20 ++++----
>  build-aux/git-hooks/pre-push             | 60 ++++++------------------
>  2 files changed, 23 insertions(+), 57 deletions(-)

I'm uncomfortable with large refactorings like this without tests to
prove that they don't also introduce a functional change.  Is it not
possible to make the substantive changes you want to make without also
including these refactorings?  Alternatively, perhaps you could write
some tests.

> +** LLM usage policies
> +
> +The GNU project doesn't allow LLM-generated outputs in contributions.
> +This applies to all code, documentation, test data, and commit messages.
> +
> +This is enforced by a git hook that runs when committing or pushing
> +changes.  This hook checks for the names of common LLMs (or their
> +harnesses), as well as the following fields in the commit message:
> +"Assisted-by:", "AI-assisted:", and "Claude-session:".
> +

The text should make it clear that it's a provisional policy pending
work with the FSF is doing with lawyers.

> -    if (status != 0) {
> -      print "Commit aborted; please see the file 'CONTRIBUTE'"
> -    }
>      exit status
>    }
>  ' <"$1"
> +lint_result=$?
> +
> +AUTHOR=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/\1/p')
> +COMMITTER=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/\1/p')
> +(echo "Author: $AUTHOR"; echo "Committer: $COMMITTER"; cat $1) | \
> +  awk -f "$HOOKS_DIR"/commit-msg-llm-check.awk
> +llm_result=$?
> +
> +if [ "$llm_result" -ne 0 ]; then
> +  echo; cat "$HOOKS_DIR"/commit-msg-llm-error.txt; echo
> +fi
> +
> +if [ "$lint_result" -ne 0 -o "$llm_result" -ne 0 ]; then
> +  echo "Commit aborted; please see the file 'CONTRIBUTE'"
> +  exit 1
> +fi

I don't think we should prioritise the LLM failure like this.
I.e. I think you should have only the latter of these if statements.

I don't think we need a special .txt file here.
The combination of CONTRIBUTE, AGENTS.md and CLAUDE.md is enough.

> +function get_short_sha(commit_sha) {
> +  cmd = ("git rev-parse --short " commia_sha)

Type "commia".  Surprised this didn't come up in testing.

Probably the refactoring to prepare a short SHA in this more principled
way (thanks) should be its own commit.

> -BEGIN {
> -  if (reason == "pre-push")
> -    verbose = 1
> -}
> -
>  /^[a-z0-9]{40}$/ {
> -  if (! check_commit_msg_files($0, verbose)) {
> +  if (! check_commit_msg_files($0, verbose))
>      status = 1
> -  }
>  }
>
>  END {
> -  if (status != 0) {
> -    if (reason == "pre-push")
> -      error_msg = "Push aborted"
> -    else
> -      error_msg = "Bad commit message"
> -    printf("%s; please see the file 'CONTRIBUTE'\n", error_msg)
> -  }
>    exit status
>  }

Not sure why you're making these structural changes.

> +# Check for fields that commonly indicate LLM usage.
> +tolower($0) ~ /^(assisted-by|ai-assisted):/ { report_error($0) }
> +tolower($0) ~ /^claude-session:/            { report_error($0) }
> +
> +# Check for common LLMs/harnesses.  We purposefully exclude "Claude"
> +# here because it's also a human name.
> +tolower($0) ~ /openai/    { report_error($0) }
> +tolower($0) ~ /anthropic/ { report_error($0) }
> +tolower($0) ~ /copilot/   { report_error($0) }

Maybe we could match 'claude' case-sensitively?

> +  # Check for LLM attributions in all outgoing commits.  We do this in a
> +  # separate pass from above so that the output is kept separate.
> +  get_outgoing_shas "$local_sha" | (while read outgoing_sha; do
> +    short_sha=$(git rev-parse --short "$outgoing_sha")
> +    get_commit_msg "$outgoing_sha" | \
> +      $awk -v commit_sha="$short_sha" -f "$HOOKS_DIR"/commit-msg-llm-check.awk
> +    status=$(expr "$?" "|" "$status")
> +  # Since commands in pipelines all run in their own subshells,
> +  # explicitly exit with the overall status so the parent shell knows
> +  # the result.
> +  done; exit "$status")
> +  llm_result=$?
> +done
> +
> +if [ "$llm_result" -ne 0 ]; then
> +  echo; cat "$HOOKS_DIR"/commit-msg-llm-error.txt; echo
> +fi
> +
> +if [ "$files_result" -ne 0 -o "$llm_result" -ne 0 ]; then
> +  echo "Push aborted; please see the file 'CONTRIBUTE'\n"
> +  exit 1
> +fi
> +
> +exit 2

Again I don't think LLM failures deserve to be treated so separately.

-- 
Sean Whitton