[PATCH bpf-next v3 00/13] bpf: Support aggregate return values up to 16 bytes

LLVM 23 can return an __int128, or a struct/union larger than 8 bytes and
no larger than 16 bytes, in the BPF R0:R2 register pair [1][2]. Before
that the BPF backend could not return such values at all: a by-value
aggregate return was rejected at compile time with "aggregate returns are
not supported", and an __int128 return failed in the backend with "unable
to allocate function return #1".

This series teaches the kernel the same convention, so that BPF programs
and kfuncs can return these values. The first 8 bytes of the value come
back in R0 and the second 8 bytes in R2. It applies to kfunc returns and
to BPF-to-BPF subprogram returns, both global and static. The main program
is unchanged: its return value is the program's exit code, so a return
larger than 8 bytes is still rejected at BPF_EXIT.

Patches 1-2 are preparation: patch 1 factors out the per-register check
used by the global return path, and patch 2 adds the shared helpers that
answer "does this subprogram return a register pair", so that the patches
which follow can be ordered independently. Patch 3 wires up the JIT side.
Patches 4-5 teach precision backtracking and live register analysis about
R2 as a second return register, ahead of the patch that starts modeling it.
Patch 6 rejects callbacks returning more than 8 bytes, since neither
bpf_callback_t nor bpf_exception_cb has a second return register. Patch 7
adds the verifier support proper. Patch 8 rejects a pair return once the
subprogram's BTF has been marked unreliable, and patch 9 relaxes
btf_distill_func_proto() and btf_validate_return_type(), which is what
makes the whole thing reachable. Patches 10-12 add selftests and patch 13
documents the convention.

Constraints worth calling out:

 - A by-value struct or union returned by a kfunc or by a global subprogram
   must be composed only of scalars. The verifier models the returned
   register bits as an unknown scalar, so a pointer member would be
   laundered into one and escape provenance and reference tracking. A
   static subprogram is verified inline and is not restricted this way.

 - Returning the pair from a kfunc needs the JIT to place the second half
   into R2, which is architecture-specific work. Architectures opt in
   through bpf_jit_supports_kfunc_ret_reg_pair(); x86_64, arm64 and riscv64
   do so here, and elsewhere bpf_add_kfunc_call() rejects such a kfunc with
   -EOPNOTSUPP. A register-pair return from a BPF subprogram needs no such
   capability.

 - The interpreter propagates R0 alone out of a subprogram, so the JIT is
   forced wherever a caller can observe the pair.

 - The compiler side requires LLVM 23 or newer. The selftests written in C
   record which compiler built them in a read-only flag and report a skip
   rather than a pass when built by anything older; the inline-asm tests do
   not depend on the compiler and run everywhere.

  [1] https://github.com/llvm/llvm-project/pull/190894
  [2] https://github.com/llvm/llvm-project/pull/206876

Changelog:
  v2 -> v3:
    - v2: https://lore.kernel.org/bpf/[email protected]/
    - Add additional guard with __SIZEOF_INT128__ for 32bit kernel.
    - Guard little endian to avoid arm64 big endian for selftests.
    - Fix test failures for gcc15.
    - Rebase to avoid conflict with latest master branch.
  v1 -> v2:
    - v1: https://lore.kernel.org/bpf/[email protected]/
    - Split the R0:R2 helpers out of the verifier patch into their own
      preparation patch, and reordered the series so the core verifier patch
      comes after the infrastructure it depends on.
    - New patch rejecting callbacks that return more than 8 bytes, both
      helper/kfunc callbacks and exception callbacks, with selftests.
    - New patch rejecting a register-pair return once btf_check_subprog_call()
      has marked the subprogram's BTF unreliable, rather than silently
      mistracking R2.
    - Folded "bpf: Force JIT for programs using the R0:R2 register pair" into
      the verifier patch.
    - Dropped "bpf: Reject >8 byte return values on return-reading trampoline
      paths" and its selftests; that went in separately as commit
      c48796aa6c39.
    - Described the register mapping as the first and second 8 bytes rather
      than the low and high 64 bits, which is only correct on little-endian,
      and reworded "16-byte" to "up to 16 bytes" where the range 9..16 was
      meant.

Yonghong Song (13):
  bpf: Factor check_global_ret_scalar_reg() out of the global return
    check
  bpf: Add helpers to describe the R0:R2 return register pair
  bpf: Wire up JIT support for 16-byte kfunc returns
  bpf: Track R2 of register-pair returns in precision backtracking
  bpf: Account R2 of register-pair returns in live register analysis
  bpf: Reject callbacks returning more than 8 bytes
  bpf: Add verifier support for 16-byte returns in R0:R2
  bpf: Reject register-pair returns when the subprog BTF is unreliable
  bpf: Enable aggregate return types up to 16 bytes
  selftests/bpf: Add C tests for 16-byte returns in R0:R2
  selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
  selftests/bpf: Add tests for callbacks returning more than 8 bytes
  Documentation/bpf: Document up to 16-byte kfunc return values in R0:R2

 Documentation/bpf/kfuncs.rst                  |  62 +++
 arch/arm64/net/bpf_jit_comp.c                 |   5 +
 arch/riscv/net/bpf_jit_comp64.c               |   5 +
 arch/x86/net/bpf_jit_comp.c                   |  21 +
 include/linux/bpf_verifier.h                  |  16 +
 include/linux/filter.h                        |   1 +
 kernel/bpf/backtrack.c                        |  61 ++-
 kernel/bpf/btf.c                              |  44 +-
 kernel/bpf/core.c                             |   5 +
 kernel/bpf/liveness.c                         |  25 +-
 kernel/bpf/verifier.c                         | 268 +++++++++--
 .../selftests/bpf/prog_tests/aggregate_ret.c  | 173 ++++++++
 .../selftests/bpf/prog_tests/exceptions.c     |   2 +
 .../selftests/bpf/prog_tests/fexit_bpf2bpf.c  |  15 +
 .../testing/selftests/bpf/prog_tests/timer.c  |   2 +
 .../selftests/bpf/progs/aggregate_ret_func.c  | 420 ++++++++++++++++++
 .../bpf/progs/aggregate_ret_int128_c.c        |  48 ++
 .../selftests/bpf/progs/aggregate_ret_kfunc.c | 126 ++++++
 .../bpf/progs/aggregate_ret_kfunc_c.c         |  66 +++
 .../selftests/bpf/progs/aggregate_ret_run.c   | 178 ++++++++
 .../bpf/progs/aggregate_ret_struct_c.c        |  82 ++++
 .../bpf/progs/aggregate_ret_target.c          |  29 ++
 .../bpf/progs/aggregate_ret_union_c.c         |  58 +++
 .../bpf/progs/btf__exceptions_ret_pair_fail.c |  10 +
 .../bpf/progs/btf__timer_ret_pair_fail.c      |  10 +
 .../selftests/bpf/progs/exceptions_fail.c     |   2 +-
 .../bpf/progs/exceptions_ret_pair_fail.c      |  30 ++
 .../selftests/bpf/progs/freplace_ret_pair.c   |  20 +
 .../selftests/bpf/progs/timer_ret_pair_fail.c |  49 ++
 .../selftests/bpf/progs/verifier_arena.c      |  38 ++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  83 ++++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  48 ++
 32 files changed, 1944 insertions(+), 58 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_run.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_target.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__exceptions_ret_pair_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__timer_ret_pair_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_ret_pair_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/freplace_ret_pair.c
 create mode 100644 tools/testing/selftests/bpf/progs/timer_ret_pair_fail.c

-- 
2.53.0-Meta