[PATCH 0/3] reftable/stack: avoid reloading the stack when locked

This patch series is based on the report by Jeff [1], where he noticed
that when creating a lot of refs within a single reference transaction,
the majority of the time was spent on fstat().

The issue stems from the fact that within the reftable library we do not
track Git reference transactions, as such any calls within the library
would potentially reload the stack to ensure that there are no
concurrent updates made to the stack. While this makes sense outside of
a reference transaction, within one, the stack is locked, so reloading
the stack is a no-op. The only time we want to reload the stack is
immediately after locking the list file, which is to catch any
concurrent updates made to the stack.

The first patch in this small series, cleans up the flow of reloading
the stack by providing a flag explicitly. The patch argues that since
all flows reload the stack, the flag can be safely removed. This
simplifies the flow of when to reload the stack.

The next two commits move the lock variable to the reftable_stack
structure and then use this information to decide if reloading of the
stack is necessary.

During benchmarking, I first tried to benchmark adding new references
against HEAD. This kicks in the DWIM ref resolution, and we iterate over
siz difference candidate ref names before settling on a match. Each such
lookup reloads the stack. This happens before the reference transaction
is created. I quickly realized that this would dominate the benchmarks,
so the benchmarks in the third patch are against a static commit OID.

The benchmarks show a consistent 1-2% improvement in clock time for
'git-update-ref(1)', but such low values could also be chalked to being
within an error rate. However, the syscall counts show that now the
calls to `newfstatat()` stay constant at around 55 calls regardless of
the number of refs to be created. Before this would grow linearly with
the number of refs.

[1]: https://lore.kernel.org/git/[email protected]/

Signed-off-by: Karthik Nayak <[email protected]>
---
Karthik Nayak (3):
      reftable/stack: remove `REFTABLE_STACK_NEW_ADDITION_RELOAD`
      reftable/stack: move list lock to `struct reftable_stack`
      reftable/stack: avoid reloading the stack when already locked

 refs/reftable-backend.c         | 18 ++++-------
 reftable/reftable-stack.h       | 17 ++--------
 reftable/stack.c                | 69 +++++++++++++++++------------------------
 reftable/stack.h                |  7 ++++-
 t/unit-tests/u-reftable-stack.c | 69 ++++++++++++++++++-----------------------
 5 files changed, 75 insertions(+), 105 deletions(-)


---
base-commit: 18e66859d87fb4b76599f73460b54f0848c76b16
change-id: 20260814-740-optimize-reloading-the-reftable-stack-f5f3adf0a0c0


Thanks
- Karthik