[PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead
Matt Turner <[email protected]>
| Newsgroups | org.nongnu.qemu-devel |
|---|---|
| Message-ID | <[email protected]> |
For guests running large amounts of code, most of what TCG executes is not
translated guest work but the fixed overhead around it. Blocks are short and
there are a great many of them, so the constant cost at each end of a block
(the interrupt poll and the can_do_io stores on entry, the dispatch on exit)
ends up dominating everything else.
The workload throughout is qemu-alpha running an emulated alpha gcc 16.2.0
compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64
host, in a --static --enable-lto --target-list=alpha-linux-user build. It
executes 34.2 billion TBs at 6.04 guest instructions each, and 24.6% of its
TB exits cannot use goto_tb. That is a representative shape for any guest
whose text is much larger than a page: indirect calls and returns
everywhere, plus direct branches that merely crossed a page boundary.
The first three patches are ordinary cleanups that stand on their own; patch
3 picked up review tags in v2. The remaining four are marked RFC individually
and are where the interesting questions are.
1 accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
curr_cflags() recomputed three unlikely tests on every one of the run's
8.4 billion dispatches, from state that changes only when gdb enables
single-step, when one-insn-per-tb is toggled, or when the log mask
moves. Fold each into tcg_cflags where it changes. -5.15%
2 accel/tcg: enlarge the TB jump cache to 64K entries
4096 entries is too small for a guest running a large program;
tb_htable_lookup() is 6.10% of samples. 16 bits is the knee of the
sizing curve, at 1 MiB per vCPU. -5.92%, -8.71% wall
3 accel/tcg: skip the can_do_io stores in user-only builds
Two stores per TB that nothing in a user-only build reads: 68 billion
of them over the run. -4.57%, -4.53% wall
4 RFC: tcg: probe the TB jump cache inline instead of calling a helper
95.8% of those 8.4 billion helper_lookup_tb_ptr() calls hit the jump
cache. Emit the probe inline (hash, four guarded loads, goto_ptr) and
call the helper only on a miss. -34.67%, -25.94% wall
5 RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
translator_use_goto_tb() refuses to chain across a page. In user-only
builds the invalidation path already covers what that was protecting
against: every mmap/mprotect/munmap reaches page_set_flags(), which
invalidates and unlinks. Lift it there for runs that can never acquire
a breakpoint, keep it for system mode. -2.75%, -4.84% wall
6 RFC: accel/tcg: poison the jump cache instead of polling for indirect
exits
A block only needs the icount_decr poll if it can leave by goto_tb;
every other exit already passes through a dispatch. Give the inline
probe its own jump cache base pointer and point it at zeroes when an
exit is requested, so every dispatch misses into the helper, which
returns the epilogue. Emit the poll only in blocks that emitted a
goto_tb. -2.52%, -1.96% wall
7 RFC: tcg: fold a guest displacement into the host addressing mode
tcg_gen_qemu_ld/st cannot express a based access, so a target with a
displacement in its encodings materializes the address with an lea
that the host addressing mode would have done for free. Fold a
preceding constant add into a new argument on the op, opt-in per
backend, wired up for x86_64 user-only. -5.70%, -3.20% wall
Each percentage is against the patch before it. Every stage was measured in
one session on the same host, so end to end, from an unmodified LTO build of
the same base to the full series:
instructions retired: 1,646,994,254,249 -> 819,262,147,022 -50.26%
wall clock: 133.19s -> 77.30s -41.96%
The two figures do not track each other, and that is the interesting part:
what the series removes is cheap, well-predicted, highly pipelined work, so
it retires far more instructions than it saves time. Patch 4 also cuts L1
icache load misses by 39.0%, because a dispatch no longer jumps into qemu's
.text and evicts translated code; qemu's own .text falls from 38.8% to 5.3%
of profile samples.
Every revision was built and measured separately, so the series bisects, and
the emulated compiler produces byte-identical assembly output at every step,
which is the correctness check these patches most need. Three new alpha
tests cover the hazards the series creates: tests/tcg/alpha/test-xpage-chain.c
and tests/tcg/alpha/gdbstub/xpage-bp.py (patch 5) and test-indirect-irq.c
(patch 6). Each fails or hangs if the mechanism it covers is removed, which
is what makes them tests of the new behavior rather than of the old.
Changes since v2
================
The biggest change is that "RFC: accel/tcg: only poll for interrupts in
blocks that can close a cycle" is dropped. Richard pointed out that it let a
straight-line run of arbitrary length go unchecked, since a block with no
backward edge polled nowhere. Patch 6 now keeps the poll where a block can
leave by goto_tb and relies on the dispatch everywhere else, which holds the
one-block bound without any analysis of the guest's control flow graph. All
of v2's measurements for that patch were taken with the dropped patch
underneath and have been replaced by a fresh measurement of the series as it
now stands.
1 Rewritten as folding the dynamic bits into CPUState::tcg_cflags where
they change, rather than caching curr_cflags() in a second field.
Monitor-side updates ('one-insn-per-tb on', 'log nochain') are queued
with async_safe_run_on_cpu() so each CPU writes its own cflags.
2 Sizing curve re-measured on top of the new patch 1, now with wall clock
alongside instructions retired.
3 Back to #ifndef CONFIG_USER_ONLY. QEMU's IS_ENABLED() is IS_EMPTY(),
true only for a symbol Meson defines empty, and CONFIG_USER_ONLY is
defined as 1, so v2's test was always false and the stores were emitted
after all. Review tags carried over; the numbers are from the working
form.
4 Folded into tcg_gen_lookup_and_goto_ptr() instead of adding a second
entry point beside it (Richard), which changed all 38 call sites.
cs_base is compared too, which is what lets the probe be enabled
generically rather than per target. Audited which targets may pass a
real PC: five do, the rest pass NULL and keep the helper call. The
breakpoint poison now happens in cpu_breakpoint_insert() rather than
only from the poisoned CPU's own main loop, since a thread dispatching
indirectly need never reach that loop.
5 Only take the shortcut when no gdbstub was requested: the same-page rule
also forces a breakpoint check on entry to every page, and without that
a chain established earlier runs past a breakpoint set later. Reported
by Richard. Changed translator_use_goto_tb() rather than
translator_is_same_page(), which i386, riscv and s390x use for something
else and which v2 perturbed as a side effect. Added the gdbstub half of
the test.
6 Rebased onto the removal described above, with the deferred-emission
machinery moved here from the dropped patch, and re-measured.
7 Unchanged in substance, re-measured on the new baseline.
What I would most like reviewed
===============================
- Patch 5 reverses a deliberate decision made in d3a2a1d803 on the
strength of an argument about the user-only invalidation paths, plus a
gate on whether gdb can ever attach.
- Patch 6's un-poison in the main loop races a concurrent poison from
another thread. I believe the existing barrier around
icount_decr.u16.high covers it, but my testing was single-threaded user
mode.
- Patch 4 treats cpu flags, cflags and cs_base as translation-time
constants in its guards, reads a jump cache entry without qatomic_read(),
and leaves one_insn_per_tb and -d nochain toggles visible only at the
next non-inline exit.
- Patch 7 only examines the immediately preceding op, refuses any access
with a slow path (so user-only, and no alignment check), and leaves the
i128 pairs alone.
- Patch 2's 1 MiB per vCPU is easy to justify for a single-vCPU
linux-user process and less obvious for system emulation with many
vCPUs. It may want to be sized per target or made tunable rather than
raised unconditionally.
Patches 4 and 7 are wired up for alpha and x86_64 respectively; everything
else is target-independent, and no other backend changes behavior or needs
touching.
v2: https://lore.kernel.org/qemu-devel/[email protected]/
Matt Turner (7):
accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
accel/tcg: enlarge the TB jump cache to 64K entries
accel/tcg: skip the can_do_io stores in user-only builds
RFC: tcg: probe the TB jump cache inline instead of calling a helper
RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
RFC: accel/tcg: poison the jump cache instead of polling for indirect
exits
RFC: tcg: fold a guest displacement into the host addressing mode
accel/tcg/cpu-exec-common.c | 33 +++-
accel/tcg/cpu-exec.c | 122 +++++++++++++++
accel/tcg/internal-common.h | 13 +-
accel/tcg/tb-jmp-cache.h | 2 +-
accel/tcg/tcg-accel-ops.c | 1 +
accel/tcg/tcg-all.c | 1 +
accel/tcg/translator.c | 94 +++++++++++-
cpu-common.c | 11 ++
cpu-target.c | 3 +
gdbstub/user.c | 14 ++
include/gdbstub/user.h | 11 ++
include/hw/core/cpu.h | 11 ++
include/system/tcg.h | 21 +++
include/tcg/tcg-op-common.h | 16 +-
include/tcg/tcg-op.h | 12 ++
include/tcg/tcg-opc.h | 9 +-
include/tcg/tcg.h | 2 +
stubs/meson.build | 1 +
stubs/tcg-cflags.c | 20 +++
target/alpha/translate.c | 4 +-
target/arm/tcg/translate-a64.c | 4 +-
target/arm/tcg/translate.c | 10 +-
target/avr/translate.c | 4 +-
target/hexagon/translate.c | 4 +-
target/hppa/translate.c | 6 +-
target/i386/tcg/translate.c | 2 +-
.../tcg/insn_trans/trans_branch.c.inc | 2 +-
target/loongarch/tcg/translate.c | 4 +-
target/m68k/translate.c | 2 +-
target/microblaze/translate.c | 4 +-
target/mips/tcg/nanomips_translate.c.inc | 2 +-
target/mips/tcg/translate.c | 6 +-
target/or1k/translate.c | 4 +-
target/ppc/translate.c | 4 +-
target/riscv/tcg/insn_trans/trans_rvzce.c.inc | 4 +-
target/riscv/tcg/translate.c | 2 +-
target/rx/translate.c | 4 +-
target/s390x/tcg/translate.c | 5 +-
target/sh4/translate.c | 4 +-
target/sparc/translate.c | 4 +-
target/tricore/translate.c | 4 +-
tcg/tcg-op-ldst.c | 3 +-
tcg/tcg-op.c | 101 +++++++++++-
tcg/tcg.c | 86 ++++++++++-
tcg/x86_64/tcg-target.c.inc | 61 ++++++++
tcg/x86_64/tcg-target.h | 3 +
tests/tcg/alpha/Makefile.target | 18 ++-
tests/tcg/alpha/gdbstub/xpage-bp.py | 34 +++++
tests/tcg/alpha/test-indirect-irq.c | 55 +++++++
tests/tcg/alpha/test-xpage-chain.c | 144 ++++++++++++++++++
util/log.c | 4 +
51 files changed, 932 insertions(+), 63 deletions(-)
create mode 100644 stubs/tcg-cflags.c
create mode 100644 tests/tcg/alpha/gdbstub/xpage-bp.py
create mode 100644 tests/tcg/alpha/test-indirect-irq.c
create mode 100644 tests/tcg/alpha/test-xpage-chain.c
--
2.54.0