[PATCH v3 4/7] RFC: tcg: probe the TB jump cache inline instead of calling a helper
Matt Turner <[email protected]>
| Newsgroups | org.nongnu.qemu-devel |
|---|---|
| Message-ID | <[email protected]> |
Every indirect branch that cannot use goto_tb ends in
tcg_gen_lookup_and_goto_ptr(), which calls helper_lookup_tb_ptr(). For an
emulated compiler that is 8.4 billion helper calls in a single translation
unit: 24.6% of all TB exits take this path, because jsr/ret/jmp have a
register destination and because goto_tb is restricted to same-page
targets.
The helper itself is already tight, but each call pays for a call frame,
the can_do_io store, the get_tb_cpu_state() indirect call through
TCGCPUOps, curr_cflags(), and a breakpoint check, before it gets to the
jump cache probe that almost always hits (95.8% for this workload).
Emit the probe inline instead. The destination PC is already in a TCG
temp, and the flags, cflags and cs_base the destination must match are
constants at translation time, so the fast path is a hash, four guarded
loads and a goto_ptr. Only a miss calls the helper, which still owns
filling the cache.
tcg_gen_lookup_and_goto_ptr() therefore takes the destination PC and the
TB being generated, and decides for itself whether to emit the probe or
the old helper call; there is no second entry point for targets that opt
in. A target that cannot name its destination in a single temp passes
NULL and gets the helper. Since the probe hashes and compares the PC as
one 64-bit value, a 32-bit guest PC also falls back.
The PC a target passes must be exactly what get_tb_cpu_state() reports for
the destination, which is the whole of the contract. alpha, loongarch,
mips, ppc and s390x pass their PC register, whose value is that pc by
construction. The rest pass NULL for now: avr's TB pc is the word address
doubled, i386's is eip before segmentation, riscv masks it to 32 bits when
xl is MXL_RV32, hppa derives it from the IAQ, hexagon adjusts it inside a
hardware loop, and sparc puts npc in cs_base so the guard could not hit
anyway. Each of those is a one-line change for whoever wants to measure it.
Two details matter for the generated code. The flags and cflags guards are
folded into a single aligned 64-bit load and compare, since the fields are
adjacent. And each path emits its own goto_ptr rather than branching to a
shared one: a temp live across the label is spilled and reloaded on every
dispatch, which cost 6.3% on its own.
The probe cannot check everything the helper checks, and the one that
matters is breakpoints. check_for_breakpoints() raises EXCP_DEBUG on an
exact pc match and selects CF_BP_PAGE cflags for the rest of the page, and
setting a breakpoint deliberately invalidates no TB, so a block translated
before the breakpoint was set is still sitting in the jump cache. Rather
than pay for a breakpoint test on the fast path, give the probe its own
base pointer, tb_jmp_cache_probe, that nothing else reads, and point it at
a page of zeroes while any breakpoint is set. Every entry the probe finds
then has a NULL tb, so every dispatch misses into the helper and the old
behaviour is restored exactly. cpu_breakpoint_insert() poisons the pointer,
so the poison takes effect at the next dispatch rather than whenever that
vCPU next reaches its main loop, which matters because a vCPU chaining
indirectly need never reach it. The main loop puts the pointer back once the
last breakpoint is gone; that is a load and a compare per block dispatched
from the main loop, and nothing at all in generated code.
The flags and cflags constants are safe against the other things that can
change them. CF_PARALLEL is only ever set by begin_parallel_context(),
which flushes first, so no block predating it survives to dispatch. gdb
single-step is only turned on with the CPU stopped, and a block translated
without CF_SINGLE_STEP can only be re-entered through tb_lookup(), which
from then on demands the new cflags -- so a stale-cflags block is never the
one running. What is left is one_insn_per_tb and -d nochain, which the
monitor can toggle under a running vCPU without a flush; see below.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO
build, on top of the preceding three patches:
before: 1,402,667,803,616 instructions
after: 916,415,123,244 instructions -34.67%
before: 115.56s wall clock
after: 85.59s wall clock -25.94%
The gap between the two is the point at which this stops being a
straight-line win: the helper call was highly predictable work that the
host pipelined well, so removing it retires far fewer instructions than it
saves time. IPC falls from 2.48 to 2.17 across this patch for that reason.
Despite emitting more code, this also reduces instruction cache pressure,
because a dispatch no longer jumps into qemu's .text and evicts translated
code:
before: 11,735,141,703 L1-icache-load-misses
after: 7,154,863,292 L1-icache-load-misses -39.0%
The mechanism is visible directly in a profile: helper_lookup_tb_ptr()
falls from 31.01% of samples to 0.35%, and qemu's own .text falls from
38.8% to 5.3%, with the balance moving into generated code.
Combined with the three preceding patches, against an unmodified LTO
build, 1,646,994,254,249 instructions fall to 916,415,123,244, or -44.36%.
The emulated compiler produces byte-identical output throughout.
Open issues, hence RFC:
- one_insn_per_tb and CPU_LOG_TB_NOCHAIN can be toggled from the monitor
while a vCPU is inside a block that was translated without them. The
block keeps dispatching inline with the old cflags until it exits for
some other reason. Poisoning the probe from
tcg_update_all_curr_cflags() would close it.
- The jump cache entry is read without qatomic_read(); entries are
invalidated concurrently by setting tb to NULL.
- Only alpha has been measured. The other four targets that pass a PC are
built and boot-tested only.
v3: Fold the fast path into tcg_gen_lookup_and_goto_ptr() instead of
adding tcg_gen_lookup_and_goto_ptr_inline() beside it (Richard). It
now takes the destination PC and the TB unconditionally, from all 38
call sites, and picks the probe or the helper itself. Translators
built for both values of TARGET_LONG_BITS -- arm, s390x, microblaze --
cannot include tcg-op.h, so the common entry point takes a TCGTemp and
reads the width from it, and tcg-op.h wraps that for everyone else;
this is the same split as tcg_gen_qemu_ld_*_chk().
Compare cs_base too. v2 listed this as an open issue, and closing it
is what lets the choice be made generically rather than per target: a
target that uses cs_base would otherwise have been enabled silently by
a decision keyed on PC width alone. It costs a load and a compare on
the fast path, and the numbers above were measured with it in place.
Audited which targets may pass a real PC, the contract being that it
is exactly what get_tb_cpu_state() reports for the destination. Five
do; the rest pass NULL and keep the helper call, sparc among them
because it puts npc in cs_base and so could essentially never hit.
Poison the probe from cpu_breakpoint_insert() rather than only from
the poisoned CPU's own main loop. gdb inserts a breakpoint into every
CPU (tcg_insert_gdbstub_breakpoint()), and a thread already inside
generated code, dispatching indirectly, need never return to the main
loop -- so it would keep dispatching inline and run past a breakpoint
another thread had just set. Upstream has no such window: its
helper_lookup_tb_ptr() sees the new breakpoint at the next indirect
branch. The un-poison in tcg_cpu_sync_jmp_cache() now re-checks after
its store, with a barrier, so that it loses the race with a concurrent
insert in the safe direction.
Signed-off-by: Matt Turner <[email protected]>
---
accel/tcg/cpu-exec.c | 102 ++++++++++++++++++
accel/tcg/internal-common.h | 2 +
cpu-common.c | 11 ++
include/hw/core/cpu.h | 9 ++
include/system/tcg.h | 9 ++
include/tcg/tcg-op-common.h | 16 ++-
include/tcg/tcg-op.h | 12 +++
stubs/tcg-cflags.c | 8 +-
target/alpha/translate.c | 4 +-
target/arm/tcg/translate-a64.c | 4 +-
target/arm/tcg/translate.c | 10 +-
target/avr/translate.c | 4 +-
target/hexagon/translate.c | 4 +-
target/hppa/translate.c | 6 +-
target/i386/tcg/translate.c | 2 +-
.../tcg/insn_trans/trans_branch.c.inc | 2 +-
target/loongarch/tcg/translate.c | 4 +-
target/m68k/translate.c | 2 +-
target/microblaze/translate.c | 4 +-
target/mips/tcg/nanomips_translate.c.inc | 2 +-
target/mips/tcg/translate.c | 6 +-
target/or1k/translate.c | 4 +-
target/ppc/translate.c | 4 +-
target/riscv/tcg/insn_trans/trans_rvzce.c.inc | 4 +-
target/riscv/tcg/translate.c | 2 +-
target/rx/translate.c | 4 +-
target/s390x/tcg/translate.c | 5 +-
target/sh4/translate.c | 4 +-
target/sparc/translate.c | 4 +-
target/tricore/translate.c | 4 +-
tcg/tcg-op.c | 92 +++++++++++++++-
31 files changed, 300 insertions(+), 50 deletions(-)
diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c
index 148e0f583e..e546f717e8 100644
--- ./accel/tcg/cpu-exec.c
+++ ./accel/tcg/cpu-exec.c
@@ -752,6 +752,99 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret)
return false;
}
+/*
+ * The inline jump cache probe reads cpu->tb_jmp_cache_probe and takes the
+ * slow path when the entry it finds has a NULL tb. Pointing the probe at a
+ * region that is all zeroes therefore forces every indirect dispatch into
+ * helper_lookup_tb_ptr(), which does the full lookup the inline probe only
+ * approximates. The real jump cache is untouched, so no contents are lost
+ * and recovery is a single store.
+ *
+ * Only ever read from, and only the tb field of one entry per dispatch, so
+ * one shared zero-filled cache is enough for every CPU.
+ */
+static const CPUJumpCache *tb_jmp_cache_poison(void)
+{
+ static CPUJumpCache *poison;
+
+ if (unlikely(poison == NULL)) {
+ /* Raced allocations are harmless: both are all zeroes. */
+ qatomic_cmpxchg(&poison, NULL, g_new0(CPUJumpCache, 1));
+ }
+ return poison;
+}
+
+/*
+ * Whether the generated code may dispatch to the next block by itself.
+ *
+ * The inline probe matches on the destination pc and on the flags and
+ * cflags the dispatching block was translated with. It does not consult
+ * cpu->breakpoints, so it must not run while one is set: setting a
+ * breakpoint deliberately invalidates nothing, and check_for_breakpoints()
+ * both raises EXCP_DEBUG on an exact match and picks CF_BP_PAGE cflags for
+ * the rest of the page. A block translated before the breakpoint was set is
+ * therefore still in the jump cache, and dispatching to it inline would step
+ * straight over the breakpoint.
+ */
+static bool tcg_cpu_may_dispatch(CPUState *cpu)
+{
+ return QTAILQ_EMPTY(&cpu->breakpoints);
+}
+
+/*
+ * Poison @cpu's probe, from any thread. Called when a breakpoint is
+ * inserted, which is what makes the poison take effect at the dispatch
+ * after the insert rather than whenever @cpu next reaches its main loop:
+ * a vCPU chaining indirectly need never reach it, and would run past a
+ * breakpoint another thread had just set.
+ *
+ * A plain store is enough. The value only ever costs a slow path that is
+ * correct on its own, and the generated code re-reads the base on every
+ * dispatch. Un-poisoning is tcg_cpu_sync_jmp_cache()'s job.
+ */
+void tcg_cpu_poison_jmp_cache(CPUState *cpu)
+{
+ if (qatomic_read(&cpu->tb_jmp_cache_probe) != NULL) {
+ qatomic_set(&cpu->tb_jmp_cache_probe,
+ (CPUJumpCache *)tb_jmp_cache_poison());
+ }
+}
+
+/*
+ * Called from the main loop, which is the only context that can establish
+ * that no reason to be poisoned is left. Cheap enough to call every time
+ * round: the common case is a load, a compare and no store at all.
+ */
+void tcg_cpu_sync_jmp_cache(CPUState *cpu)
+{
+ CPUJumpCache *want;
+
+ if (qatomic_read(&cpu->tb_jmp_cache_probe) == NULL) {
+ return; /* not realized, or already unrealized */
+ }
+
+ want = tcg_cpu_may_dispatch(cpu)
+ ? cpu->tb_jmp_cache
+ : (CPUJumpCache *)tb_jmp_cache_poison();
+
+ if (qatomic_read(&cpu->tb_jmp_cache_probe) != want) {
+ qatomic_set(&cpu->tb_jmp_cache_probe, want);
+
+ /*
+ * Un-poisoning races a concurrent tcg_cpu_poison_jmp_cache(): the
+ * reason may have appeared after tcg_cpu_may_dispatch() read it and
+ * the poison may have landed before the store above. Order the
+ * store against a re-read, and lose the race in the safe direction.
+ */
+ if (want == cpu->tb_jmp_cache) {
+ smp_mb();
+ if (!tcg_cpu_may_dispatch(cpu)) {
+ tcg_cpu_poison_jmp_cache(cpu);
+ }
+ }
+ }
+}
+
void tcg_kick_vcpu_thread(CPUState *cpu)
{
/*
@@ -964,6 +1057,13 @@ cpu_exec_loop(CPUState *cpu, SyncClocks *sc)
break;
}
+ /*
+ * Reaching here means the main loop has just re-evaluated
+ * everything the inline probe assumes, so this is where the
+ * probe is allowed to come back after a poison.
+ */
+ tcg_cpu_sync_jmp_cache(cpu);
+
tb = tb_lookup(cpu, s);
if (tb == NULL) {
CPUJumpCache *jc;
@@ -1072,6 +1172,7 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp)
tcg_update_cflags(cpu);
cpu->tb_jmp_cache = g_new0(CPUJumpCache, 1);
+ qatomic_set(&cpu->tb_jmp_cache_probe, cpu->tb_jmp_cache);
tlb_init(cpu);
#ifndef CONFIG_USER_ONLY
tcg_iommu_init_notifier_list(cpu);
@@ -1089,5 +1190,6 @@ void tcg_exec_unrealizefn(CPUState *cpu)
#endif /* !CONFIG_USER_ONLY */
tlb_destroy(cpu);
+ qatomic_set(&cpu->tb_jmp_cache_probe, NULL);
g_free_rcu(cpu->tb_jmp_cache, rcu);
}
diff --git ./accel/tcg/internal-common.h ./accel/tcg/internal-common.h
index 853d1b51ee..9d1f6712d6 100644
--- ./accel/tcg/internal-common.h
+++ ./accel/tcg/internal-common.h
@@ -144,6 +144,8 @@ void page_table_config_init(void);
G_NORETURN void cpu_io_recompile(CPUState *cpu, uintptr_t retaddr);
#endif /* CONFIG_USER_ONLY */
+void tcg_cpu_sync_jmp_cache(CPUState *cpu);
+
void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr);
void tb_set_jmp_target(TranslationBlock *tb, int n, uintptr_t addr);
diff --git ./cpu-common.c ./cpu-common.c
index adb76b3a78..3aed0156e6 100644
--- ./cpu-common.c
+++ ./cpu-common.c
@@ -22,6 +22,7 @@
#include "exec/cpu-common.h"
#include "hw/core/cpu.h"
#include "qemu/lockable.h"
+#include "system/tcg.h"
#include "trace/trace-root.h"
QemuMutex qemu_cpu_list_lock;
@@ -429,6 +430,16 @@ int cpu_breakpoint_insert(CPUState *cpu, vaddr pc, int flags,
*breakpoint = bp;
}
+ /*
+ * Nothing is invalidated here, so blocks translated before this point
+ * are still live and still dispatch to each other without consulting
+ * cpu->breakpoints. Stop the ones that can: a TCG vCPU dispatching
+ * inline reads a base pointer that this poisons, so the next dispatch
+ * takes the slow path and sees the new breakpoint. @cpu may be another
+ * thread, and may be running.
+ */
+ tcg_cpu_poison_jmp_cache(cpu);
+
trace_breakpoint_insert(cpu->cpu_index, pc, flags);
return 0;
}
diff --git ./include/hw/core/cpu.h ./include/hw/core/cpu.h
index 81af7b9ee1..bd2cdd2a0b 100644
--- ./include/hw/core/cpu.h
+++ ./include/hw/core/cpu.h
@@ -519,6 +519,15 @@ struct CPUState {
MemoryRegion *memory;
struct CPUJumpCache *tb_jmp_cache;
+ /*
+ * @tb_jmp_cache_probe: base the inline jump cache probe reads.
+ *
+ * Normally @tb_jmp_cache. Pointed at a shared page of zeroes to force
+ * every inline dispatch to miss and fall back to helper_lookup_tb_ptr();
+ * see tcg_cpu_sync_jmp_cache(). NULL before tcg_exec_realizefn() and
+ * after tcg_exec_unrealizefn().
+ */
+ struct CPUJumpCache *tb_jmp_cache_probe;
GArray *gdb_regs;
int gdb_num_regs;
diff --git ./include/system/tcg.h ./include/system/tcg.h
index 2c2dbc753b..bf05db1329 100644
--- ./include/system/tcg.h
+++ ./include/system/tcg.h
@@ -29,6 +29,15 @@ extern bool tcg_allowed;
void tcg_update_cflags(CPUState *cpu);
void tcg_update_all_cflags(void);
+/*
+ * Force @cpu's generated code back into the slow dispatch path, which
+ * re-checks everything the inline jump cache probe assumes. Safe to call
+ * from any thread, and a no-op for a CPU that is not running TCG. Call
+ * whenever something the probe cannot see changes under a running vCPU;
+ * the main loop undoes it once the reason is gone.
+ */
+void tcg_cpu_poison_jmp_cache(CPUState *cpu);
+
/**
* qemu_tcg_mttcg_enabled:
* Check whether we are running MultiThread TCG or not.
diff --git ./include/tcg/tcg-op-common.h ./include/tcg/tcg-op-common.h
index 9b321f959c..ba580c7fb7 100644
--- ./include/tcg/tcg-op-common.h
+++ ./include/tcg/tcg-op-common.h
@@ -75,15 +75,25 @@ void tcg_gen_exit_tb(const TranslationBlock *tb, unsigned idx);
void tcg_gen_goto_tb(unsigned idx);
/**
- * tcg_gen_lookup_and_goto_ptr() - look up the current TB, jump to it if valid
- * @addr: Guest address of the target TB
+ * tcg_gen_lookup_and_goto_ptr() - look up the destination TB, jump to it
+ * @pc: temp holding the destination guest PC, or NULL
+ * @tb: the translation block being generated
*
* If the TB is not valid, jump to the epilogue.
*
+ * The lookup is normally a call to helper_lookup_tb_ptr(). If @pc is
+ * non-NULL and the destination can be keyed on it directly, the jump cache
+ * is probed inline instead and only a miss reaches the helper. @pc must
+ * then hold exactly the value get_tb_cpu_state() reports as the pc for the
+ * destination; a target whose pc is derived (avr's word address, i386's
+ * eip before segmentation) must pass NULL. The destination is required to
+ * match @tb's flags, cflags and cs_base, which is what makes them
+ * constants in the probe.
+ *
* This operation is optional. If the TCG backend does not implement goto_ptr,
* this op is equivalent to calling tcg_gen_exit_tb() with 0 as the argument.
*/
-void tcg_gen_lookup_and_goto_ptr(void);
+void tcg_gen_lookup_and_goto_ptr_tmp(TCGTemp *pc, const TranslationBlock *tb);
void tcg_gen_plugin_cb(unsigned from);
void tcg_gen_plugin_mem_cb(TCGv_i64 addr, unsigned meminfo);
diff --git ./include/tcg/tcg-op.h ./include/tcg/tcg-op.h
index 3721164236..b6c7c6fea2 100644
--- ./include/tcg/tcg-op.h
+++ ./include/tcg/tcg-op.h
@@ -49,6 +49,18 @@ typedef TCGv_i64 TCGv;
#error Unhandled TARGET_LONG_BITS value
#endif
+/*
+ * See tcg_gen_lookup_and_goto_ptr_tmp(). @pc may be NULL, for a target
+ * whose guest PC is not directly the key the jump cache is indexed by.
+ * A translator that is built for more than one value of TARGET_LONG_BITS,
+ * and so cannot include this header, calls the _tmp() form directly.
+ */
+static inline void
+tcg_gen_lookup_and_goto_ptr(TCGv pc, const TranslationBlock *tb)
+{
+ tcg_gen_lookup_and_goto_ptr_tmp(pc ? tcgv_tl_temp(pc) : NULL, tb);
+}
+
#if TARGET_LONG_BITS == 64
#define tcg_gen_movi_tl tcg_gen_movi_i64
#define tcg_gen_mov_tl tcg_gen_mov_i64
diff --git ./stubs/tcg-cflags.c ./stubs/tcg-cflags.c
index cb278e94aa..4ac8a82e92 100644
--- ./stubs/tcg-cflags.c
+++ ./stubs/tcg-cflags.c
@@ -1,6 +1,6 @@
/*
- * Stub for tcg_update_all_cflags(), for binaries that link util/log.c
- * or cpu-target.c but not TCG.
+ * Stubs for the TCG entry points in system/tcg.h, for binaries that link
+ * util/log.c, cpu-target.c or cpu-common.c but not TCG.
*
* SPDX-License-Identifier: GPL-2.0-or-later
*/
@@ -14,3 +14,7 @@ void tcg_update_cflags(CPUState *cpu)
void tcg_update_all_cflags(void)
{
}
+
+void tcg_cpu_poison_jmp_cache(CPUState *cpu)
+{
+}
diff --git ./target/alpha/translate.c ./target/alpha/translate.c
index c66e3f9c14..822f5cc120 100644
--- ./target/alpha/translate.c
+++ ./target/alpha/translate.c
@@ -449,7 +449,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx, int32_t disp)
tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx);
} else {
gen_pc_disp(ctx, cpu_pc, disp);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
}
}
@@ -2917,7 +2917,7 @@ static void alpha_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
gen_pc_disp(ctx, cpu_pc, 0);
/* FALLTHRU */
case DISAS_PC_UPDATED:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
break;
case DISAS_PC_UPDATED_NOCHAIN:
tcg_gen_exit_tb(NULL, 0);
diff --git ./target/arm/tcg/translate-a64.c ./target/arm/tcg/translate-a64.c
index 4f9a93950b..d1dd33a1af 100644
--- ./target/arm/tcg/translate-a64.c
+++ ./target/arm/tcg/translate-a64.c
@@ -562,7 +562,7 @@ static void gen_goto_tb(DisasContext *s, unsigned tb_slot_idx, int64_t diff)
if (s->ss_active) {
gen_step_complete_exception(s);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, s->base.tb);
s->base.is_jmp = DISAS_NORETURN;
}
}
@@ -11250,7 +11250,7 @@ static void aarch64_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
gen_a64_update_pc(dc, 4);
/* fall through */
case DISAS_JUMP:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
break;
case DISAS_NORETURN:
case DISAS_SWI:
diff --git ./target/arm/tcg/translate.c ./target/arm/tcg/translate.c
index c866148383..ca701b9cbc 100644
--- ./target/arm/tcg/translate.c
+++ ./target/arm/tcg/translate.c
@@ -1306,9 +1306,9 @@ void write_neon_element64(TCGv_i64 src, int reg, int ele, MemOp memop)
}
}
-static void gen_goto_ptr(void)
+static void gen_goto_ptr(DisasContext *s)
{
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(NULL, s->base.tb);
}
/* This will end the TB but doesn't guarantee we'll return to
@@ -1336,7 +1336,7 @@ static void gen_goto_tb(DisasContext *s, unsigned tb_slot_idx, int64_t diff)
tcg_gen_exit_tb(s->base.tb, tb_slot_idx);
} else {
gen_update_pc(s, diff);
- gen_goto_ptr();
+ gen_goto_ptr(s);
}
s->base.is_jmp = DISAS_NORETURN;
}
@@ -1373,7 +1373,7 @@ static void gen_jmp_tb(DisasContext *s, int64_t diff, int tbno)
* and don't chain to another TB.
*/
gen_update_pc(s, diff);
- gen_goto_ptr();
+ gen_goto_ptr(s);
s->base.is_jmp = DISAS_NORETURN;
break;
default:
@@ -6858,7 +6858,7 @@ static void arm_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
gen_update_pc(dc, curr_insn_len(dc));
/* fall through */
case DISAS_JUMP:
- gen_goto_ptr();
+ gen_goto_ptr(dc);
break;
case DISAS_UPDATE_EXIT:
gen_update_pc(dc, curr_insn_len(dc));
diff --git ./target/avr/translate.c ./target/avr/translate.c
index 3c57606097..8f2e0baa67 100644
--- ./target/avr/translate.c
+++ ./target/avr/translate.c
@@ -992,7 +992,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx,
tcg_gen_exit_tb(tb, tb_slot_idx);
} else {
tcg_gen_movi_i32(cpu_pc, dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
ctx->base.is_jmp = DISAS_NORETURN;
}
@@ -2778,7 +2778,7 @@ static void avr_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
/* fall through */
case DISAS_LOOKUP:
if (!force_exit) {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
break;
}
/* fall through */
diff --git ./target/hexagon/translate.c ./target/hexagon/translate.c
index 06a8159d28..cc230b08d1 100644
--- ./target/hexagon/translate.c
+++ ./target/hexagon/translate.c
@@ -181,7 +181,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx,
if (move_to_pc) {
tcg_gen_movi_tl(hex_gpr[HEX_REG_PC], dest);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
}
@@ -218,7 +218,7 @@ static void gen_end_tb(DisasContext *ctx)
gen_set_label(skip);
gen_goto_tb(ctx, 1, ctx->next_PC, false);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
ctx->base.is_jmp = DISAS_NORETURN;
diff --git ./target/hppa/translate.c ./target/hppa/translate.c
index 002189ddfb..cf8f1a2c13 100644
--- ./target/hppa/translate.c
+++ ./target/hppa/translate.c
@@ -816,7 +816,7 @@ static void gen_goto_tb(DisasContext *ctx, int which,
tcg_gen_goto_tb(which);
tcg_gen_exit_tb(ctx->base.tb, which);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
}
@@ -2027,7 +2027,7 @@ static bool do_ibranch(DisasContext *ctx, unsigned link,
store_psw_xb(ctx, PSW_B);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
ctx->base.is_jmp = DISAS_NORETURN;
return nullify_end(ctx);
}
@@ -4838,7 +4838,7 @@ static void hppa_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
}
/* FALLTHRU */
case DISAS_IAQ_N_UPDATED:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
break;
case DISAS_EXIT:
tcg_gen_exit_tb(NULL, 0);
diff --git ./target/i386/tcg/translate.c ./target/i386/tcg/translate.c
index 2115c5cd24..66a0ee3cdf 100644
--- ./target/i386/tcg/translate.c
+++ ./target/i386/tcg/translate.c
@@ -2005,7 +2005,7 @@ gen_eob(DisasContext *s, int mode)
} else if (mode == DISAS_JUMP &&
/* give irqs a chance to happen */
!inhibit_reset) {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, s->base.tb);
} else {
tcg_gen_exit_tb(NULL, 0);
}
diff --git ./target/loongarch/tcg/insn_trans/trans_branch.c.inc ./target/loongarch/tcg/insn_trans/trans_branch.c.inc
index da07778658..57d9d47353 100644
--- ./target/loongarch/tcg/insn_trans/trans_branch.c.inc
+++ ./target/loongarch/tcg/insn_trans/trans_branch.c.inc
@@ -27,7 +27,7 @@ static bool trans_jirl(DisasContext *ctx, arg_jirl *a)
tcg_gen_mov_tl(cpu_pc, addr);
tcg_gen_movi_tl(dest, make_address_pc(ctx, ctx->base.pc_next + 4));
gen_set_gpr(a->rd, dest, EXT_NONE);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
ctx->base.is_jmp = DISAS_NORETURN;
return true;
}
diff --git ./target/loongarch/tcg/translate.c ./target/loongarch/tcg/translate.c
index 124dce6269..a45a51852a 100644
--- ./target/loongarch/tcg/translate.c
+++ ./target/loongarch/tcg/translate.c
@@ -111,7 +111,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx, vaddr dest)
tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx);
} else {
tcg_gen_movi_tl(cpu_pc, dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
}
}
@@ -311,7 +311,7 @@ static void loongarch_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
switch (ctx->base.is_jmp) {
case DISAS_STOP:
tcg_gen_movi_tl(cpu_pc, ctx->base.pc_next);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
break;
case DISAS_TOO_MANY:
gen_goto_tb(ctx, 0, ctx->base.pc_next);
diff --git ./target/m68k/translate.c ./target/m68k/translate.c
index 138c89d3e5..73691bc0d1 100644
--- ./target/m68k/translate.c
+++ ./target/m68k/translate.c
@@ -6095,7 +6095,7 @@ static void m68k_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
if (dc->ss_active) {
gen_raise_exception_format2(dc, EXCP_TRACE, dc->pc_prev);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
}
break;
case DISAS_EXIT:
diff --git ./target/microblaze/translate.c ./target/microblaze/translate.c
index 8b219afb5d..851b372f8f 100644
--- ./target/microblaze/translate.c
+++ ./target/microblaze/translate.c
@@ -127,7 +127,7 @@ static void gen_goto_tb(DisasContext *dc, unsigned tb_slot_idx, vaddr dest)
tcg_gen_exit_tb(dc->base.tb, tb_slot_idx);
} else {
tcg_gen_movi_i32(cpu_pc, dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(NULL, dc->base.tb);
}
dc->base.is_jmp = DISAS_NORETURN;
}
@@ -1764,7 +1764,7 @@ static void mb_tr_tb_stop(DisasContextBase *dcb, CPUState *cs)
/* Indirect jump (or direct jump w/ goto_tb disabled) */
tcg_gen_mov_i32(cpu_pc, cpu_btarget);
tcg_gen_discard_i32(cpu_btarget);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(NULL, dc->base.tb);
return;
default:
diff --git ./target/mips/tcg/nanomips_translate.c.inc ./target/mips/tcg/nanomips_translate.c.inc
index 4b0b01ba37..007e29f9ac 100644
--- ./target/mips/tcg/nanomips_translate.c.inc
+++ ./target/mips/tcg/nanomips_translate.c.inc
@@ -2406,7 +2406,7 @@ static void gen_compute_nanomips_pbalrsc_branch(DisasContext *ctx, int rs,
/* unconditional branch to register */
tcg_gen_mov_tl(cpu_PC, btarget);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_PC, ctx->base.tb);
}
/* nanoMIPS Branches */
diff --git ./target/mips/tcg/translate.c ./target/mips/tcg/translate.c
index e3467d1525..73abfbb5d4 100644
--- ./target/mips/tcg/translate.c
+++ ./target/mips/tcg/translate.c
@@ -4374,7 +4374,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx,
tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx);
} else {
gen_save_pc(dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_PC, ctx->base.tb);
}
}
@@ -11014,7 +11014,7 @@ static void gen_branch(DisasContext *ctx, int insn_bytes)
} else {
tcg_gen_mov_tl(cpu_PC, btarget);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_PC, ctx->base.tb);
break;
default:
LOG_DISAS("unknown branch 0x%x\n", proc_hflags);
@@ -15244,7 +15244,7 @@ static void mips_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
switch (ctx->base.is_jmp) {
case DISAS_STOP:
gen_save_pc(ctx->base.pc_next);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_PC, ctx->base.tb);
break;
case DISAS_NEXT:
case DISAS_TOO_MANY:
diff --git ./target/or1k/translate.c ./target/or1k/translate.c
index eb4485312f..4907284a6d 100644
--- ./target/or1k/translate.c
+++ ./target/or1k/translate.c
@@ -1605,7 +1605,7 @@ static void openrisc_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
/* The jump destination is indirect/computed; use jmp_pc. */
tcg_gen_mov_i32(cpu_pc, jmp_pc);
tcg_gen_discard_i32(jmp_pc);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
break;
}
/* The jump destination is direct; use jmp_pc_imm.
@@ -1622,7 +1622,7 @@ static void openrisc_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
break;
}
tcg_gen_movi_i32(cpu_pc, jmp_dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
break;
case DISAS_EXIT:
diff --git ./target/ppc/translate.c ./target/ppc/translate.c
index 06ed2adf10..42924281b0 100644
--- ./target/ppc/translate.c
+++ ./target/ppc/translate.c
@@ -3664,7 +3664,7 @@ static void gen_lookup_and_goto_ptr(DisasContext *ctx)
pmu_count_insns(ctx);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_nip, ctx->base.tb);
}
}
@@ -6690,7 +6690,7 @@ static void ppc_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
pmu_count_insns(ctx);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_nip, ctx->base.tb);
break;
case DISAS_EXIT_UPDATE:
diff --git ./target/riscv/tcg/insn_trans/trans_rvzce.c.inc ./target/riscv/tcg/insn_trans/trans_rvzce.c.inc
index 71b4ca5473..3f1e7c039e 100644
--- ./target/riscv/tcg/insn_trans/trans_rvzce.c.inc
+++ ./target/riscv/tcg/insn_trans/trans_rvzce.c.inc
@@ -213,7 +213,7 @@ static bool gen_pop(DisasContext *ctx, arg_cmpp *a, bool ret, bool ret_val)
}
#endif
tcg_gen_mov_tl(cpu_pc, ret_addr);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
ctx->base.is_jmp = DISAS_NORETURN;
}
@@ -334,7 +334,7 @@ static bool trans_cm_jalt(DisasContext *ctx, arg_cm_jalt *a)
tcg_gen_mov_tl(cpu_pc, addr);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
ctx->base.is_jmp = DISAS_NORETURN;
return true;
}
diff --git ./target/riscv/tcg/translate.c ./target/riscv/tcg/translate.c
index 9684dbe752..8475ab43b4 100644
--- ./target/riscv/tcg/translate.c
+++ ./target/riscv/tcg/translate.c
@@ -287,7 +287,7 @@ static void lookup_and_goto_ptr(DisasContext *ctx)
gen_helper_itrigger_match(tcg_env);
}
#endif
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
static void exit_tb(DisasContext *ctx)
diff --git ./target/rx/translate.c ./target/rx/translate.c
index 132d495710..e5a9783d84 100644
--- ./target/rx/translate.c
+++ ./target/rx/translate.c
@@ -161,7 +161,7 @@ static void gen_goto_tb(DisasContext *dc, unsigned tb_slot_idx, vaddr dest)
tcg_gen_exit_tb(dc->base.tb, tb_slot_idx);
} else {
tcg_gen_movi_i32(cpu_pc, dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
}
dc->base.is_jmp = DISAS_NORETURN;
}
@@ -2242,7 +2242,7 @@ static void rx_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
gen_goto_tb(ctx, 0, dcbase->pc_next);
break;
case DISAS_JUMP:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
break;
case DISAS_UPDATE:
tcg_gen_movi_i32(cpu_pc, ctx->base.pc_next);
diff --git ./target/s390x/tcg/translate.c ./target/s390x/tcg/translate.c
index 1b6023168b..607c039419 100644
--- ./target/s390x/tcg/translate.c
+++ ./target/s390x/tcg/translate.c
@@ -1162,7 +1162,7 @@ static DisasJumpType help_branch(DisasContext *s, DisasCompare *c,
tcg_gen_goto_tb(0);
tcg_gen_exit_tb(s->base.tb, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(tcgv_i64_temp(psw_addr), s->base.tb);
}
gen_set_label(lab);
@@ -6477,7 +6477,8 @@ static void s390x_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
if (dc->exit_to_mainloop) {
tcg_gen_exit_tb(NULL, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(tcgv_i64_temp(psw_addr),
+ dc->base.tb);
}
break;
default:
diff --git ./target/sh4/translate.c ./target/sh4/translate.c
index 373950fd66..a4be456bd9 100644
--- ./target/sh4/translate.c
+++ ./target/sh4/translate.c
@@ -242,7 +242,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx, vaddr dest)
if (use_exit_tb(ctx)) {
tcg_gen_exit_tb(NULL, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
}
ctx->base.is_jmp = DISAS_NORETURN;
@@ -258,7 +258,7 @@ static void gen_jump(DisasContext * ctx)
if (use_exit_tb(ctx)) {
tcg_gen_exit_tb(NULL, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
ctx->base.is_jmp = DISAS_NORETURN;
} else {
diff --git ./target/sparc/translate.c ./target/sparc/translate.c
index 3156be6a94..2ae0a02c44 100644
--- ./target/sparc/translate.c
+++ ./target/sparc/translate.c
@@ -376,7 +376,7 @@ static void gen_goto_tb(DisasContext *s, unsigned tb_slot_idx,
/* jump to another page: we can use an indirect jump */
tcg_gen_movi_tl(cpu_pc, pc);
tcg_gen_movi_tl(cpu_npc, npc);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, s->base.tb);
}
}
@@ -5807,7 +5807,7 @@ static void sparc_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
tcg_gen_movi_tl(cpu_npc, dc->npc);
}
if (may_lookup) {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
} else {
tcg_gen_exit_tb(NULL, 0);
}
diff --git ./target/tricore/translate.c ./target/tricore/translate.c
index 8cd6b58f66..1d7f54f6df 100644
--- ./target/tricore/translate.c
+++ ./target/tricore/translate.c
@@ -2857,7 +2857,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_index, vaddr dest)
tcg_gen_exit_tb(ctx->base.tb, tb_slot_index);
} else {
gen_save_pc(dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
ctx->base.is_jmp = DISAS_NORETURN;
}
@@ -8478,7 +8478,7 @@ static void tricore_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
tcg_gen_exit_tb(NULL, 0);
break;
case DISAS_JUMP:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
break;
case DISAS_NORETURN:
break;
diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c
index 28d3b2a847..ce77541eab 100644
--- ./tcg/tcg-op.c
+++ ./tcg/tcg-op.c
@@ -28,6 +28,8 @@
#include "tcg/tcg-op-common.h"
#include "exec/translation-block.h"
#include "exec/plugin-gen.h"
+#include "hw/core/cpu.h"
+#include "../accel/tcg/tb-jmp-cache.h"
#include "tcg-internal.h"
#include "tcg-has.h"
@@ -2715,7 +2717,81 @@ void tcg_gen_goto_tb(unsigned idx)
tcg_gen_op1i(INDEX_op_goto_tb, 0, idx);
}
-void tcg_gen_lookup_and_goto_ptr(void)
+static void gen_jmp_cache_probe(TCGv_i64 pc, const TranslationBlock *tb)
+{
+ TCGv_ptr jc, ent, tbp, ptr;
+ TCGv_i64 h, tmp;
+ TCGLabel *slow;
+ uint64_t fpair;
+
+ QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) != 16);
+ QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, cflags) !=
+ offsetof(TranslationBlock, flags) + 4);
+
+ jc = tcg_temp_ebb_new_ptr();
+ ent = tcg_temp_ebb_new_ptr();
+ tbp = tcg_temp_ebb_new_ptr();
+ ptr = tcg_temp_ebb_new_ptr();
+ h = tcg_temp_ebb_new_i64();
+ tmp = tcg_temp_ebb_new_i64();
+ slow = gen_new_label();
+
+ /* h = tb_jmp_cache_hash_func(pc) * sizeof(array[0]) */
+ tcg_gen_shri_i64(h, pc, TB_JMP_CACHE_BITS);
+ tcg_gen_xor_i64(h, h, pc);
+ tcg_gen_andi_i64(h, h, TB_JMP_CACHE_SIZE - 1);
+ tcg_gen_shli_i64(h, h, 4);
+
+ /*
+ * Not cpu->tb_jmp_cache: the probe reads its own base so that the main
+ * loop can poison it, which is how conditions the probe cannot test for
+ * itself force every dispatch back into the helper. See
+ * tcg_cpu_sync_jmp_cache().
+ */
+ tcg_gen_ld_ptr(jc, tcg_env,
+ offsetof(CPUState, tb_jmp_cache_probe) - sizeof(CPUState));
+ tcg_gen_trunc_i64_ptr(ent, h);
+ tcg_gen_add_ptr(ent, jc, ent);
+
+ tcg_gen_ld_ptr(tbp, ent, offsetof(CPUJumpCache, array[0].tb));
+ tcg_gen_brcondi_ptr(TCG_COND_EQ, tbp, 0, slow);
+
+ tcg_gen_ld_i64(tmp, ent, offsetof(CPUJumpCache, array[0].pc));
+ tcg_gen_brcond_i64(TCG_COND_NE, tmp, pc, slow);
+
+ /*
+ * flags and cflags are adjacent uint32_t, so one aligned 64-bit load
+ * and compare covers both.
+ */
+#if HOST_BIG_ENDIAN
+ fpair = ((uint64_t)tb->flags << 32) | tb->cflags;
+#else
+ fpair = ((uint64_t)tb->cflags << 32) | tb->flags;
+#endif
+ tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, flags));
+ tcg_gen_brcondi_i64(TCG_COND_NE, tmp, fpair, slow);
+
+ /*
+ * The destination must have been translated with the same cs_base, which
+ * the pc alone does not imply on a target that uses it.
+ */
+ tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, cs_base));
+ tcg_gen_brcondi_i64(TCG_COND_NE, tmp, tb->cs_base, slow);
+
+ tcg_gen_ld_ptr(ptr, tbp, offsetof(TranslationBlock, tc.ptr));
+ tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
+
+ /*
+ * Emit a second goto_ptr rather than branching to a shared one: a temp
+ * live across the label would be spilled and reloaded on every dispatch.
+ */
+ gen_set_label(slow);
+ ptr = tcg_temp_ebb_new_ptr();
+ gen_helper_lookup_tb_ptr(ptr, tcg_env);
+ tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
+}
+
+void tcg_gen_lookup_and_goto_ptr_tmp(TCGTemp *pc, const TranslationBlock *tb)
{
TCGv_ptr ptr;
@@ -2724,7 +2800,21 @@ void tcg_gen_lookup_and_goto_ptr(void)
return;
}
+ /*
+ * No icount_decr poll is needed for this exit: the helper is called on
+ * every dispatch and returns to the main loop while an exit is pending.
+ */
plugin_gen_disable_mem_helpers();
+
+ /*
+ * The inline probe hashes and compares the pc as a single 64-bit value.
+ * A target with a 32-bit guest PC keeps the helper call.
+ */
+ if (pc && pc->type == TCG_TYPE_I64) {
+ gen_jmp_cache_probe(temp_tcgv_i64(pc), tb);
+ return;
+ }
+
ptr = tcg_temp_ebb_new_ptr();
gen_helper_lookup_tb_ptr(ptr, tcg_env);
tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
--
2.54.0