[PATCH 2/3] target/arm: Fix RCsc ordering of store-release instructions

The Armv8 memory model has no RCpc store: every store-release in the
architecture, including the AArch64 STLR and STLPUR and the A32/T32
STL, STLB and STLH, carries RCsc semantics (Arm Architecture Reference
Manual, DDI 0487, section B2.3.8 "Acquire and release semantics").
The architecture therefore orders a Store-Release tagged RCsc before
any subsequent Load-Acquire tagged RCsc of the same shareability
domain, which is what forbids the [0,0] outcome of a Store Buffering
litmus test that pairs "STLR X=1; LDAR Y" on one PE with
"STLR Y=1; LDAR X" on another.

The translators only emit a barrier before the store, which orders the
memory operations preceding the store before the store itself (release
semantics).  Nothing prevents the store from being reordered with the
memory operations that follow it, so under MTTCG the host CPU may
execute the load-acquire that immediately follows a store-release
before the store has left its store buffer:

  * on an x86 host (TSO), store-load is the only reordering the
    hardware may perform, and it is exactly the ordering that is
    missing here;

  * on an aarch64 host, no dmb follows the store either.

The defect is easily observed with a bare-metal Store Buffering litmus
test on a 2 vCPU virt guest: "STLR X=1; LDAR Y" on CPU 0 and
"STLR Y=1; LDAR X" on CPU 1 produce the architecturally forbidden [0,0]
result in 6 to 7 out of 100000 iterations, while inserting a DMB ISH
between the store and the load on each CPU brings the number to 0,
which validates the harness.

Fix this by emitting a full barrier after the store as well, for
STLR/STLLR and STLPUR on the AArch64 side and STL/STLB/STLH on the
A32/T32 side.  The TCG_MO_ST_LD bit included in TCG_MO_ALL makes the
x86 backend emit a serializing lock-prefixed instruction after the
store, draining the store buffer before any subsequent load executes,
and the aarch64 backend emit dmb ish; the store therefore becomes
globally visible before any subsequent memory operation of the same
vCPU, and the RCsc store->load ordering is satisfied.  This is
stronger than the architecture strictly requires, since the barrier
also orders the store with respect to subsequent RCpc operations such
as LDAPR, but this matches how QEMU already implements FENCE-like
ordering and keeps the fix simple.  After the fix the litmus test no
longer produces the forbidden outcome.

The load-acquire side needs no change: the barrier emitted after an
RCsc LDAR already orders it with all subsequent memory operations, and
QEMU deliberately implements the RCpc LDAPR family as full load-acquire.

Signed-off-by: LIU Zhiwei <[email protected]>
---
 target/arm/tcg/translate-a64.c | 13 +++++++++++++
 target/arm/tcg/translate.c     |  6 ++++++
 2 files changed, 19 insertions(+)

diff --git a/target/arm/tcg/translate-a64.c b/target/arm/tcg/translate-a64.c
index 4f9a93950b..b5bf977910 100644
--- a/target/arm/tcg/translate-a64.c
+++ b/target/arm/tcg/translate-a64.c
@@ -3672,6 +3672,13 @@ static bool trans_STLR(DisasContext *s, arg_stlr *a)
                                 true, a->rn != 31, memop);
     do_gpr_st(s, cpu_reg(s, a->rt), clean_addr, memop, true, a->rt,
               iss_sf, a->lasr);
+    /*
+     * STLR is an RCsc store-release: the Armv8 architecture orders it
+     * before any subsequent RCsc load-acquire, which is what forbids the
+     * store-buffering litmus test.  The barrier above only orders
+     * previous accesses before the store.
+     */
+    tcg_gen_mb(TCG_MO_ALL | TCG_BAR_SC);
     return true;
 }
 
@@ -4415,6 +4422,12 @@ static bool trans_STLR_i(DisasContext *s, arg_ldapr_stlr_i *a)
     /* Store-Release semantics */
     tcg_gen_mb(TCG_MO_ALL | TCG_BAR_STRL);
     do_gpr_st(s, cpu_reg(s, a->rt), clean_addr, mop, true, a->rt, iss_sf, true);
+    /*
+     * Like STLR, STLPUR is an RCsc store-release and must be ordered
+     * before any subsequent RCsc load-acquire, so it needs a barrier
+     * after the store as well.
+     */
+    tcg_gen_mb(TCG_MO_ALL | TCG_BAR_SC);
     return true;
 }
 
diff --git a/target/arm/tcg/translate.c b/target/arm/tcg/translate.c
index 1770428d3c..bddac41bf9 100644
--- a/target/arm/tcg/translate.c
+++ b/target/arm/tcg/translate.c
@@ -4259,6 +4259,12 @@ static bool op_stl(DisasContext *s, arg_STL *a, MemOp mop)
     tmp = load_reg(s, a->rt);
     tcg_gen_mb(TCG_MO_ALL | TCG_BAR_STRL);
     gen_aa32_st_i32(s, tmp, addr, get_mem_index(s), mop | MO_ALIGN);
+    /*
+     * STL has the same RCsc store-release semantics as the AArch64 STLR
+     * and must be ordered before a subsequent RCsc load-acquire (LDA).
+     * The barrier above only orders previous accesses before the store.
+     */
+    tcg_gen_mb(TCG_MO_ALL | TCG_BAR_SC);
     disas_set_da_iss(s, mop, a->rt | ISSIsAcqRel | ISSIsWrite);
 
     return true;
-- 
2.43.0