Re: [RFC] aarch64: Split selected LDPs to improve store forwarding

On Mon, Aug 17, 2026 at 07:42:46PM +0800, huzife wrote:
> Yes, the benefit of this optimization is microarchitecture-dependent.  We
> have observed a benefit on Kunpeng's hip12 machine.
> 
> I constructed a small assembly benchmark to compare an LDP with two
> LDRs when a preceding 64-bit STR overlaps the low 64-bit half of the
> 128-bit load window.  The core sequence is essentially:
> 
>         str     x0, [base, #16]
>         ldp     x0, x1, [base, #16]
> 
> compared with:
> 
>         str     x0, [base, #16]
>         ldr     x0, [base, #16]
>         ldr     x1, [base, #24]
> 
> The value loaded into x0 is used by the STR in the next iteration, forming
> a dependency chain that makes the store-to-load forwarding latency
> visible.  On Hip12, the two-LDR version is 130% faster than the LDP
> version.

This benchmark looks unrealistic, because I'd expect the compiler to apply
store forwarding itself unless there's some more complicated code in between.
In the included tests you're using volatile asm with memory clobbers to prevent
the forwarding.  What happens to prevent compiler store forwarding in the
real-world workloads this affects?

Alice


> 
> I agree that rejecting the pair before it is formed is cleaner than
> forming an LDP and splitting it again in a later pass.  We are currently
> reworking the implementation in this direction: the AArch64 LDP
> pair-fusion pass will recognize this specific store/load overlap pattern
> and reject the LDP fusion, leaving the two scalar loads unchanged.  The
> check will need to apply to both the early and late pair-fusion passes.
> This heuristic will be disabled by default and enabled automatically
> through the target tuning policy for affected  micro-arch.
> 
> Thanks,
> huzife
> 
> Alex Coplan <[email protected]> 于2026年8月17日周一 17:20写道：
> 
> > On 16/08/2026 20:51, Andrea Pinski wrote:
> > > On Sun, Aug 16, 2026 at 8:17 PM huzife <[email protected]> wrote:
> > > >
> > > > A load pair that partially overlaps a recent scalar store can prevent
> > the
> > > > load from using store-to-load forwarding.  Splitting the pair lets the
> > > > overlapping half use the forwarding path.
> > > >
> > > > Add a late AArch64 RTL pass that handles non-writeback SI and DI GPR
> > load
> > > > pairs.  Search a bounded part of the containing basic block for a
> > scalar
> > > > store to either half, and replace the pair only when both scalar loads
> > are
> > > > recognized.  Reject volatile, frame-related and exception-sensitive
> > forms,
> > > > and disable the transformation for size optimization.  Keep the pass
> > off by
> > > > default behind a target parameter while its profitability is evaluated.
> > >
> > >
> > > I am not sure we want this.  Is this dependent on the micro-arch? Can
> > > you expand on which micro-arch where this helps out?
> > > Also shouldn't instead of a separate pass which undoes what an earlier
> > > pass. Won't it be better to improve the heuristics of ldp_fusion pass
> > > to reject this from happening in the first place for these cases?
> >
> > I agree with everything Andrea said here, it would be good to know which
> > u-arches are expected to benefit, and potentially which workloads show a
> > benefit with this transformation.
> >
> > If we do decide to to do this, I agree that it would be better to try
> > and prevent the pairs from being formed in the first place (i.e. in
> > pair-fusion) if possible.
> >
> > Thanks,
> > Alex