Re: [RFC] aarch64: Split selected LDPs to improve store forwarding

On Mon, Aug 17, 2026 at 8:22 PM Alice Carlotti <[email protected]> wrote:
>
> On Mon, Aug 17, 2026 at 07:42:46PM +0800, huzife wrote:
> > Yes, the benefit of this optimization is microarchitecture-dependent.  We
> > have observed a benefit on Kunpeng's hip12 machine.
> >
> > I constructed a small assembly benchmark to compare an LDP with two
> > LDRs when a preceding 64-bit STR overlaps the low 64-bit half of the
> > 128-bit load window.  The core sequence is essentially:
> >
> >         str     x0, [base, #16]
> >         ldp     x0, x1, [base, #16]
> >
> > compared with:
> >
> >         str     x0, [base, #16]
> >         ldr     x0, [base, #16]
> >         ldr     x1, [base, #24]
> >
> > The value loaded into x0 is used by the STR in the next iteration, forming
> > a dependency chain that makes the store-to-load forwarding latency
> > visible.  On Hip12, the two-LDR version is 130% faster than the LDP
> > version.
>
> This benchmark looks unrealistic, because I'd expect the compiler to apply
> store forwarding itself unless there's some more complicated code in between.
> In the included tests you're using volatile asm with memory clobbers to prevent
> the forwarding.  What happens to prevent compiler store forwarding in the
> real-world workloads this affects?
>
> Alice

Thanks.  A real case exists in RapidJSON's IStreamWrapper benchmark.
The relevant source is:

    while (isw.Take() != '\0');

    Ch Take() {
        Ch c = *current_;
        Read();
        return c;
    }

    void Read() {
        if (current_ < bufferLast_)
            ++current_;
        else
            ...
    }

The hot loop contains the following sequence across the loop backedge:

loop:
        ldp     x1, x0, [sp, #128]   // bufferLast_, current_
        ...
        add     x0, x0, #1
        str     x0, [sp, #136]       // current_ = current_ + 1
        cbnz    w28, loop

The compiler cannot forward this store because it is on the loop
backedge and does not dominate the load at the loop header.  The header
also has incoming paths from the initial entry, buffer refill, and EOF
handling, so the loaded value cannot be replaced by the stored value on
all paths.

The STR overlaps the high 64-bit half of the LDP in the next iteration.
With the optimization enabled, the LDP is split into:

        ldr     x0, [sp, #136]
        ldr     x1, [sp, #128]

On hip12, this reduced the execution time of IStreamWrapper_Unbuffered
by 32.12% and IStreamWrapper_Setbuffered by 32.67%.

The currently posted patch only searches for stores within the same
basic block, so it does not handle this case.  The latest version under
development uses DF information to search across basic-block
boundaries.

Thanks,
huzife

>
>
> >
> > I agree that rejecting the pair before it is formed is cleaner than
> > forming an LDP and splitting it again in a later pass.  We are currently
> > reworking the implementation in this direction: the AArch64 LDP
> > pair-fusion pass will recognize this specific store/load overlap pattern
> > and reject the LDP fusion, leaving the two scalar loads unchanged.  The
> > check will need to apply to both the early and late pair-fusion passes.
> > This heuristic will be disabled by default and enabled automatically
> > through the target tuning policy for affected  micro-arch.
> >
> > Thanks,
> > huzife
> >
> > Alex Coplan <[email protected]> 于2026年8月17日周一 17:20写道：
> >
> > > On 16/08/2026 20:51, Andrea Pinski wrote:
> > > > On Sun, Aug 16, 2026 at 8:17 PM huzife <[email protected]> wrote:
> > > > >
> > > > > A load pair that partially overlaps a recent scalar store can prevent
> > > the
> > > > > load from using store-to-load forwarding.  Splitting the pair lets the
> > > > > overlapping half use the forwarding path.
> > > > >
> > > > > Add a late AArch64 RTL pass that handles non-writeback SI and DI GPR
> > > load
> > > > > pairs.  Search a bounded part of the containing basic block for a
> > > scalar
> > > > > store to either half, and replace the pair only when both scalar loads
> > > are
> > > > > recognized.  Reject volatile, frame-related and exception-sensitive
> > > forms,
> > > > > and disable the transformation for size optimization.  Keep the pass
> > > off by
> > > > > default behind a target parameter while its profitability is evaluated.
> > > >
> > > >
> > > > I am not sure we want this.  Is this dependent on the micro-arch? Can
> > > > you expand on which micro-arch where this helps out?
> > > > Also shouldn't instead of a separate pass which undoes what an earlier
> > > > pass. Won't it be better to improve the heuristics of ldp_fusion pass
> > > > to reject this from happening in the first place for these cases?
> > >
> > > I agree with everything Andrea said here, it would be good to know which
> > > u-arches are expected to benefit, and potentially which workloads show a
> > > benefit with this transformation.
> > >
> > > If we do decide to to do this, I agree that it would be better to try
> > > and prevent the pairs from being formed in the first place (i.e. in
> > > pair-fusion) if possible.
> > >
> > > Thanks,
> > > Alex