Re: [RFC] aarch64: Split selected LDPs to improve store forwarding
huzife <[email protected]>
| Newsgroups | gmane.comp.gcc.patches |
|---|---|
| Message-ID | <CAP72AsEjw7eg-b=Rjci_=zvTaYLjuaUaDOp+wponR=_MFFzTuw@mail.gmail.com> |
Yes, the benefit of this optimization is microarchitecture-dependent. We
have observed a benefit on Kunpeng's hip12 machine.
I constructed a small assembly benchmark to compare an LDP with two
LDRs when a preceding 64-bit STR overlaps the low 64-bit half of the
128-bit load window. The core sequence is essentially:
str x0, [base, #16]
ldp x0, x1, [base, #16]
compared with:
str x0, [base, #16]
ldr x0, [base, #16]
ldr x1, [base, #24]
The value loaded into x0 is used by the STR in the next iteration, forming
a dependency chain that makes the store-to-load forwarding latency
visible. On Hip12, the two-LDR version is 130% faster than the LDP
version.
I agree that rejecting the pair before it is formed is cleaner than
forming an LDP and splitting it again in a later pass. We are currently
reworking the implementation in this direction: the AArch64 LDP
pair-fusion pass will recognize this specific store/load overlap pattern
and reject the LDP fusion, leaving the two scalar loads unchanged. The
check will need to apply to both the early and late pair-fusion passes.
This heuristic will be disabled by default and enabled automatically
through the target tuning policy for affected micro-arch.
Thanks,
huzife
Alex Coplan <[email protected]> 于2026年8月17日周一 17:20写道：
> On 16/08/2026 20:51, Andrea Pinski wrote:
> > On Sun, Aug 16, 2026 at 8:17 PM huzife <[email protected]> wrote:
> > >
> > > A load pair that partially overlaps a recent scalar store can prevent
> the
> > > load from using store-to-load forwarding. Splitting the pair lets the
> > > overlapping half use the forwarding path.
> > >
> > > Add a late AArch64 RTL pass that handles non-writeback SI and DI GPR
> load
> > > pairs. Search a bounded part of the containing basic block for a
> scalar
> > > store to either half, and replace the pair only when both scalar loads
> are
> > > recognized. Reject volatile, frame-related and exception-sensitive
> forms,
> > > and disable the transformation for size optimization. Keep the pass
> off by
> > > default behind a target parameter while its profitability is evaluated.
> >
> >
> > I am not sure we want this. Is this dependent on the micro-arch? Can
> > you expand on which micro-arch where this helps out?
> > Also shouldn't instead of a separate pass which undoes what an earlier
> > pass. Won't it be better to improve the heuristics of ldp_fusion pass
> > to reject this from happening in the first place for these cases?
>
> I agree with everything Andrea said here, it would be good to know which
> u-arches are expected to benefit, and potentially which workloads show a
> benefit with this transformation.
>
> If we do decide to to do this, I agree that it would be better to try
> and prevent the pairs from being formed in the first place (i.e. in
> pair-fusion) if possible.
>
> Thanks,
> Alex
>
> >
> > >
> > > gcc/ChangeLog:
> > >
> > > * config.gcc (aarch64*-*-*): Add aarch64-ldp-split.o.
> > > * config/aarch64/aarch64-ldp-split.cc: New file.
> > > * config/aarch64/aarch64-passes.def: Insert
> pass_split_ldp_stlf before
> > > pass_sched2.
> > > * config/aarch64/aarch64-protos.h (make_pass_split_ldp_stlf):
> Declare.
> > > * config/aarch64/aarch64.opt: Add aarch64-split-ldp-stlf
> parameter.
> > > * config/aarch64/t-aarch64 (aarch64-ldp-split.o): New rule.
> > > * doc/params.texi (aarch64-split-ldp-stlf): Document.
> > >
> > > gcc/testsuite/ChangeLog:
> > >
> > > * gcc.target/aarch64/split-ldp-stlf-1.c: New test.
> > > * gcc.target/aarch64/split-ldp-stlf-2.c: New test.
> > > * gcc.target/aarch64/split-ldp-stlf-3.c: New test.
> > > * gcc.target/aarch64/split-ldp-stlf-4.c: New test.
> > > * gcc.target/aarch64/split-ldp-stlf-5.c: New test.
> > > * gcc.target/aarch64/split-ldp-stlf-6.c: New test.
> > >
> > > Signed-off-by: huzife <[email protected]>
> > > ---
> > > gcc/config.gcc | 1 +
> > > gcc/config/aarch64/aarch64-ldp-split.cc | 316 ++++++++++++++++++
> > > gcc/config/aarch64/aarch64-passes.def | 1 +
> > > gcc/config/aarch64/aarch64-protos.h | 1 +
> > > gcc/config/aarch64/aarch64.opt | 5 +
> > > gcc/config/aarch64/t-aarch64 | 6 +
> > > gcc/doc/params.texi | 8 +
> > > .../gcc.target/aarch64/split-ldp-stlf-1.c | 43 +++
> > > .../gcc.target/aarch64/split-ldp-stlf-2.c | 13 +
> > > .../gcc.target/aarch64/split-ldp-stlf-3.c | 22 ++
> > > .../gcc.target/aarch64/split-ldp-stlf-4.c | 14 +
> > > .../gcc.target/aarch64/split-ldp-stlf-5.c | 13 +
> > > .../gcc.target/aarch64/split-ldp-stlf-6.c | 13 +
> > > 13 files changed, 456 insertions(+)
> > > create mode 100644 gcc/config/aarch64/aarch64-ldp-split.cc
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-1.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-2.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-3.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-4.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-5.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-6.c
> > >
> > > diff --git a/gcc/config.gcc b/gcc/config.gcc
> > > index 0477bf22c8a..00aa965f162 100644
> > > --- a/gcc/config.gcc
> > > +++ b/gcc/config.gcc
> > > @@ -362,6 +362,7 @@ aarch64*-*-*)
> > > extra_objs="${extra_objs} aarch-bti-insert.o"
> > > extra_objs="${extra_objs} aarch64-early-ra.o"
> > > extra_objs="${extra_objs} aarch64-ldp-fusion.o"
> > > + extra_objs="${extra_objs} aarch64-ldp-split.o"
> > > extra_objs="${extra_objs} aarch64-sched-dispatch.o"
> > > extra_objs="${extra_objs} aarch64-json-tunings-printer.o"
> > > extra_objs="${extra_objs} aarch64-json-tunings-parser.o"
> > > diff --git a/gcc/config/aarch64/aarch64-ldp-split.cc
> b/gcc/config/aarch64/aarch64-ldp-split.cc
> > > new file mode 100644
> > > index 00000000000..a4c2c159529
> > > --- /dev/null
> > > +++ b/gcc/config/aarch64/aarch64-ldp-split.cc
> > > @@ -0,0 +1,316 @@
> > > +/* Split selected load pairs to improve store-to-load forwarding on
> AArch64.
> > > + Copyright (C) 2026 Free Software Foundation, Inc.
> > > +
> > > + This file is part of GCC.
> > > +
> > > + GCC is free software; you can redistribute it and/or modify it
> under
> > > + the terms of the GNU General Public License as published by the
> Free
> > > + Software Foundation; either version 3, or (at your option) any
> later
> > > + version.
> > > +
> > > + GCC is distributed in the hope that it will be useful, but WITHOUT
> ANY
> > > + WARRANTY; without even the implied warranty of MERCHANTABILITY or
> > > + FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
> License
> > > + for more details.
> > > +
> > > + You should have received a copy of the GNU General Public License
> > > + along with GCC; see the file COPYING3. If not see
> > > + <http://www.gnu.org/licenses/>. */
> > > +
> > > +#define IN_TARGET_CODE 1
> > > +
> > > +#include "config.h"
> > > +#include "system.h"
> > > +#include "coretypes.h"
> > > +#include "backend.h"
> > > +#include "rtl.h"
> > > +#include "df.h"
> > > +#include "memmodel.h"
> > > +#include "emit-rtl.h"
> > > +#include "tree-pass.h"
> > > +#include "insn-attr.h"
> > > +#include "insn-constants.h"
> > > +#include "insn-config.h"
> > > +#include "recog.h"
> > > +#include "cfgrtl.h"
> > > +#include "predict.h"
> > > +
> > > +namespace {
> > > +
> > > +/* A non-writeback GPR load pair that is safe for this pass to
> split. */
> > > +struct ldp_info
> > > +{
> > > + rtx dests[2];
> > > + rtx mem;
> > > + rtx base;
> > > + machine_mode mode;
> > > + HOST_WIDE_INT offset;
> > > + HOST_WIDE_INT access_size;
> > > +};
> > > +
> > > +/* Return true if REG is a hard general-purpose register in MODE. */
> > > +
> > > +static bool
> > > +gpr_operand_p (rtx reg, machine_mode mode)
> > > +{
> > > + return (REG_P (reg)
> > > + && HARD_REGISTER_P (reg)
> > > + && GP_REGNUM_P (REGNO (reg))
> > > + && GET_MODE (reg) == mode);
> > > +}
> > > +
> > > +/* Extract the MEM from one half of an AArch64 load pair. */
> > > +
> > > +static rtx
> > > +extract_ldp_mem (rtx set, rtx dest, int unspec_code)
> > > +{
> > > + if (GET_CODE (set) != SET || !rtx_equal_p (SET_DEST (set), dest))
> > > + return NULL_RTX;
> > > +
> > > + rtx src = SET_SRC (set);
> > > + if (GET_CODE (src) != UNSPEC
> > > + || XINT (src, 1) != unspec_code
> > > + || XVECLEN (src, 0) != 1)
> > > + return NULL_RTX;
> > > +
> > > + rtx mem = XVECEXP (src, 0, 0);
> > > + return MEM_P (mem) ? mem : NULL_RTX;
> > > +}
> > > +
> > > +/* If INSN is an ordinary, non-writeback SI or DI GPR LDP, describe
> it in
> > > + INFO and return true. Deliberately reject FP/SIMD, TImode,
> volatile,
> > > + frame-related, and EH-sensitive forms. */
> > > +
> > > +static bool
> > > +extract_ldp (rtx_insn *insn, ldp_info *info)
> > > +{
> > > + if (!NONDEBUG_INSN_P (insn)
> > > + || INSN_CODE (insn) < 0
> > > + || get_attr_ldpstp (insn) != LDPSTP_LDP
> > > + || RTX_FRAME_RELATED_P (insn)
> > > + || find_reg_note (insn, REG_EH_REGION, NULL_RTX))
> > > + return false;
> > > +
> > > + rtx pattern = PATTERN (insn);
> > > + if (GET_CODE (pattern) != PARALLEL || XVECLEN (pattern, 0) != 2)
> > > + return false;
> > > +
> > > + rtx sets[2] = { XVECEXP (pattern, 0, 0), XVECEXP (pattern, 0, 1) };
> > > + if (GET_CODE (sets[0]) != SET || GET_CODE (sets[1]) != SET)
> > > + return false;
> > > +
> > > + info->dests[0] = SET_DEST (sets[0]);
> > > + info->dests[1] = SET_DEST (sets[1]);
> > > + info->mode = GET_MODE (info->dests[0]);
> > > + if ((info->mode != SImode && info->mode != DImode)
> > > + || !gpr_operand_p (info->dests[0], info->mode)
> > > + || !gpr_operand_p (info->dests[1], info->mode)
> > > + || reg_overlap_mentioned_p (info->dests[0], info->dests[1]))
> > > + return false;
> > > +
> > > + rtx mem0 = extract_ldp_mem (sets[0], info->dests[0],
> UNSPEC_LDP_FST);
> > > + rtx mem1 = extract_ldp_mem (sets[1], info->dests[1],
> UNSPEC_LDP_SND);
> > > + if (!mem0
> > > + || !mem1
> > > + || !rtx_equal_p (mem0, mem1)
> > > + || MEM_VOLATILE_P (mem0)
> > > + || side_effects_p (XEXP (mem0, 0))
> > > + || !MEM_SIZE_KNOWN_P (mem0)
> > > + || !MEM_SIZE (mem0).is_constant ())
> > > + return false;
> > > +
> > > + info->access_size = GET_MODE_SIZE (info->mode).to_constant ();
> > > + if (!known_eq (MEM_SIZE (mem0), 2 * info->access_size))
> > > + return false;
> > > +
> > > + poly_int64 offset;
> > > + info->base = strip_offset (XEXP (mem0, 0), &offset);
> > > + if (!REG_P (info->base) || !offset.is_constant (&info->offset))
> > > + return false;
> > > +
> > > + info->mem = mem0;
> > > + return true;
> > > +}
> > > +
> > > +/* Return true if INSN is a nonvolatile GPR store of one LDP half,
> using the
> > > + same unmodified base as INFO and writing exactly the low or high
> half. */
> > > +
> > > +static bool
> > > +forwarding_store_p (rtx_insn *insn, const ldp_info &info)
> > > +{
> > > + if (!NONDEBUG_INSN_P (insn) || RTX_FRAME_RELATED_P (insn))
> > > + return false;
> > > +
> > > + rtx set = single_set (insn);
> > > + if (!set || !MEM_P (SET_DEST (set)))
> > > + return false;
> > > +
> > > + rtx mem = SET_DEST (set);
> > > + rtx value = SET_SRC (set);
> > > + if (GET_MODE (mem) != info.mode
> > > + || !gpr_operand_p (value, info.mode)
> > > + || MEM_VOLATILE_P (mem)
> > > + || side_effects_p (XEXP (mem, 0))
> > > + || !MEM_SIZE_KNOWN_P (mem)
> > > + || !known_eq (MEM_SIZE (mem), info.access_size))
> > > + return false;
> > > +
> > > + poly_int64 poly_offset;
> > > + rtx base = strip_offset (XEXP (mem, 0), &poly_offset);
> > > + HOST_WIDE_INT offset;
> > > + if (!REG_P (base)
> > > + || !rtx_equal_p (base, info.base)
> > > + || !poly_offset.is_constant (&offset))
> > > + return false;
> > > +
> > > + return (offset == info.offset
> > > + || offset == info.offset + info.access_size);
> > > +}
> > > +
> > > +/* Search backwards in the containing basic block for a nearby store
> that
> > > + makes splitting PAIR profitable. Stop once the base value
> changes. */
> > > +
> > > +static rtx_insn *
> > > +find_forwarding_store (rtx_insn *pair, const ldp_info &info)
> > > +{
> > > + basic_block bb = BLOCK_FOR_INSN (pair);
> > > + unsigned int distance = 0;
> > > + unsigned int max_distance = (unsigned int) aarch64_split_ldp_stlf;
> > > +
> > > + for (rtx_insn *insn = PREV_INSN (pair);
> > > + insn && BLOCK_FOR_INSN (insn) == bb;
> > > + insn = PREV_INSN (insn))
> > > + {
> > > + if (!NONDEBUG_INSN_P (insn))
> > > + continue;
> > > +
> > > + if (++distance > max_distance)
> > > + break;
> > > +
> > > + if (CALL_P (insn) || modified_in_p (info.base, insn))
> > > + break;
> > > +
> > > + if (forwarding_store_p (insn, info))
> > > + return insn;
> > > + }
> > > +
> > > + return NULL;
> > > +}
> > > +
> > > +/* Replace PAIR with two individually-recognized scalar loads. Load
> the
> > > + non-base destination first when the low destination overlaps the
> base. */
> > > +
> > > +static bool
> > > +split_ldp (rtx_insn *pair, const ldp_info &info)
> > > +{
> > > + rtx mems[2] = {
> > > + adjust_address_nv (info.mem, info.mode, 0),
> > > + adjust_address_nv (info.mem, info.mode, info.access_size)
> > > + };
> > > +
> > > + int order[2] = { 0, 1 };
> > > + if (reg_overlap_mentioned_p (info.base, info.dests[0]))
> > > + std::swap (order[0], order[1]);
> > > +
> > > + start_sequence ();
> > > + emit_insn (gen_rtx_SET (info.dests[order[0]], mems[order[0]]));
> > > + emit_insn (gen_rtx_SET (info.dests[order[1]], mems[order[1]]));
> > > + rtx_insn *seq = get_insns ();
> > > + end_sequence ();
> > > +
> > > + unshare_all_rtl_in_chain (seq);
> > > + for (rtx_insn *insn = seq; insn; insn = NEXT_INSN (insn))
> > > + if (!NONDEBUG_INSN_P (insn) || recog_memoized (insn) < 0)
> > > + {
> > > + if (dump_file)
> > > + fprintf (dump_file,
> > > + "Rejecting LDP %d: scalar replacement is not
> recognized\n",
> > > + INSN_UID (pair));
> > > + return false;
> > > + }
> > > +
> > > + if (dump_file)
> > > + {
> > > + fprintf (dump_file, "Splitting LDP %d into:\n", INSN_UID
> (pair));
> > > + for (rtx_insn *insn = seq; insn; insn = NEXT_INSN (insn))
> > > + print_rtl_single (dump_file, insn);
> > > + }
> > > +
> > > + emit_insn_before_setloc (seq, pair, INSN_LOCATION (pair));
> > > + delete_insn (pair);
> > > + return true;
> > > +}
> > > +
> > > +const pass_data pass_data_split_ldp_stlf =
> > > +{
> > > + RTL_PASS, /* type. */
> > > + "split_ldp_stlf", /* name. */
> > > + OPTGROUP_NONE, /* optinfo_flags. */
> > > + TV_MACH_DEP, /* tv_id. */
> > > + 0, /* properties_required. */
> > > + 0, /* properties_provided. */
> > > + 0, /* properties_destroyed. */
> > > + 0, /* todo_flags_start. */
> > > + TODO_df_verify | TODO_df_finish, /* todo_flags_finish. */
> > > +};
> > > +
> > > +class pass_split_ldp_stlf : public rtl_opt_pass
> > > +{
> > > +public:
> > > + pass_split_ldp_stlf (gcc::context *ctxt)
> > > + : rtl_opt_pass (pass_data_split_ldp_stlf, ctxt)
> > > + {}
> > > +
> > > + bool gate (function *fn) final override
> > > + {
> > > + return (optimize >= 1
> > > + && !optimize_debug
> > > + && reload_completed
> > > + && aarch64_split_ldp_stlf
> > > + && !optimize_function_for_size_p (fn)
> > > + && !fn->can_throw_non_call_exceptions);
> > > + }
> > > +
> > > + unsigned int execute (function *fn) final override
> > > + {
> > > + unsigned int split_count = 0;
> > > + basic_block bb;
> > > +
> > > + FOR_EACH_BB_FN (bb, fn)
> > > + {
> > > + rtx_insn *insn;
> > > + rtx_insn *next;
> > > + FOR_BB_INSNS_SAFE (bb, insn, next)
> > > + {
> > > + ldp_info info;
> > > + if (!extract_ldp (insn, &info))
> > > + continue;
> > > +
> > > + rtx_insn *store = find_forwarding_store (insn, info);
> > > + if (!store)
> > > + continue;
> > > +
> > > + if (dump_file)
> > > + {
> > > + fprintf (dump_file, "Found forwarding store for LDP
> %d:\n",
> > > + INSN_UID (insn));
> > > + print_rtl_single (dump_file, store);
> > > + }
> > > +
> > > + split_count += split_ldp (insn, info);
> > > + }
> > > + }
> > > +
> > > + if (dump_file)
> > > + fprintf (dump_file, "Split %u LDP instructions\n", split_count);
> > > + return 0;
> > > + }
> > > +};
> > > +
> > > +} // namespace
> > > +
> > > +rtl_opt_pass *
> > > +make_pass_split_ldp_stlf (gcc::context *ctxt)
> > > +{
> > > + return new pass_split_ldp_stlf (ctxt);
> > > +}
> > > diff --git a/gcc/config/aarch64/aarch64-passes.def
> b/gcc/config/aarch64/aarch64-passes.def
> > > index 2df7459af92..594348a0532 100644
> > > --- a/gcc/config/aarch64/aarch64-passes.def
> > > +++ b/gcc/config/aarch64/aarch64-passes.def
> > > @@ -26,4 +26,5 @@ INSERT_PASS_BEFORE
> (pass_late_thread_prologue_and_epilogue, 1, pass_late_track_s
> > > INSERT_PASS_BEFORE (pass_shorten_branches, 1, pass_insert_bti);
> > > INSERT_PASS_BEFORE (pass_early_remat, 1, pass_ldp_fusion);
> > > INSERT_PASS_BEFORE (pass_peephole2, 1, pass_ldp_fusion);
> > > +INSERT_PASS_BEFORE (pass_sched2, 1, pass_split_ldp_stlf);
> > > INSERT_PASS_BEFORE (pass_free_cfg, 1, pass_narrow_gp_writes);
> > > diff --git a/gcc/config/aarch64/aarch64-protos.h
> b/gcc/config/aarch64/aarch64-protos.h
> > > index bcc833cfaa1..ec8be386c29 100644
> > > --- a/gcc/config/aarch64/aarch64-protos.h
> > > +++ b/gcc/config/aarch64/aarch64-protos.h
> > > @@ -1268,6 +1268,7 @@ rtl_opt_pass *make_pass_late_track_speculation
> (gcc::context *);
> > > rtl_opt_pass *make_pass_insert_bti (gcc::context *ctxt);
> > > rtl_opt_pass *make_pass_switch_pstate_sm (gcc::context *ctxt);
> > > rtl_opt_pass *make_pass_ldp_fusion (gcc::context *);
> > > +rtl_opt_pass *make_pass_split_ldp_stlf (gcc::context *);
> > > rtl_opt_pass *make_pass_narrow_gp_writes (gcc::context *);
> > >
> > > poly_uint64 aarch64_regmode_natural_size (machine_mode);
> > > diff --git a/gcc/config/aarch64/aarch64.opt
> b/gcc/config/aarch64/aarch64.opt
> > > index deccc6c8887..21e9500996d 100644
> > > --- a/gcc/config/aarch64/aarch64.opt
> > > +++ b/gcc/config/aarch64/aarch64.opt
> > > @@ -461,6 +461,11 @@ individual writeback accesses where possible. A
> value of two means we
> > > also try to opportunistically form writeback opportunities by folding
> in
> > > trailing destructive updates of the base register used by a pair.
> > >
> > > +-param=aarch64-split-ldp-stlf=
> > > +Target Joined UInteger Var(aarch64_split_ldp_stlf) Init(0)
> IntegerRange(0, 32) Param
> > > +Maximum number of preceding instructions to search when splitting an
> LDP
> > > +to improve store-to-load forwarding. A value of zero disables the
> pass.
> > > +
> > > -param=aarch64-tag-memory-loop-threshold=
> > > Target Joined UInteger Var(aarch64_tag_memory_loop_threshold)
> Init(10) IntegerRange(0, 65536) Param
> > > Param to control the threshold in number of granules beyond which an
> > > diff --git a/gcc/config/aarch64/t-aarch64
> b/gcc/config/aarch64/t-aarch64
> > > index 3942a1049b9..936540aeb4e 100644
> > > --- a/gcc/config/aarch64/t-aarch64
> > > +++ b/gcc/config/aarch64/t-aarch64
> > > @@ -238,6 +238,12 @@ aarch64-ldp-fusion.o:
> $(srcdir)/config/aarch64/aarch64-ldp-fusion.cc \
> > > $(COMPILER) -c $(ALL_COMPILERFLAGS) $(ALL_CPPFLAGS)
> $(INCLUDES) \
> > > $(srcdir)/config/aarch64/aarch64-ldp-fusion.cc
> > >
> > > +aarch64-ldp-split.o: $(srcdir)/config/aarch64/aarch64-ldp-split.cc \
> > > + $(CONFIG_H) $(SYSTEM_H) $(CORETYPES_H) $(BACKEND_H) $(RTL_H)
> $(DF_H) \
> > > + $(INSN_ATTR_H) insn-config.h $(RECOG_H) tree-pass.h
> > > + $(COMPILER) -c $(ALL_COMPILERFLAGS) $(ALL_CPPFLAGS)
> $(INCLUDES) \
> > > + $(srcdir)/config/aarch64/aarch64-ldp-split.cc
> > > +
> > > aarch64-sched-dispatch.o:
> $(srcdir)/config/aarch64/aarch64-sched-dispatch.cc \
> > > $(CONFIG_H) $(SYSTEM_H) $(CORETYPES_H) $(BACKEND_H) $(RTL_H) \
> > > $(INSN_ATTR_H) $(REGSET_H) sched-int.h $(DUMPFILE_H) \
> > > diff --git a/gcc/doc/params.texi b/gcc/doc/params.texi
> > > index 26b34a59aa9..1ae93ae6c75 100644
> > > --- a/gcc/doc/params.texi
> > > +++ b/gcc/doc/params.texi
> > > @@ -2061,6 +2061,14 @@ accesses where possible. A value of two means
> we also try to opportunistically
> > > form writeback opportunities by folding in trailing destructive
> updates of the
> > > base register used by a pair.
> > >
> > > +@paindex aarch64-split-ldp-stlf
> > > +@item aarch64-split-ldp-stlf
> > > +Maximum number of preceding instructions that the AArch64 backend
> searches
> > > +for a scalar store that writes one half of a later @code{ldp}. When
> such a
> > > +store is found, the backend can split the @code{ldp} into two scalar
> loads to
> > > +improve store-to-load forwarding. A value of zero, which is the
> default,
> > > +disables the transformation.
> > > +
> > > @paindex aarch64-loop-vect-issue-rate-niters
> > > @item aarch64-loop-vect-issue-rate-niters
> > > The tuning for some AArch64 CPUs tries to take both latencies and
> issue
> > > diff --git a/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-1.c
> b/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-1.c
> > > new file mode 100644
> > > index 00000000000..ca4ca4821b1
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-1.c
> > > @@ -0,0 +1,43 @@
> > > +/* { dg-options "-O2 --param=aarch64-split-ldp-stlf=16" } */
> > > +
> > > +__attribute__ ((noinline, noclone))
> > > +long
> > > +split_low (long *p, long value)
> > > +{
> > > + p[1] = value;
> > > + __asm__ __volatile__ ("" ::: "memory");
> > > + return p[1] + p[2];
> > > +}
> > > +
> > > +__attribute__ ((noinline, noclone))
> > > +long
> > > +split_high (long *p, long value)
> > > +{
> > > + p[2] = value;
> > > + __asm__ __volatile__ ("" ::: "memory");
> > > + return p[1] + p[2];
> > > +}
> > > +
> > > +__attribute__ ((noinline, noclone))
> > > +int
> > > +split_low_si (int *p, int value)
> > > +{
> > > + p[1] = value;
> > > + __asm__ __volatile__ ("" ::: "memory");
> > > + return p[1] + p[2];
> > > +}
> > > +
> > > +extern void consume (long, long);
> > > +
> > > +__attribute__ ((noinline, noclone))
> > > +void
> > > +split_base_dest_overlap (long *p, long value)
> > > +{
> > > + p[1] = value;
> > > + __asm__ __volatile__ ("" ::: "memory");
> > > + consume (p[1], p[2]);
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-not {\tldp\t} } } */
> > > +/* { dg-final { scan-assembler-times {\tldr\t} 8 } } */
> > > +/* { dg-final { scan-assembler-times {\tldr\tx1, \[x0,
> 16\]\n\tldr\tx0, \[x0, 8\]} 1 } } */
> > > diff --git a/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-2.c
> b/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-2.c
> > > new file mode 100644
> > > index 00000000000..2ad7e2d5825
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-2.c
> > > @@ -0,0 +1,13 @@
> > > +/* { dg-options "-O2" } */
> > > +
> > > +__attribute__ ((noinline, noclone))
> > > +long
> > > +disabled_by_default (long *p, long value)
> > > +{
> > > + p[1] = value;
> > > + __asm__ __volatile__ ("" ::: "memory");
> > > + return p[1] + p[2];
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times {\tldp\t} 1 } } */
> > > +/* { dg-final { scan-assembler-not {\tldr\t} } } */
> > > diff --git a/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-3.c
> b/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-3.c
> > > new file mode 100644
> > > index 00000000000..930ada6836d
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-3.c
> > > @@ -0,0 +1,22 @@
> > > +/* { dg-options "-O2 --param=aarch64-split-ldp-stlf=16" } */
> > > +
> > > +__attribute__ ((noinline, noclone))
> > > +long
> > > +no_overlap (long *p, long value)
> > > +{
> > > + p[4] = value;
> > > + __asm__ __volatile__ ("" ::: "memory");
> > > + return p[1] + p[2];
> > > +}
> > > +
> > > +__attribute__ ((noinline, noclone))
> > > +long
> > > +wrong_store_width (long *p, int value)
> > > +{
> > > + *(int *) ((char *) p + 8) = value;
> > > + __asm__ __volatile__ ("" ::: "memory");
> > > + return p[1] + p[2];
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times {\tldp\t} 2 } } */
> > > +/* { dg-final { scan-assembler-not {\tldr\t} } } */
> > > diff --git a/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-4.c
> b/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-4.c
> > > new file mode 100644
> > > index 00000000000..3e08bcbea05
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-4.c
> > > @@ -0,0 +1,14 @@
> > > +/* { dg-options "-O2 --param=aarch64-split-ldp-stlf=16" } */
> > > +/* { dg-additional-options "-fnon-call-exceptions" } */
> > > +
> > > +__attribute__ ((noinline, noclone))
> > > +long
> > > +preserve_faulting_pair (long *p, long value)
> > > +{
> > > + p[1] = value;
> > > + __asm__ __volatile__ ("" ::: "memory");
> > > + return p[1] + p[2];
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times {\tldp\t} 1 } } */
> > > +/* { dg-final { scan-assembler-not {\tldr\t} } } */
> > > diff --git a/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-5.c
> b/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-5.c
> > > new file mode 100644
> > > index 00000000000..a19551c4374
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-5.c
> > > @@ -0,0 +1,13 @@
> > > +/* { dg-options "-Os --param=aarch64-split-ldp-stlf=16" } */
> > > +
> > > +__attribute__ ((noinline, noclone))
> > > +long
> > > +optimize_for_size (long *p, long value)
> > > +{
> > > + p[1] = value;
> > > + __asm__ __volatile__ ("" ::: "memory");
> > > + return p[1] + p[2];
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times {\tldp\t} 1 } } */
> > > +/* { dg-final { scan-assembler-not {\tldr\t} } } */
> > > diff --git a/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-6.c
> b/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-6.c
> > > new file mode 100644
> > > index 00000000000..bd59fe77b47
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/aarch64/split-ldp-stlf-6.c
> > > @@ -0,0 +1,13 @@
> > > +/* { dg-options "-O2 --param=aarch64-split-ldp-stlf=1" } */
> > > +
> > > +__attribute__ ((noinline, noclone))
> > > +long
> > > +outside_search_window (long *p, long value)
> > > +{
> > > + p[1] = value;
> > > + __asm__ __volatile__ ("" ::: "memory");
> > > + return p[1] + p[2];
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times {\tldp\t} 1 } } */
> > > +/* { dg-final { scan-assembler-not {\tldr\t} } } */
> > > --
> > > 2.43.0
> > >
>