RE: [PATCH] aarch64: Fix invalid CSE for RDRFFS [PR 126629]

Hi Alfie,

> -----Original Message-----
> From: Alfie Richards <[email protected]>
> Sent: 07 August 2026 14:05
> To: Tamar Christina <[email protected]>
> Cc: [email protected]; Alex Coplan <[email protected]>; Alice
> Carlotti <[email protected]>; [email protected];
> [email protected]; Richard Earnshaw <[email protected]>;
> Wilco Dijkstra <[email protected]>; [email protected]
> Subject: Re: [PATCH] aarch64: Fix invalid CSE for RDRFFS [PR 126629]
> 
> On 07/08/2026 13:16, Tamar Christina wrote:
> > Hi Alfie,
> >
> > The 08/07/2026 11:31, Alfie Richards wrote:
> >> Hi Tamar,
> >> Sorry replying from phone so poor formatting.
> >>
> >>> I think the idea is good, but I don't quite like that now every first faulting
> load shape
> >>> needs Its own new update ffr and that the same UNSPEC has different
> amount of arguments.
> >>
> >> I do agree this is the part of the patch I was most uncomfortable with.
> >>
> >>> And because these are pseudo registers and we only have one of them the
> REGNOs are the same
> >>> and so it assumes they are the same.  The patch fixes it by adding the load
> arguments to the
> >>> UNSPEC_UPDATE_FFR to make them unique to the load.
> >>
> >> Essentially yes but it’s also because the whole FFR expression tree matches
> going back to the setffr instructions (I believe)
> >>
> >>> That should give us a unique aarch64_update_ffr_for_load and also like
> your patch still
> >>> allow removals of redundant setffrs and rdffrs but be a bit simpler and
> easier to backport.
> >>
> >> Yeah I like that idea, and I think it should work. I did think about something
> similar but my only hesitation was around the situation where you do two
> identical loads. I like in that situation with my patch that the duplicate load and
> rdffr gets removed whereas with the solution you propose we would remove
> the ldffr but keep the duplicate rdffr (and hidden non-instructions). Not a big
> issue as it’s a pretty nonsense case but that’s why I went the way I did.
> >>
> >
> > I'm not sure I understand which scenario this is.
> >
> > Applying your patch:
> >
> >    #include <arm_sve.h>
> >
> >    void
> >    same_load_read_same_load_read (unsigned char *p, unsigned char *o1,
> >                                   unsigned char *o2, unsigned long *r1,
> >                                   unsigned long *r2)
> >    {
> >      svbool_t pt = svptrue_b8 ();
> >
> >      svsetffr ();
> >
> >      svuint8_t a = svldff1_u8 (pt, p);
> >      unsigned long n1 = svcntp_b8 (pt, svrdffr ());
> >
> >      svuint8_t b = svldff1_u8 (pt, p);
> >      unsigned long n2 = svcntp_b8 (pt, svrdffr ());
> >
> >      svst1_u8 (pt, o1, a);
> >      svst1_u8 (pt, o2, b);
> >      *r1 = n1;
> >      *r2 = n2;
> >    }
> 
> Ah so it would need to be the same situation as the original case with
> the double setffr, which is important to make the "expression" trees
> identical. I guess this make this opimiation so restrictive to the point
> of being even more negligable.
> 
> So:
> 
>      #include <arm_sve.h>
> 
>      void
>      same_load_read_same_load_read (unsigned char *p, unsigned char *o1,
>                                     unsigned char *o2, unsigned long*r1,
>                                     unsigned long *r2)
>      {
>        svbool_t pt = svptrue_b8 ();
> 
>        svsetffr ();
> 
>        svuint8_t a = svldff1_u8 (pt, p);
>        unsigned long n1 = svcntp_b8 (pt, svrdffr ());
> 
>        svsetffr ();
> 
>        svuint8_t b = svldff1_u8 (pt, p);
>        unsigned long n2 = svcntp_b8 (pt, svrdffr ());
> 
>        svst1_u8 (pt, o1, a);
>        svst1_u8 (pt, o2, b);
>        *r1 = n1;
>        *r2 = n2;
>      }
> 
> Gives:
> 
> same_load_read_same_load_read:
> .LFB2:
> 	.cfi_startproc
> 	ptrue	p7.b, all
> 	setffr
> 	ldff1b	z31.b, p7/z, [x0]
> 	st1b	z31.b, p7, [x1]
> 	rdffr	p15.b
> 	cntp	x5, p7, p15.b
> 	st1b	z31.b, p7, [x2]
> 	str	x5, [x3]
> 	str	x5, [x4]
> 	ret
> 	.cfi_endproc
> 
> But again, pretty niche case.

For that all you need is to use the value result of the load
as the unique token.  So instead of const_int you have

(define_insn "aarch64_update_ffr_for_load"
  [(set (reg:VNx16BI FFR_REGNUM)
	(unspec:VNx16BI [(reg:VNx16BI FFRT_REGNUM)
			 (reg:VNx16BI FFR_REGNUM)
			 (match_operand 0 "register_operand" "w")]
			UNSPEC_UPDATE_FFR))]

And then the emitters become

rtx res = e.use_contiguous_load_insn (icode);
emit_insn (gen_aarch64_update_ffr_for_load (res));
return res;

etc.  When the optimizers prove the loads are the same the results
become the same pseudos or get an equivalent relationship set which
makes the UNSPEC_UPDATE_FFRs the same and so they can be CSEd.

Thanks,
Tamar

> 
> Thanks,
> Alfie
> >
> > Or did I misunderstand the case you were talking about?
> >
> > Thanks,
> > Tamar
> >
> >> Thoughts? Happy to take your solution if you’d still rather.
> >>
> >> KR,
> >> Alfie
> >>
> >> ________________________________
> >> ​​​​​From: Tamar Christina <[email protected]>
> >> Sent: 07 August 2026 09:23
> >> To: Alfie Richards <[email protected]>; [email protected]
> <[email protected]>
> >> Cc: Alfie Richards <[email protected]>; Alex Coplan
> <[email protected]>; Alice Carlotti <[email protected]>;
> [email protected] <[email protected]>;
> [email protected] <[email protected]>; Richard Earnshaw
> <[email protected]>; Wilco Dijkstra <[email protected]>;
> [email protected] <[email protected]>
> >> Subject: RE: [PATCH] aarch64: Fix invalid CSE for RDRFFS [PR 126629]
> >>
> >>> -----Original Message-----
> >>> From: Alfie Richards <[email protected]>
> >>> Sent: 05 August 2026 16:42
> >>> To: [email protected]
> >>> Cc: Alfie Richards <[email protected]>; Alex Coplan
> >>> <[email protected]>; Alice Carlotti <[email protected]>;
> >>> [email protected]; [email protected]; Richard
> Earnshaw
> >>> <[email protected]>; Tamar Christina
> <[email protected]>;
> >>> Wilco Dijkstra <[email protected]>; [email protected]
> >>> Subject: [PATCH] aarch64: Fix invalid CSE for RDRFFS [PR 126629]
> >>>
> >>> Fixes the invalid CSE'ing of the FFR reads by adding the arguments from
> >>> the FFR loads to the associated FFR updates.
> >>>
> >>> This isn't quite perfectly true, as the hardware presumably could set
> >>> return different FFR values from the exact same load happening twice, but
> that
> >>> is a nonsense situation where the optimisation is valid.
> >>>
> >>
> >> Hi Alfie,
> >>
> >> I think the idea is good, but I don't quite like that now every first faulting
> load shape
> >> needs Its own new update ffr and that the same UNSPEC has different
> amount of arguments.
> >>
> >> If I’m not mistaken the problem is that CSE lib essentially sees
> >>
> >> (set (reg:VNx16BI FFR_REGNUM)
> >>       (unspec:VNx16BI [(reg:VNx16BI FFRT_REGNUM)
> >>                        (reg:VNx16BI FFR_REGNUM)]
> >>                       UNSPEC_UPDATE_FFR))
> >> (set (reg:VNx16BI FFR_REGNUM)
> >>       (unspec:VNx16BI [(reg:VNx16BI FFRT_REGNUM)
> >>                        (reg:VNx16BI FFR_REGNUM)]
> >>                       UNSPEC_UPDATE_FFR))
> >>
> >> And because these are pseudo registers and we only have one of them the
> REGNOs are the same
> >> and so it assumes they are the same.  The patch fixes it by adding the load
> arguments to the
> >> UNSPEC_UPDATE_FFR to make them unique to the load.
> >>
> >> But I think we can fix this simpler by just adding a unique token to the
> aarch64_update_ffr_for_load.
> >>
> >> Like
> >>
> >> (define_insn "aarch64_update_ffr_for_load"
> >>    [(set (reg:VNx16BI FFR_REGNUM)
> >>          (unspec:VNx16BI [(reg:VNx16BI FFRT_REGNUM)
> >>                           (reg:VNx16BI FFR_REGNUM)
> >>                           (match_operand 0 "const_int_operand" "n")]
> >>                          UNSPEC_UPDATE_FFR))]
> >>
> >> Then change
> >>
> >> emit_insn (gen_aarch64_update_ffr_for_load ());
> >>
> >> into
> >>
> >> emit_insn (gen_aarch64_update_ffr_for_load (GEN_INT (get_max_uid ())));
> >>
> >> the exact value of which doesn't really matter as long as it's unique (maybe
> make a helper).
> >>
> >> That should give us a unique aarch64_update_ffr_for_load and also like
> your patch still
> >> allow removals of redundant setffrs and rdffrs but be a bit simpler and
> easier to backport.
> >>
> >> Thanks,
> >> Tamar
> >>
> >>>         PR 126629
> >>>
> >>> gcc/ChangeLog:
> >>>
> >>>         * config/aarch64/aarch64-sve-builtins-base.cc:
> >>>         (svldff1_gather_impl::expand): Remove
> >>> gen_aarch64_update_ffr_for_load.
> >>>         (svldff1_svldff1_gather_extend::expand): Ditto.
> >>>         (svldxf1_impl::expand): Ditto.
> >>>         (svldxf1_extend_impl::expand): Ditto.
> >>>         * config/aarch64/aarch64-sve.md (aarch64_update_ffr_for_load):
> >>> Remove.
> >>>         (*aarch64_update_ffr_for_continuous_load): New.
> >>>         (*aarch64_update_ffr_for_gather_load): New.
> >>>         (@aarch64_ld<fn>f1<mode>): New.
> >>>         (*aarch64_ld<fn>f1<mode>): Change to not be expanded directly.
> >>>         (@aarch64_ld<fn>f1_<ANY_EXTEND:optab><SVE_HSDI:mode><SVE_
> >>> PARTIAL_I:mode>): New.
> >>>         (*aarch64_ld<fn>f1_<ANY_EXTEND:optab><SVE_HSDI:mode><SVE_P
> >>> ARTIAL_I:mode>): Change to not be expanded directly.
> >>>         (@aarch64_ldff1_gather<mode>): New.
> >>>         (*aarch64_ldff1_gather<mode>): Change to not be expanded directly.
> >>>         (@aarch64_ldff1_gather_<ANY_EXTEND:optab><VNx4_WIDE:mode>
> >>> <VNx4_NARROW:mode>): New.
> >>>         (*aarch64_ldff1_gather_<ANY_EXTEND:optab><VNx4_WIDE:mode><
> >>> VNx4_NARROW:mode>): Change to not be expanded directly.
> >>>         (@aarch64_ldff1_gather_<ANY_EXTEND:optab><VNx2_WIDE:mode>
> >>> <VNx2_NARROW:mode>): New.
> >>>         (*aarch64_ldff1_gather_<ANY_EXTEND:optab><VNx2_WIDE:mode><
> >>> VNx2_NARROW:mode>): Change to not be expanded directly.
> >>>
> >>> gcc/testsuite/ChangeLog:
> >>>
> >>>         * gcc.target/aarch64/sve/pr126629.c: New test.
> >>>         * gcc.target/aarch64/sve/pr126629_extend.c: New test.
> >>>         * gcc.target/aarch64/sve/pr126629_gather.c: New test.
> >>>         * gcc.target/aarch64/sve/pr126629_gather2.c: New test.
> >>>
> >>> -- >8 --
> >>>
> >>> Bootstrapped and regression tested on aarch64.
> >>>
> >>> Okay for master and backport?
> >>>
> >>> Thanks,
> >>> Alfie
> >>>
> >>> ---
> >>>   .../aarch64/aarch64-sve-builtins-base.cc      |   4 -
> >>>   gcc/config/aarch64/aarch64-sve.md             | 206 ++++++++++++++++--
> >>>   .../gcc.target/aarch64/sve/pr126629.c         |  23 ++
> >>>   .../gcc.target/aarch64/sve/pr126629_extend.c  |  23 ++
> >>>   .../gcc.target/aarch64/sve/pr126629_gather.c  |  24 ++
> >>>   .../gcc.target/aarch64/sve/pr126629_gather2.c |  24 ++
> >>>   6 files changed, 279 insertions(+), 25 deletions(-)
> >>>   create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/pr126629.c
> >>>   create mode 100644
> >>> gcc/testsuite/gcc.target/aarch64/sve/pr126629_extend.c
> >>>   create mode 100644
> >>> gcc/testsuite/gcc.target/aarch64/sve/pr126629_gather.c
> >>>   create mode 100644
> >>> gcc/testsuite/gcc.target/aarch64/sve/pr126629_gather2.c
> >>>
> >>> diff --git a/gcc/config/aarch64/aarch64-sve-builtins-base.cc
> >>> b/gcc/config/aarch64/aarch64-sve-builtins-base.cc
> >>> index 7f047bb6468..186f223ac63 100644
> >>> --- a/gcc/config/aarch64/aarch64-sve-builtins-base.cc
> >>> +++ b/gcc/config/aarch64/aarch64-sve-builtins-base.cc
> >>> @@ -2001,7 +2001,6 @@ public:
> >>>     {
> >>>       /* See the block comment in aarch64-sve.md for details about the
> >>>          FFR handling.  */
> >>> -    emit_insn (gen_aarch64_update_ffr_for_load ());
> >>>
> >>>       e.prepare_gather_address_operands (1);
> >>>       /* Put the predicate last, since ldff1_gather uses the same operand
> >>> @@ -2023,7 +2022,6 @@ public:
> >>>     {
> >>>       /* See the block comment in aarch64-sve.md for details about the
> >>>          FFR handling.  */
> >>> -    emit_insn (gen_aarch64_update_ffr_for_load ());
> >>>
> >>>       e.prepare_gather_address_operands (1);
> >>>       /* Put the predicate last, since ldff1_gather uses the same operand
> >>> @@ -2075,7 +2073,6 @@ public:
> >>>     {
> >>>       /* See the block comment in aarch64-sve.md for details about the
> >>>          FFR handling.  */
> >>> -    emit_insn (gen_aarch64_update_ffr_for_load ());
> >>>
> >>>       machine_mode mode = e.vector_mode (0);
> >>>       return e.use_contiguous_load_insn (code_for_aarch64_ldf1
> (m_unspec,
> >>> mode));
> >>> @@ -2103,7 +2100,6 @@ public:
> >>>     {
> >>>       /* See the block comment in aarch64-sve.md for details about the
> >>>          FFR handling.  */
> >>> -    emit_insn (gen_aarch64_update_ffr_for_load ());
> >>>
> >>>       insn_code icode = code_for_aarch64_ldf1 (m_unspec, extend_rtx_code
> (),
> >>>                                              e.vector_mode (0),
> >>> diff --git a/gcc/config/aarch64/aarch64-sve.md
> >>> b/gcc/config/aarch64/aarch64-sve.md
> >>> index 105b34eb8fa..1e131cc404a 100644
> >>> --- a/gcc/config/aarch64/aarch64-sve.md
> >>> +++ b/gcc/config/aarch64/aarch64-sve.md
> >>> @@ -1150,10 +1150,31 @@ (define_insn "aarch64_wrffr"
> >>>   ;; so that the FFR value is live on entry to the region and so that the FFR
> >>>   ;; value visibly changes within the region.  This is used (possibly multiple
> >>>   ;; times) in an FFRT region that includes LDFF1 or LDNF1 instructions.
> >>> -(define_insn "aarch64_update_ffr_for_load"
> >>> +(define_insn "*aarch64_update_ffr_for_continuous_load"
> >>>     [(set (reg:VNx16BI FFR_REGNUM)
> >>> -     (unspec:VNx16BI [(reg:VNx16BI FFRT_REGNUM)
> >>> -                      (reg:VNx16BI FFR_REGNUM)]
> >>> UNSPEC_UPDATE_FFR))]
> >>> +     (unspec:VNx16BI
> >>> +      [(reg:VNx16BI FFRT_REGNUM)
> >>> +       (reg:VNx16BI FFR_REGNUM)
> >>> +       (match_operand 0 "" "X")
> >>> +       (match_operand 1 "" "X")
> >>> +      ] UNSPEC_UPDATE_FFR))]
> >>> +  "TARGET_SVE"
> >>> +  ""
> >>> +  [(set_attr "type" "no_insn")]
> >>> +)
> >>> +
> >>> +(define_insn "*aarch64_update_ffr_for_gather_load"
> >>> +  [(set (reg:VNx16BI FFR_REGNUM)
> >>> +     (unspec:VNx16BI
> >>> +      [(reg:VNx16BI FFRT_REGNUM)
> >>> +       (reg:VNx16BI FFR_REGNUM)
> >>> +       (match_operand 0 "" "X")
> >>> +       (match_operand 1 "" "X")
> >>> +       (match_operand 2 "" "X")
> >>> +       (match_operand 3 "" "X")
> >>> +       (match_operand 4 "" "X")
> >>> +      ] UNSPEC_UPDATE_FFR)
> >>> +     )]
> >>>     "TARGET_SVE"
> >>>     ""
> >>>     [(set_attr "type" "no_insn")]
> >>> @@ -1448,7 +1469,26 @@ (define_insn_and_rewrite
> >>> "*aarch64_load_<ANY_EXTEND:optab>_mov<SVE_HSDI:mode><SV
> >>>   ;; -------------------------------------------------------------------------
> >>>
> >>>   ;; Contiguous non-extending first-faulting or non-faulting loads.
> >>> -(define_insn "@aarch64_ld<fn>f1<mode>"
> >>> +(define_expand "@aarch64_ld<fn>f1<mode>"
> >>> +  [(set (reg:VNx16BI FFR_REGNUM) ; Update the FFR
> >>> +     (unspec:VNx16BI
> >>> +      [(reg:VNx16BI FFRT_REGNUM)
> >>> +       (reg:VNx16BI FFR_REGNUM)
> >>> +       (match_dup 2)
> >>> +       (match_dup 1)
> >>> +      ] UNSPEC_UPDATE_FFR)
> >>> +     )
> >>> +   (set (match_operand:SVE_FULL 0 "register_operand")
> >>> +     (unspec:SVE_FULL
> >>> +       [(match_operand:<VPRED> 2 "register_operand")
> >>> +        (match_operand:SVE_FULL 1 "aarch64_sve_ld<fn>f1_operand")
> >>> +        (reg:VNx16BI FFRT_REGNUM)]
> >>> +       SVE_LDFF1_LDNF1))]
> >>> +  "TARGET_SVE && TARGET_NON_STREAMING"
> >>> +  {}
> >>> +)
> >>> +
> >>> +(define_insn "*aarch64_ld<fn>f1<mode>"
> >>>     [(set (match_operand:SVE_FULL 0 "register_operand" "=w")
> >>>         (unspec:SVE_FULL
> >>>           [(match_operand:<VPRED> 2 "register_operand" "Upl")
> >>> @@ -1479,7 +1519,36 @@ (define_insn "@aarch64_ld<fn>f1<mode>"
> >>>   ;; -------------------------------------------------------------------------
> >>>
> >>>   ;; Predicated first-faulting or non-faulting load and extend.
> >>> -(define_insn_and_rewrite
> >>>
> "@aarch64_ld<fn>f1_<ANY_EXTEND:optab><SVE_HSDI:mode><SVE_PARTIAL
> >>> _I:mode>"
> >>> +(define_expand
> >>>
> "@aarch64_ld<fn>f1_<ANY_EXTEND:optab><SVE_HSDI:mode><SVE_PARTIAL
> >>> _I:mode>"
> >>> +  [(set (reg:VNx16BI FFR_REGNUM) ; Update the FFR
> >>> +     (unspec:VNx16BI
> >>> +      [(reg:VNx16BI FFRT_REGNUM)
> >>> +       (reg:VNx16BI FFR_REGNUM)
> >>> +       (match_dup 2)
> >>> +       (match_dup 1)
> >>> +      ] UNSPEC_UPDATE_FFR)
> >>> +     )
> >>> +   (set (match_operand:SVE_HSDI 0 "register_operand")
> >>> +     (unspec:SVE_HSDI
> >>> +       [(match_operand:<SVE_HSDI:VPRED> 3 "general_operand")
> >>> +        (ANY_EXTEND:SVE_HSDI
> >>> +          (unspec:SVE_PARTIAL_I
> >>> +            [(match_operand:<SVE_PARTIAL_I:VPRED> 2 "register_operand")
> >>> +             (match_operand:SVE_PARTIAL_I 1
> >>> "aarch64_sve_ld<fn>f1_operand")
> >>> +             (reg:VNx16BI FFRT_REGNUM)]
> >>> +            SVE_LDFF1_LDNF1))]
> >>> +       UNSPEC_PRED_X))]
> >>> +  "TARGET_SVE
> >>> +   && TARGET_NON_STREAMING
> >>> +   && (~<SVE_HSDI:narrower_mask> & <SVE_PARTIAL_I:self_mask>) ==
> 0"
> >>> +  {
> >>> +    if (!CONSTANT_P (operands[3]))
> >>> +      operands[3] = CONSTM1_RTX (<SVE_HSDI:VPRED>mode);
> >>> +  }
> >>> +)
> >>> +
> >>> +;; Predicated first-faulting or non-faulting load and extend.
> >>> +(define_insn
> >>>
> "*aarch64_ld<fn>f1_<ANY_EXTEND:optab><SVE_HSDI:mode><SVE_PARTIAL_
> >>> I:mode>"
> >>>     [(set (match_operand:SVE_HSDI 0 "register_operand" "=w")
> >>>         (unspec:SVE_HSDI
> >>>           [(match_operand:<SVE_HSDI:VPRED> 3 "general_operand"
> >>> "UplDnm")
> >>> @@ -1494,10 +1563,6 @@ (define_insn_and_rewrite
> >>> "@aarch64_ld<fn>f1_<ANY_EXTEND:optab><SVE_HSDI:mode><SV
> >>>      && TARGET_NON_STREAMING
> >>>      && (~<SVE_HSDI:narrower_mask> & <SVE_PARTIAL_I:self_mask>) == 0"
> >>>
> >>>
> "ld<fn>f1<ANY_EXTEND:s><SVE_PARTIAL_I:Vesize>\t%0.<SVE_HSDI:Vctype>,
> >>> %2/z, %1"
> >>> -  "&& !CONSTANT_P (operands[3])"
> >>> -  {
> >>> -    operands[3] = CONSTM1_RTX (<SVE_HSDI:VPRED>mode);
> >>> -  }
> >>>     [(set_attr "sve_type" "sve_load_1reg")]
> >>>   )
> >>>
> >>> @@ -1907,7 +1972,33 @@ (define_insn_and_rewrite
> >>> "*aarch64_gather_load_<ANY_EXTEND:optab><SVE_2HSDI:mode
> >>>
> >>>   ;; Predicated first-faulting gather loads for 32-bit elements.  Operand
> >>>   ;; 3 is true for unsigned extension and false for signed extension.
> >>> -(define_insn "@aarch64_ldff1_gather<mode>"
> >>> +(define_expand "@aarch64_ldff1_gather<mode>"
> >>> +  [(set (reg:VNx16BI FFR_REGNUM) ; Update the FFR
> >>> +     (unspec:VNx16BI
> >>> +      [(reg:VNx16BI FFRT_REGNUM)
> >>> +       (reg:VNx16BI FFR_REGNUM)
> >>> +       (match_dup 5)
> >>> +       (match_dup 1)
> >>> +       (match_dup 2)
> >>> +       (match_dup 3)
> >>> +       (match_dup 4)
> >>> +      ] UNSPEC_UPDATE_FFR)
> >>> +     )
> >>> +   (set (match_operand:SVE_FULL_S 0 "register_operand") ; The actual
> load
> >>> +     (unspec:SVE_FULL_S
> >>> +       [(match_operand:VNx4BI 5 "register_operand")
> >>> +        (match_operand:DI 1 "aarch64_sve_gather_offset_w")
> >>> +        (match_operand:VNx4SI 2 "register_operand")
> >>> +        (match_operand:DI 3 "const_int_operand")
> >>> +        (match_operand:DI 4 "aarch64_gather_scale_operand_w")
> >>> +        (mem:BLK (scratch))
> >>> +        (reg:VNx16BI FFRT_REGNUM)]
> >>> +       UNSPEC_LDFF1_GATHER))]
> >>> +  "TARGET_SVE && TARGET_NON_STREAMING"
> >>> +  {}
> >>> +)
> >>> +
> >>> +(define_insn "*aarch64_ldff1_gather<mode>"
> >>>     [(set (match_operand:SVE_FULL_S 0 "register_operand")
> >>>         (unspec:SVE_FULL_S
> >>>           [(match_operand:VNx4BI 5 "register_operand")
> >>> @@ -1938,7 +2029,33 @@ (define_insn
> "@aarch64_ldff1_gather<mode>"
> >>>
> >>>   ;; Predicated first-faulting gather loads for 64-bit elements.  The value
> >>>   ;; of operand 3 doesn't matter in this case.
> >>> -(define_insn "@aarch64_ldff1_gather<mode>"
> >>> +(define_expand "@aarch64_ldff1_gather<mode>"
> >>> +  [(set (reg:VNx16BI FFR_REGNUM) ; Update the FFR
> >>> +     (unspec:VNx16BI
> >>> +      [(reg:VNx16BI FFRT_REGNUM)
> >>> +       (reg:VNx16BI FFR_REGNUM)
> >>> +       (match_dup 5)
> >>> +       (match_dup 1)
> >>> +       (match_dup 2)
> >>> +       (match_dup 3)
> >>> +       (match_dup 4)
> >>> +      ] UNSPEC_UPDATE_FFR)
> >>> +     )
> >>> +   (set (match_operand:SVE_FULL_D 0 "register_operand")
> >>> +     (unspec:SVE_FULL_D
> >>> +       [(match_operand:VNx2BI 5 "register_operand")
> >>> +        (match_operand:DI 1 "aarch64_sve_gather_offset_d")
> >>> +        (match_operand:VNx2DI 2 "register_operand")
> >>> +        (match_operand:DI 3 "const_int_operand")
> >>> +        (match_operand:DI 4 "aarch64_gather_scale_operand_d")
> >>> +        (mem:BLK (scratch))
> >>> +        (reg:VNx16BI FFRT_REGNUM)]
> >>> +       UNSPEC_LDFF1_GATHER))]
> >>> +  "TARGET_SVE && TARGET_NON_STREAMING"
> >>> +  {}
> >>> +)
> >>> +
> >>> +(define_insn "*aarch64_ldff1_gather<mode>"
> >>>     [(set (match_operand:SVE_FULL_D 0 "register_operand")
> >>>         (unspec:SVE_FULL_D
> >>>           [(match_operand:VNx2BI 5 "register_operand")
> >>> @@ -2032,7 +2149,40 @@ (define_insn
> >>> "*aarch64_ldff1_gather<mode>_uxtw"
> >>>
> >>>   ;; Predicated extending first-faulting gather loads for 32-bit elements.
> >>>   ;; Operand 3 is true for unsigned extension and false for signed extension.
> >>> -(define_insn_and_rewrite
> >>>
> "@aarch64_ldff1_gather_<ANY_EXTEND:optab><VNx4_WIDE:mode><VNx4_
> >>> NARROW:mode>"
> >>> +(define_expand
> >>>
> "@aarch64_ldff1_gather_<ANY_EXTEND:optab><VNx4_WIDE:mode><VNx4_
> >>> NARROW:mode>"
> >>> +  [(set (reg:VNx16BI FFR_REGNUM) ; Update the FFR
> >>> +     (unspec:VNx16BI
> >>> +      [(reg:VNx16BI FFRT_REGNUM)
> >>> +       (reg:VNx16BI FFR_REGNUM)
> >>> +       (match_dup 5)
> >>> +       (match_dup 1)
> >>> +       (match_dup 2)
> >>> +       (match_dup 3)
> >>> +       (match_dup 4)
> >>> +      ] UNSPEC_UPDATE_FFR)
> >>> +     )
> >>> +   (set (match_operand:VNx4_WIDE 0 "register_operand")
> >>> +     (unspec:VNx4_WIDE
> >>> +       [(match_operand:VNx4BI 6 "general_operand")
> >>> +        (ANY_EXTEND:VNx4_WIDE
> >>> +          (unspec:VNx4_NARROW
> >>> +            [(match_operand:VNx4BI 5 "register_operand")
> >>> +             (match_operand:DI 1
> >>> "aarch64_sve_gather_offset_<VNx4_NARROW:Vesize>")
> >>> +             (match_operand:VNx4_WIDE 2 "register_operand")
> >>> +             (match_operand:DI 3 "const_int_operand")
> >>> +             (match_operand:DI 4
> >>> "aarch64_gather_scale_operand_<VNx4_NARROW:Vesize>")
> >>> +             (mem:BLK (scratch))
> >>> +             (reg:VNx16BI FFRT_REGNUM)]
> >>> +            UNSPEC_LDFF1_GATHER))]
> >>> +       UNSPEC_PRED_X))]
> >>> +  "TARGET_SVE && TARGET_NON_STREAMING"
> >>> +  {
> >>> +    if (!CONSTANT_P (operands[6]))
> >>> +      operands[6] = CONSTM1_RTX (VNx4BImode);
> >>> +  }
> >>> +)
> >>> +
> >>> +(define_insn
> >>>
> "*aarch64_ldff1_gather_<ANY_EXTEND:optab><VNx4_WIDE:mode><VNx4_N
> >>> ARROW:mode>"
> >>>     [(set (match_operand:VNx4_WIDE 0 "register_operand")
> >>>         (unspec:VNx4_WIDE
> >>>           [(match_operand:VNx4BI 6 "general_operand")
> >>> @@ -2062,16 +2212,34 @@ (define_insn_and_rewrite
> >>> "@aarch64_ldff1_gather_<ANY_EXTEND:optab><VNx4_WIDE:mod
> >>>        [&w, rk,                     w, Ui1, i,   Upl, UplDnm]
> >>> ldff1<ANY_EXTEND:s><VNx4_NARROW:Vesize>\t%0.s, %5/z, [%1, %2.s,
> uxtw
> >>> %p4]
> >>>        [?w, rk,                     0, Ui1, i,   Upl, UplDnm] ^
> >>>     }
> >>> -  "&& !CONSTANT_P (operands[6])"
> >>> -  {
> >>> -    operands[6] = CONSTM1_RTX (VNx4BImode);
> >>> -  }
> >>>     [(set_attr "sve_type" "sve_gatherload_32")]
> >>>   )
> >>>
> >>>   ;; Predicated extending first-faulting gather loads for 64-bit elements.
> >>>   ;; The value of operand 3 doesn't matter in this case.
> >>> -(define_insn_and_rewrite
> >>>
> "@aarch64_ldff1_gather_<ANY_EXTEND:optab><VNx2_WIDE:mode><VNx2_
> >>> NARROW:mode>"
> >>> +(define_expand
> >>>
> "@aarch64_ldff1_gather_<ANY_EXTEND:optab><VNx2_WIDE:mode><VNx2_
> >>> NARROW:mode>"
> >>> +  [(set (match_operand:VNx2_WIDE 0 "register_operand")
> >>> +     (unspec:VNx2_WIDE
> >>> +       [(match_operand:VNx2BI 6 "general_operand")
> >>> +        (ANY_EXTEND:VNx2_WIDE
> >>> +          (unspec:VNx2_NARROW
> >>> +            [(match_operand:VNx2BI 5 "register_operand")
> >>> +             (match_operand:DI 1
> >>> "aarch64_sve_gather_offset_<VNx2_NARROW:Vesize>")
> >>> +             (match_operand:VNx2_WIDE 2 "register_operand")
> >>> +             (match_operand:DI 3 "const_int_operand")
> >>> +             (match_operand:DI 4
> >>> "aarch64_gather_scale_operand_<VNx2_NARROW:Vesize>")
> >>> +             (mem:BLK (scratch))
> >>> +             (reg:VNx16BI FFRT_REGNUM)]
> >>> +            UNSPEC_LDFF1_GATHER))]
> >>> +       UNSPEC_PRED_X))]
> >>> +  "TARGET_SVE && TARGET_NON_STREAMING"
> >>> +  {
> >>> +    if (!CONSTANT_P (operands[6]))
> >>> +      operands[6] = CONSTM1_RTX (VNx2BImode);
> >>> +  }
> >>> +)
> >>> +
> >>> +(define_insn
> >>>
> "*aarch64_ldff1_gather_<ANY_EXTEND:optab><VNx2_WIDE:mode><VNx2_N
> >>> ARROW:mode>"
> >>>     [(set (match_operand:VNx2_WIDE 0 "register_operand")
> >>>         (unspec:VNx2_WIDE
> >>>           [(match_operand:VNx2BI 6 "general_operand")
> >>> @@ -2097,10 +2265,6 @@ (define_insn_and_rewrite
> >>> "@aarch64_ldff1_gather_<ANY_EXTEND:optab><VNx2_WIDE:mod
> >>>        [&w, rk,                     w, i, i,   Upl, UplDnm]
> >>> ldff1<ANY_EXTEND:s><VNx2_NARROW:Vesize>\t%0.d, %5/z, [%1, %2.d,
> lsl
> >>> %p4]
> >>>        [?w, rk,                     w, i, i,   Upl, UplDnm] ^
> >>>     }
> >>> -  "&& !CONSTANT_P (operands[6])"
> >>> -  {
> >>> -    operands[6] = CONSTM1_RTX (VNx2BImode);
> >>> -  }
> >>>     [(set_attr "sve_type" "sve_gatherload_64")]
> >>>   )
> >>>
> >>> diff --git a/gcc/testsuite/gcc.target/aarch64/sve/pr126629.c
> >>> b/gcc/testsuite/gcc.target/aarch64/sve/pr126629.c
> >>> new file mode 100644
> >>> index 00000000000<tel:00000000000>..64cb3ece330
> >>> --- /dev/null
> >>> +++ b/gcc/testsuite/gcc.target/aarch64/sve/pr126629.c
> >>> @@ -0,0 +1,23 @@
> >>> +/* { dg-do compile } */
> >>> +/* { dg-options "-O3" } */
> >>> +
> >>> +#include <arm_sve.h>
> >>> +
> >>> + void
> >>> +two_scans (unsigned char *p, unsigned char *q, unsigned long *r1,
> >>> +           unsigned long *r2, unsigned char *o1, unsigned char *o2)
> >>> +{
> >>> +  svbool_t pt = svptrue_b8 ();
> >>> +  svsetffr ();
> >>> +  svuint8_t a = svldff1_u8 (pt, p);
> >>> +  unsigned long n1 = svcntp_b8 (pt, svrdffr ());
> >>> +  svsetffr ();
> >>> +  svuint8_t b = svldff1_u8 (pt, q);
> >>> +  unsigned long n2 = svcntp_b8 (pt, svrdffr ());
> >>> +  svst1_u8 (pt, o1, a);
> >>> +  svst1_u8 (pt, o2, b);
> >>> +  *r1 = n1;
> >>> +  *r2 = n2;
> >>> +}
> >>> +
> >>> +/* { dg-final { scan-assembler-times {\trdffr} 2 } } */
> >>> diff --git a/gcc/testsuite/gcc.target/aarch64/sve/pr126629_extend.c
> >>> b/gcc/testsuite/gcc.target/aarch64/sve/pr126629_extend.c
> >>> new file mode 100644
> >>> index 00000000000<tel:00000000000>..909a9f1e651
> >>> --- /dev/null
> >>> +++ b/gcc/testsuite/gcc.target/aarch64/sve/pr126629_extend.c
> >>> @@ -0,0 +1,23 @@
> >>> +/* { dg-do compile } */
> >>> +/* { dg-options "-O3" } */
> >>> +
> >>> +#include <arm_sve.h>
> >>> +
> >>> + void
> >>> +two_scans (unsigned char *p, unsigned char *q, unsigned long *r1,
> >>> +           unsigned long *r2, unsigned int *o1, unsigned int *o2)
> >>> +{
> >>> +  svbool_t pt = svptrue_b8 ();
> >>> +  svsetffr ();
> >>> +  svuint32_t a = svldff1ub_u32 (pt, p);
> >>> +  unsigned long n1 = svcntp_b8 (pt, svrdffr ());
> >>> +  svsetffr ();
> >>> +  svuint32_t b = svldff1ub_u32 (pt, q);
> >>> +  unsigned long n2 = svcntp_b8 (pt, svrdffr ());
> >>> +  svst1_u32 (pt, o1, a);
> >>> +  svst1_u32 (pt, o2, b);
> >>> +  *r1 = n1;
> >>> +  *r2 = n2;
> >>> +}
> >>> +
> >>> +/* { dg-final { scan-assembler-times {\trdffr} 2 } } */
> >>> diff --git a/gcc/testsuite/gcc.target/aarch64/sve/pr126629_gather.c
> >>> b/gcc/testsuite/gcc.target/aarch64/sve/pr126629_gather.c
> >>> new file mode 100644
> >>> index 00000000000<tel:00000000000>..2348082fde5
> >>> --- /dev/null
> >>> +++ b/gcc/testsuite/gcc.target/aarch64/sve/pr126629_gather.c
> >>> @@ -0,0 +1,24 @@
> >>> +/* { dg-do compile } */
> >>> +/* { dg-options "-O3" } */
> >>> +
> >>> +#include <arm_sve.h>
> >>> +
> >>> + void
> >>> +two_scans (unsigned char *p, unsigned char *q, unsigned long *r1,
> >>> +           unsigned long *r2, unsigned *o1, unsigned *o2,
> >>> +           svuint32_t offset1, svuint32_t offset2)
> >>> +{
> >>> +  svbool_t pt = svptrue_b8 ();
> >>> +  svsetffr ();
> >>> +  svuint32_t a = svldff1sb_gather_offset_u32 (pt, p, offset1);
> >>> +  unsigned long n1 = svcntp_b8 (pt, svrdffr ());
> >>> +  svsetffr ();
> >>> +  svuint32_t b = svldff1sb_gather_offset_u32 (pt, q, offset2);
> >>> +  unsigned long n2 = svcntp_b8 (pt, svrdffr ());
> >>> +  svst1_u32 (pt, o1, a);
> >>> +  svst1_u32 (pt, o2, b);
> >>> +  *r1 = n1;
> >>> +  *r2 = n2;
> >>> +}
> >>> +
> >>> +/* { dg-final { scan-assembler-times {\trdffr} 2 } } */
> >>> diff --git a/gcc/testsuite/gcc.target/aarch64/sve/pr126629_gather2.c
> >>> b/gcc/testsuite/gcc.target/aarch64/sve/pr126629_gather2.c
> >>> new file mode 100644
> >>> index 00000000000<tel:00000000000>..bb9257fe382
> >>> --- /dev/null
> >>> +++ b/gcc/testsuite/gcc.target/aarch64/sve/pr126629_gather2.c
> >>> @@ -0,0 +1,24 @@
> >>> +/* { dg-do compile } */
> >>> +/* { dg-options "-O3" } */
> >>> +
> >>> +#include <arm_sve.h>
> >>> +
> >>> + void
> >>> +two_scans (svuint32_t p, svuint32_t q, unsigned long *r1,
> >>> +           unsigned long *r2, unsigned *o1, unsigned *o2,
> >>> +           long int offset1, long int offset2)
> >>> +{
> >>> +  svbool_t pt = svptrue_b8 ();
> >>> +  svsetffr ();
> >>> +  svuint32_t a = svldff1sb_gather_u32base_offset_u32 (pt, p, offset1);
> >>> +  unsigned long n1 = svcntp_b8 (pt, svrdffr ());
> >>> +  svsetffr ();
> >>> +  svuint32_t b = svldff1sb_gather_u32base_offset_u32 (pt, q, offset2);
> >>> +  unsigned long n2 = svcntp_b8 (pt, svrdffr ());
> >>> +  svst1_u32 (pt, o1, a);
> >>> +  svst1_u32 (pt, o2, b);
> >>> +  *r1 = n1;
> >>> +  *r2 = n2;
> >>> +}
> >>> +
> >>> +/* { dg-final { scan-assembler-times {\trdffr} 2 } } */
> >>> --
> >>> 2.34.1
> >>
> >