Re: [PATCH] aarch64: Use SVE SPLICE for scalable recurrences
Kyrylo Tkachov <[email protected]>
| Newsgroups | gmane.comp.gcc.patches |
|---|---|
| Message-ID | <[email protected]> |
Hi Tamar, > On 20 Aug 2026, at 15:52, Tamar Christina <[email protected]> wrote: > >> -----Original Message----- >> From: [email protected] <[email protected]> >> Sent: 20 August 2026 13:34 >> To: [email protected] >> Cc: Tamar Christina <[email protected]>; Kyrylo Tkachov >> <[email protected]> >> Subject: [PATCH] aarch64: Use SVE SPLICE for scalable recurrences >> >> From: Kyrylo Tkachov <[email protected]> >> >> The vectorizer represents a first-order recurrence as a scalable >> permutation that takes the final D elements from one vector and the >> remaining elements from the next vector. The AArch64 permutation >> expander did not recognize this variable-length shape. >> >> Recognize the exact two-input permutation and build a predicate for D >> elements. Reverse that predicate and use the SVE SPLICE pattern. Reject >> fixed-length vectors, rotations, and distances that cannot be represented >> at the minimum vector length. >> >> The compile test checks seven direct SPLICE sequences across byte, >> halfword, word, and doubleword modes. A 67-element runtime test covers >> distances 1 and 2 with two seed values, and partial final vectors. > > Nice! > >> >> On an example testcase: >> >> typedef __UINT32_TYPE__ uint32_t; >> >> void >> recur (const uint32_t *__restrict a, uint32_t *__restrict b, >> const uint32_t *__restrict init) >> { >> uint32_t prev = *init; >> for (int i = 0; i < 67; ++i) >> { >> b[i] = a[i] - prev; >> prev = a[i]; >> } >> } >> >> With -O3 -march=armv8.2-a+sve >> We emitted before: >> >> ldr w4, [x2] >> mov w3, 0 >> .p2align 5,,15 >> .L2: >> mov w2, w4 >> ldr w4, [x0, x3] >> sub w2, w4, w2 >> str w2, [x1, x3] >> add x3, x3, 4 >> cmp x3, 268 >> bne .L2 >> >> And after this patch: >> mov w3, 0 >> cntw x5 >> ptrue p5.b, all >> mov w4, 67 >> ptrue p6.b, vl1 >> mov p7.b, p5.b >> ld1rw z31.s, p5/z, [x2] >> rev p6.s, p6.s >> .p2align 5,,15 >> .L2: >> mov z30.d, z31.d >> ld1w z31.s, p7/z, [x0, x3, lsl 2] >> splice z30.s, p6, z30.s, z31.s >> sub z30.s, z31.s, z30.s >> st1w z30.s, p7, [x1, x3, lsl 2] >> add x3, x3, x5 >> whilelo p7.s, w3, w4 >> b.any .L2 >> >> This also vectorizes some loops in OpenEXR 4.0.0. >> >> Upstream PXR24, RLE, ZIP, and ZIPS tests pass, and the baseline and >> patched encoders produced identical files. A paired exrmetrics run on an >> NVIDIA Vera system measured the following write-time improvements. >> >> Flowers.exr multipart.0001.exr >> PXR24, float 2.28% 4.58% >> PXR24, original 3.22% 8.07% >> RLE 13.52% 23.27% >> ZIP 3.67% 9.59% >> ZIPS 3.15% 5.62% >> >> Bootstrapped and tested on aarch64-none-linux-gnu. >> Ok for trunk? >> Thanks, >> Kyrill >> >> gcc/ChangeLog: >> >> * config/aarch64/aarch64.cc (aarch64_evpc_splice): New function. >> (aarch64_expand_vec_perm_const_1): Use it. >> >> gcc/testsuite/ChangeLog: >> >> * gcc.target/aarch64/sve/perm_splice_1.c: New test. >> * gcc.target/aarch64/sve/perm_splice_1-run.c: Likewise. >> >> Signed-off-by: Kyrylo Tkachov <[email protected]> >> --- >> gcc/config/aarch64/aarch64.cc | 42 +++++++++ >> .../aarch64/sve/perm_splice_1-run.c | 94 +++++++++++++++++++ >> .../gcc.target/aarch64/sve/perm_splice_1.c | 63 +++++++++++++ >> 3 files changed, 199 insertions(+) >> create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1- >> run.c >> create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1.c >> >> diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc >> index fdffb13ad22..2e440ce916d 100644 >> --- a/gcc/config/aarch64/aarch64.cc >> +++ b/gcc/config/aarch64/aarch64.cc >> @@ -28261,6 +28261,46 @@ aarch64_evpc_ext (struct expand_vec_perm_d >> *d) >> return true; >> } >> >> +/* Return true if D describes a scalable-vector permutation that takes the >> + last DIST elements from the first input and the remaining elements from >> + the second input. */ >> + >> +static bool >> +aarch64_evpc_splice (struct expand_vec_perm_d *d) >> +{ >> + poly_int64 nelt = d->perm.length (); >> + HOST_WIDE_INT dist; >> + >> + if (d->vec_flags != VEC_SVE_DATA >> + || d->one_vector_p >> + || nelt.is_constant () >> + || !(nelt - d->perm[0]).is_constant (&dist) >> + || !IN_RANGE (dist, 1, INT_MAX) >> + || !d->perm.series_p (0, 1, nelt - dist, 1)) >> + return false; >> + >> + machine_mode pred_mode = aarch64_sve_pred_mode (d->vmode); >> + if (aarch64_svpattern_for_vl (pred_mode, dist) >> + == AARCH64_NUM_SVPATTERNS) >> + return false; >> + >> + if (d->testing_p) >> + return true; >> + >> + rtx_vector_builder builder (pred_mode, dist, 2); >> + for (HOST_WIDE_INT i = 0; i < dist; ++i) >> + builder.quick_push (CONST1_RTX (BImode)); >> + for (HOST_WIDE_INT i = 0; i < dist; ++i) >> + builder.quick_push (CONST0_RTX (BImode)); >> + >> + rtx head = force_reg (pred_mode, builder.build ()); > > Since you already have the pattern above, why not save the result of the > aarch64_svpattern_for... > > and then replace this builder code with > > rtx head = aarch64_ptrue_reg (pred_mode, pattern); > > which is slightly cleaner? There is no aarch64_ptrue_reg overload taking an aarch64_svpattern. The three are (machine_mode), (machine_mode, unsigned int), (machine_mode, machine_mode), so this silently selects the unsigned int one via enum-to-int conversion. If we add one such overload I think it’d be a separate patch that can clean up other opportunities like this around here? Thanks, Kyrill > > LGTM with that change. > > Thanks, > Tamar > >> + rtx pred = gen_reg_rtx (pred_mode); >> + emit_insn (gen_aarch64_sve_rev (pred_mode, pred, head)); >> + emit_insn (gen_aarch64_sve_splice (d->vmode, d->target, pred, >> + d->op0, d->op1)); >> + return true; >> +} >> + >> /* Recognize patterns for the REV{64,32,16} insns, which reverse elements >> within each 64-bit, 32-bit or 16-bit granule. */ >> >> @@ -28786,6 +28826,8 @@ aarch64_expand_vec_perm_const_1 (struct >> expand_vec_perm_d *d) >> return true; >> else if (aarch64_evpc_rev_global (d)) >> return true; >> + else if (aarch64_evpc_splice (d)) >> + return true; >> else if (aarch64_evpc_ext (d)) >> return true; >> else if (aarch64_evpc_dup (d)) >> diff --git a/gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1-run.c >> b/gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1-run.c >> new file mode 100644 >> index 00000000000..b2c13120c53 >> --- /dev/null >> +++ b/gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1-run.c >> @@ -0,0 +1,94 @@ >> +/* { dg-do run } */ >> +/* { dg-require-effective-target aarch64_sve_hw } */ >> +/* { dg-options "-O3 -march=armv8.2-a+sve -mautovec-preference=sve- >> only" } */ >> +/* { dg-additional-options "-msve-vector-bits=scalable" } */ >> +/* { dg-additional-options "-fvect-cost-model=unlimited" } */ >> + >> +typedef __UINT8_TYPE__ uint8_t; >> +typedef __UINT16_TYPE__ uint16_t; >> +typedef __UINT32_TYPE__ uint32_t; >> +typedef __UINT64_TYPE__ uint64_t; >> + >> +#define N 67 >> + >> +#define DEFINE_RECUR(TYPE, SUFFIX) \ >> + __attribute__((noipa)) \ >> + static void \ >> + recur1_##SUFFIX (TYPE *__restrict a, TYPE *__restrict b, \ >> + const TYPE *__restrict init) \ >> + { \ >> + TYPE prev = init[0]; \ >> + for (int i = 0; i < N; ++i) \ >> + { \ >> + b[i] = a[i] - prev; \ >> + prev = a[i]; \ >> + } \ >> + } \ >> + \ >> + __attribute__((noipa)) \ >> + static void \ >> + recur2_##SUFFIX (TYPE *__restrict a, TYPE *__restrict b, \ >> + const TYPE *__restrict init) \ >> + { \ >> + TYPE prev0 = init[0]; \ >> + TYPE prev1 = init[0]; \ >> + for (int i = 0; i + 1 < N; i += 2) \ >> + { \ >> + b[i] = a[i] - prev0; \ >> + prev0 = a[i]; \ >> + b[i + 1] = a[i + 1] - prev1; \ >> + prev1 = a[i + 1]; \ >> + } \ >> + b[N - 1] = 0; \ >> + } >> + >> +DEFINE_RECUR (uint8_t, u8) >> +DEFINE_RECUR (uint16_t, u16) >> +DEFINE_RECUR (uint32_t, u32) >> +DEFINE_RECUR (uint64_t, u64) >> + >> +#define CHECK_RECUR(TYPE, SUFFIX) \ >> + do \ >> + { \ >> + TYPE a[N]; \ >> + TYPE b[N]; \ >> + const TYPE init[2] = { (TYPE) 19, (TYPE) 43 }; \ >> + \ >> + for (int i = 0; i < N; ++i) \ >> + a[i] = (TYPE) ((unsigned int) i * (unsigned int) i \ >> + + 5U * (unsigned int) i + 11U); \ >> + \ >> + for (int seed = 0; seed < 2; ++seed) \ >> + { \ >> + recur1_##SUFFIX (a, b, &init[seed]); \ >> + if (b[0] != (TYPE) (a[0] - init[seed])) \ >> + __builtin_abort (); \ >> + _Pragma ("GCC novector") \ >> + for (int i = 1; i < N; ++i) \ >> + if (b[i] != (TYPE) (a[i] - a[i - 1])) \ >> + __builtin_abort (); \ >> + \ >> + recur2_##SUFFIX (a, b, &init[seed]); \ >> + if (b[0] != (TYPE) (a[0] - init[seed]) \ >> + || b[1] != (TYPE) (a[1] - init[seed])) \ >> + __builtin_abort (); \ >> + _Pragma ("GCC novector") \ >> + for (int i = 2; i < N - 1; ++i) \ >> + if (b[i] != (TYPE) (a[i] - a[i - 2])) \ >> + __builtin_abort (); \ >> + if (b[N - 1] != 0) \ >> + __builtin_abort (); \ >> + } \ >> + } \ >> + while (0) >> + >> +int >> +main (void) >> +{ >> + CHECK_RECUR (uint8_t, u8); >> + CHECK_RECUR (uint16_t, u16); >> + CHECK_RECUR (uint32_t, u32); >> + CHECK_RECUR (uint64_t, u64); >> + >> + return 0; >> +} >> diff --git a/gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1.c >> b/gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1.c >> new file mode 100644 >> index 00000000000..c2a9879008b >> --- /dev/null >> +++ b/gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1.c >> @@ -0,0 +1,63 @@ >> +/* { dg-do compile } */ >> +/* { dg-options "-O3 -march=armv8.2-a+sve -mautovec-preference=sve- >> only" } */ >> +/* { dg-additional-options "-msve-vector-bits=scalable" } */ >> +/* { dg-additional-options "-fvect-cost-model=unlimited -fdump-tree-vect- >> details" } */ >> + >> +typedef __UINT8_TYPE__ uint8_t; >> +typedef __UINT16_TYPE__ uint16_t; >> +typedef __UINT32_TYPE__ uint32_t; >> +typedef __UINT64_TYPE__ uint64_t; >> + >> +#define DEF_RECUR1(TYPE, SUFFIX) \ >> + __attribute__((noipa)) \ >> + void \ >> + recur1_##SUFFIX (TYPE *__restrict a, TYPE *__restrict b, \ >> + const TYPE *__restrict init) \ >> + { \ >> + TYPE prev = *init; \ >> + for (int i = 0; i < 67; ++i) \ >> + { \ >> + b[i] = a[i] - prev; \ >> + prev = a[i]; \ >> + } \ >> + } >> + >> +#define DEF_RECUR2(TYPE, SUFFIX) \ >> + __attribute__((noipa)) \ >> + void \ >> + recur2_##SUFFIX (TYPE *__restrict a, TYPE *__restrict b, \ >> + const TYPE *__restrict init) \ >> + { \ >> + TYPE prev0 = *init; \ >> + TYPE prev1 = *init; \ >> + for (int i = 0; i < 66; i += 2) \ >> + { \ >> + b[i] = a[i] - prev0; \ >> + prev0 = a[i]; \ >> + b[i + 1] = a[i + 1] - prev1; \ >> + prev1 = a[i + 1]; \ >> + } \ >> + } >> + >> +DEF_RECUR1 (uint8_t, u8) >> +DEF_RECUR1 (uint16_t, u16) >> +DEF_RECUR1 (uint32_t, u32) >> +DEF_RECUR1 (uint64_t, u64) >> + >> +DEF_RECUR2 (uint8_t, u8) >> +DEF_RECUR2 (uint16_t, u16) >> +DEF_RECUR2 (uint32_t, u32) >> + >> +/* { dg-final { scan-assembler-times {\tptrue\tp[0-9]+\.b, vl1} 4 } } */ >> +/* { dg-final { scan-assembler-times {\tptrue\tp[0-9]+\.b, vl2} 1 } } */ >> +/* { dg-final { scan-assembler-times {\tptrue\tp[0-9]+\.h, vl2} 1 } } */ >> +/* { dg-final { scan-assembler-times {\tptrue\tp[0-9]+\.s, vl2} 1 } } */ >> +/* { dg-final { scan-assembler-times {\trev\tp[0-9]+\.b, p[0-9]+\.b} 2 } } */ >> +/* { dg-final { scan-assembler-times {\trev\tp[0-9]+\.h, p[0-9]+\.h} 2 } } */ >> +/* { dg-final { scan-assembler-times {\trev\tp[0-9]+\.s, p[0-9]+\.s} 2 } } */ >> +/* { dg-final { scan-assembler-times {\trev\tp[0-9]+\.d, p[0-9]+\.d} 1 } } */ >> +/* { dg-final { scan-assembler-times {\tsplice\tz[0-9]+\.b, p[0-9]+, z[0-9]+\.b, >> z[0-9]+\.b} 2 } } */ >> +/* { dg-final { scan-assembler-times {\tsplice\tz[0-9]+\.h, p[0-9]+, z[0-9]+\.h, >> z[0-9]+\.h} 2 } } */ >> +/* { dg-final { scan-assembler-times {\tsplice\tz[0-9]+\.s, p[0-9]+, z[0-9]+\.s, >> z[0-9]+\.s} 2 } } */ >> +/* { dg-final { scan-assembler-times {\tsplice\tz[0-9]+\.d, p[0-9]+, z[0-9]+\.d, >> z[0-9]+\.d} 1 } } */ >> +/* { dg-final { scan-tree-dump-times "vectorized 1 loops in function" 7 "vect" >> } } */ >> -- >> 2.50.1 (Apple Git-155)