[patch v3]middle-end: Teach ranger about POLY_INT_CST ranges for VRP.
Tamar Christina <[email protected]>
| Newsgroups | gmane.comp.gcc.patches |
|---|---|
| Message-ID | <[email protected]> |
A an area where LLVM generates much better code than GCC today is in
optimization of VLA branches guarded by VLA constants.
Normally in GIMPLE POLY_INT_CSTs have no upport and lower bound because well,
they're poly. However SVE has defined minimum and maximum vector sizes in
the architecture[1] and so for us we do have bounds on these constants.
[1] https://developer.arm.com/documentation/102476/0101/Introducing-SVE
Today LLVM optimizes these simple expressions
#include <arm_sve.h>
int g (void)
{
unsigned int vl = svcntb ();
return vl < 257;
}
int h (void)
{
return svcntw () <= 64;
}
away to
g():
mov w0, #1
ret
h():
mov w0, #1
ret
which is right because both are always true for any SVE vector length.
while GCC generates:
g():
cntb x0
cmp w0, 257
cset w0, cc
ret
h():
cntw x0
cmp x0, 65
cset w0, cc
ret
This patch extends ranger with a target hook poly_int_indeterminate_bound which
is different from the current costing only hooks used in the vectorizer because
those hooks can't be relied upon for correctness.
This new hook allows us specify the minimum and maximum bounds of a POLY_INT_CST
and have ranger use it in range_query::get_tree_range though specifying the
indeterminate bound of a POLY target.
This gets GCC to fold away many known true or known false comparisons in
codegen today and we generate much simpler loop pre-headers.
I have kept the AArch64 parts in this patch to get some extra eyes on it, but
will split them out on commit.
Bootstrapped Regtested on aarch64-none-linux-gnu,
arm-none-linux-gnueabihf, x86_64-pc-linux-gnu
-m32, -m64 and no issues.
Ok for master?
Thanks,
Tamar
gcc/ChangeLog:
* config/aarch64/aarch64.cc (aarch64_poly_int_indeterminate_bound): New.
(TARGET_POLY_INT_INDETERMINATE_BOUND ): Implement hook using it.
* target.def (poly_int_indeterminate_bound): New.
* doc/tm.texi.in: Document it.
* doc/tm.texi: Regenerate.
* value-query.cc (range_query::get_tree_range): Use it.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/sve/slp_12.c: Update testcases
* gcc.target/aarch64/sve/cnt_fold_7.c: New test.
* gcc.target/aarch64/sve/cnt_fold_7_run.c: New test.
---
diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index fdffb13ad222b7e86709a2e7caef520dbb77d050..da1e9eb7bc858de8563b2ab4f9e42d67aaf625aa 100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -31283,6 +31283,14 @@ aarch64_estimated_poly_value (poly_int64 val,
return val.coeffs[0] + val.coeffs[1] * over_128 / 128;
}
+/* Implement TARGET_POLY_INT_INDETERMINATE_BOUND. */
+
+static poly_uint64
+aarch64_poly_int_indeterminate_bound ()
+{
+ return poly_uint64 (0, 15);
+}
+
/* Return true for types that could be supported as SIMD return or
argument types. */
@@ -34605,6 +34613,9 @@ aarch64_libgcc_floating_mode_supported_p
#undef TARGET_ESTIMATED_POLY_VALUE
#define TARGET_ESTIMATED_POLY_VALUE aarch64_estimated_poly_value
+#undef TARGET_POLY_INT_INDETERMINATE_BOUND
+#define TARGET_POLY_INT_INDETERMINATE_BOUND aarch64_poly_int_indeterminate_bound
+
#undef TARGET_ATTRIBUTE_TABLE
#define TARGET_ATTRIBUTE_TABLE aarch64_attribute_table
diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi
index 85e175256833316597b0ab944e8c1f9cfe8d68c6..2613b95775bb0ab82174c202bab2ff9ff25ed1ba 100644
--- a/gcc/doc/tm.texi
+++ b/gcc/doc/tm.texi
@@ -7491,6 +7491,14 @@ the @code{POLY_VALUE_MIN}, @code{POLY_VALUE_MAX} and
implementation returns the lowest possible value of @var{val}.
@end deftypefn
+@deftypefn {Target Hook} poly_uint64 TARGET_POLY_INT_INDETERMINATE_BOUND (void)
+Return a @code{poly_uint64} whose coefficients give conservative
+inclusive upper bounds for the corresponding @code{poly_int}
+indeterminates. The constant coefficient is ignored and should be zero.
+This hook is used for correctness rather than for costing, so the returned
+bounds must be safe for all architecturally-valid runtime values.
+@end deftypefn
+
@deftypefn {Target Hook} bool TARGET_AVOID_STORE_FORWARDING_P (vec<store_fwd_info>, @var{rtx}, @var{int}, @var{bool})
Given a list of stores and a load instruction that reads from the location
of the stores, this hook decides if it's profitable to emit additional code
diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in
index 1a9edd0635d03cc7afbffee8e2ec028d6097a5e1..0d2a76ec6b0b09224aafb7fe68dfa49afb5b12ec 100644
--- a/gcc/doc/tm.texi.in
+++ b/gcc/doc/tm.texi.in
@@ -4794,6 +4794,8 @@ Define this macro if a non-short-circuit operation produced by
@hook TARGET_ESTIMATED_POLY_VALUE
+@hook TARGET_POLY_INT_INDETERMINATE_BOUND
+
@hook TARGET_AVOID_STORE_FORWARDING_P
@node Scheduling
diff --git a/gcc/target.def b/gcc/target.def
index 884fe1bd57e17b3ae92cd052a340f02a2559da31..bf2825e991d619166f281bef006b6f14288e603d 100644
--- a/gcc/target.def
+++ b/gcc/target.def
@@ -4217,6 +4217,16 @@ implementation returns the lowest possible value of @var{val}.",
HOST_WIDE_INT, (poly_int64 val, poly_value_estimate_kind kind),
default_estimated_poly_value)
+DEFHOOK
+(poly_int_indeterminate_bound,
+ "Return a @code{poly_uint64} whose coefficients give conservative\n\
+inclusive upper bounds for the corresponding @code{poly_int}\n\
+indeterminates. The constant coefficient is ignored and should be zero.\n\
+This hook is used for correctness rather than for costing, so the returned\n\
+bounds must be safe for all architecturally-valid runtime values.",
+ poly_uint64, (void),
+ NULL)
+
/* Permit speculative instructions in delay slots during delayed-branch
scheduling. */
DEFHOOK
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/cnt_fold_7.c b/gcc/testsuite/gcc.target/aarch64/sve/cnt_fold_7.c
new file mode 100644
index 0000000000000000000000000000000000000000..d64933527da7e3e85f37dc231355eb4cd30a1422
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/cnt_fold_7.c
@@ -0,0 +1,154 @@
+/* { dg-do compile } */
+/* { dg-options "-O2" } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+#include <arm_sve.h>
+
+/*
+** b_lt_257:
+** mov w0, 1
+** ret
+*/
+int
+b_lt_257 (void)
+{
+ unsigned int vl = svcntb ();
+
+ return vl < 257;
+}
+
+/*
+** h_ge_8:
+** mov w0, 1
+** ret
+*/
+int
+h_ge_8 (void)
+{
+ return svcnth () >= 8;
+}
+
+/*
+** w_le_64:
+** mov w0, 1
+** ret
+*/
+int
+w_le_64 (void)
+{
+ return svcntw () <= 64;
+}
+
+/*
+** d_gt_1:
+** mov w0, 1
+** ret
+*/
+int
+d_gt_1 (void)
+{
+ return svcntd () > 1;
+}
+
+/*
+** b_ne_0:
+** mov w0, 1
+** ret
+*/
+int
+b_ne_0 (void)
+{
+ return svcntb () != 0;
+}
+
+/*
+** b_le_15:
+** mov w0, 0
+** ret
+*/
+int
+b_le_15 (void)
+{
+ return svcntb () <= 15;
+}
+
+/*
+** h_lt_8:
+** mov w0, 0
+** ret
+*/
+int
+h_lt_8 (void)
+{
+ return svcnth () < 8;
+}
+
+/*
+** w_gt_64:
+** mov w0, 0
+** ret
+*/
+int
+w_gt_64 (void)
+{
+ return svcntw () > 64;
+}
+
+/*
+** d_eq_0:
+** mov w0, 0
+** ret
+*/
+int
+d_eq_0 (void)
+{
+ return svcntd () == 0;
+}
+
+/*
+** b_lt_256:
+** cntb x0
+** cmp x0, 256
+** cset w0, cc
+** ret
+*/
+int
+b_lt_256 (void)
+{
+ return svcntb () < 256;
+}
+
+/*
+** w_gt_4:
+** cntw x0
+** cmp x0, 4
+** cset w0, hi
+** ret
+*/
+int
+w_gt_4 (void)
+{
+ return svcntw () > 4;
+}
+
+/*
+** b_pat_all_lt_257:
+** mov w0, 1
+** ret
+*/
+int
+b_pat_all_lt_257 (void)
+{
+ return svcntb_pat (SV_ALL) < 257;
+}
+
+/*
+** w_pat_all_le_64:
+** mov w0, 1
+** ret
+*/
+int
+w_pat_all_le_64 (void)
+{
+ return svcntw_pat (SV_ALL) <= 64;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/cnt_fold_7_run.c b/gcc/testsuite/gcc.target/aarch64/sve/cnt_fold_7_run.c
new file mode 100644
index 0000000000000000000000000000000000000000..59217ab343ec03509bccc6d12098e6264adaa8b1
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/cnt_fold_7_run.c
@@ -0,0 +1,47 @@
+/* { dg-do run { target aarch64_sve_hw } } */
+/* { dg-options "-O2" } */
+
+#include <arm_sve.h>
+
+#define CHECK(EXPR) \
+ do \
+ { \
+ if (!(EXPR)) \
+ __builtin_abort (); \
+ } \
+ while (0)
+
+int
+main (void)
+{
+ unsigned int b = svcntb ();
+ unsigned int h = svcnth ();
+ unsigned int w = svcntw ();
+ unsigned int d = svcntd ();
+
+ CHECK (b >= 16 && b <= 256);
+ CHECK (h >= 8 && h <= 128);
+ CHECK (w >= 4 && w <= 64);
+ CHECK (d >= 2 && d <= 32);
+
+ CHECK (b < 257);
+ CHECK (h >= 8);
+ CHECK (w <= 64);
+ CHECK (d > 1);
+ CHECK (b != 0);
+
+ CHECK (!(b <= 15));
+ CHECK (!(h < 8));
+ CHECK (!(w > 64));
+ CHECK (!(d == 0));
+
+ CHECK ((svcntb () < 256) == (b < 256));
+ CHECK ((svcntw () > 4) == (w > 4));
+
+ CHECK (svcntb_pat (SV_ALL) == b);
+ CHECK (svcntw_pat (SV_ALL) == w);
+ CHECK (svcntb_pat (SV_ALL) < 257);
+ CHECK (svcntw_pat (SV_ALL) <= 64);
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_12.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_12.c
index 6ba4c9651b67d7e2495b58ffc0ed01f563c428b6..c81845f2c87f9e68e7933cded9868d1a732a8854 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/slp_12.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_12.c
@@ -47,14 +47,12 @@ TEST_ALL (VEC_PERM)
/* We should use WHILEs for all accesses. */
/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.b} 20 } } */
-/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.h} 18 } } */
-/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.s} 27 } } */
-/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.d} 24 } } */
+/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.h} 16 } } */
+/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.s} 21 } } */
+/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.d} 15 } } */
/* 6 for the 8-bit types and 2 for the 16-bit types. */
/* { dg-final { scan-assembler-times {\tuqdecb\t} 8 } } */
-/* 4 for the 16-bit types and 3 for the 32-bit types. */
-/* { dg-final { scan-assembler-times {\tuqdech\t} 7 } } */
-/* 6 for the 32-bit types and 3 for the 64-bit types. */
-/* { dg-final { scan-assembler-times {\tuqdecw\t} 9 } } */
-/* { dg-final { scan-assembler-times {\tuqdecd\t} 6 } } */
+/* { dg-final { scan-assembler-times {\tuqdech\t} 2 } } */
+/* { dg-final { scan-assembler-times {\tuqdecw\t} 3 } } */
+/* { dg-final { scan-assembler-not {\tuqdecd\t} } } */
diff --git a/gcc/value-query.cc b/gcc/value-query.cc
index decf4e4d0ac0eddb2dd6657072bf7c62c2e2ed00..53728addefc84b43fc08f1cff680a9affa7c041b 100644
--- a/gcc/value-query.cc
+++ b/gcc/value-query.cc
@@ -27,11 +27,13 @@ along with GCC; see the file COPYING3. If not see
#include "gimple.h"
#include "ssa.h"
#include "tree-pretty-print.h"
+#include "tree-ssanames.h"
#include "fold-const.h"
#include "value-query.h"
#include "alloc-pool.h"
#include "gimple-range.h"
#include "value-range-storage.h"
+#include "target.h"
// range_query default methods.
@@ -404,8 +406,98 @@ range_query::get_tree_range (vrange &r, tree expr, gimple *stmt,
if (POLY_INT_CST_P (expr))
{
unsigned int precision = TYPE_PRECISION (type);
- r.set_varying (type);
- r.update_bitmask ({ wi::zero (precision), get_nonzero_bits (expr) });
+ signop sign = TYPE_SIGN (type);
+ bool have_poly_bound = targetm.poly_int_indeterminate_bound;
+ poly_uint64 indeterminate_bound;
+
+ if (have_poly_bound)
+ indeterminate_bound = targetm.poly_int_indeterminate_bound ();
+
+ poly_wide_int val = wi::to_poly_wide (expr);
+ wide_int type_min = wi::to_wide (TYPE_MIN_VALUE (type));
+ wide_int type_max = wi::to_wide (TYPE_MAX_VALUE (type));
+
+ /* Start with the invariant part of the poly-int, then account
+ for each coefficient below.
+
+ The target hook gives a per-coefficient upper bound for the
+ indeterminate. Since those indeterminates are unsigned and
+ nonnegative, a positive coefficient can only increase the upper
+ bound and a negative coefficient can only decrease the lower
+ bound. The opposite bound is unaffected by that coefficient:
+
+ [A, +C] with C >= 0 => max += C * bound
+ [A, -C] with C >= 0 => min -= C * bound. */
+ wide_int bounds[2]
+ = { wide_int::from (val.coeffs[0], precision, sign),
+ wide_int::from (val.coeffs[0], precision, sign) };
+ bool ovf[2] = { false, false };
+
+ for (unsigned int i = 1; i < NUM_POLY_INT_COEFFS; ++i)
+ {
+ wide_int coeff = wide_int::from (val.coeffs[i], precision, sign);
+ if (wi::eq_p (coeff, 0))
+ continue;
+
+ /* Select the only bound affected by this coefficient. A
+ negative coefficient contributes to the minimum and a positive
+ coefficient contributes to the maximum. */
+ bool coeff_neg = wi::neg_p (coeff, sign);
+ wide_int &bound = bounds[coeff_neg ? 0 : 1];
+ bool &bound_ovf = ovf[coeff_neg ? 0 : 1];
+
+ if (bound_ovf)
+ continue;
+
+ /* A missing hook, or a -1 bound for this coefficient, means the
+ indeterminate has no finite target-specific limit. Treat that
+ like an overflow of the affected bound. */
+ if (!have_poly_bound
+ || indeterminate_bound.coeffs[i] == HOST_WIDE_INT_M1U)
+ bound_ovf = true;
+ else
+ {
+ wide_int indeterminate
+ = wi::uhwi (indeterminate_bound.coeffs[i], precision);
+ wi::overflow_type mul_ovf = wi::OVF_NONE;
+ wide_int term = wi::mul (coeff, indeterminate, sign,
+ &mul_ovf);
+ wi::overflow_type add_ovf = wi::OVF_NONE;
+ bound = wi::add (bound, term, sign, &add_ovf);
+ bound_ovf = (mul_ovf != wi::OVF_NONE
+ || add_ovf != wi::OVF_NONE);
+ }
+
+ if (TYPE_OVERFLOW_WRAPS (type) && bound_ovf)
+ {
+ r.set_varying (type);
+ return true;
+ }
+
+ if (bound_ovf)
+ {
+ if (coeff_neg)
+ bounds[0] = type_min;
+ else
+ bounds[1] = type_max;
+ }
+ }
+
+ if (!wi::le_p (bounds[0], bounds[1], sign))
+ {
+ r.set_varying (type);
+ return true;
+ }
+
+ irange &ir = as_a <irange> (r);
+ ir.set (type, bounds[0], bounds[1]);
+
+ /* Preserve alignment/step information that is not visible in the
+ intervals. For example, a poly-int like [8, 8] can
+ only produce multiples of 8, but the interval range might be
+ [8, 136], which also contains values with low bits set. */
+ ir.update_bitmask (irange_bitmask (wi::zero (precision),
+ get_nonzero_bits (expr)));
return true;
}
break;
--
rb20685.patch
(text/x-diff, 11.8 KB)
diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index fdffb13ad222b7e86709a2e7caef520dbb77d050..da1e9eb7bc858de8563b2ab4f9e42d67aaf625aa 100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -31283,6 +31283,14 @@ aarch64_estimated_poly_value (poly_int64 val,
return val.coeffs[0] + val.coeffs[1] * over_128 / 128;
}
+/* Implement TARGET_POLY_INT_INDETERMINATE_BOUND. */
+
+static poly_uint64
+aarch64_poly_int_indeterminate_bound ()
+{
+ return poly_uint64 (0, 15);
+}
+
/* Return true for types that could be supported as SIMD return or
argument types. */
@@ -34605,6 +34613,9 @@ aarch64_libgcc_floating_mode_supported_p
#undef TARGET_ESTIMATED_POLY_VALUE
#define TARGET_ESTIMATED_POLY_VALUE aarch64_estimated_poly_value
+#undef TARGET_POLY_INT_INDETERMINATE_BOUND
+#define TARGET_POLY_INT_INDETERMINATE_BOUND aarch64_poly_int_indeterminate_bound
+
#undef TARGET_ATTRIBUTE_TABLE
#define TARGET_ATTRIBUTE_TABLE aarch64_attribute_table
diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi
index 85e175256833316597b0ab944e8c1f9cfe8d68c6..2613b95775bb0ab82174c202bab2ff9ff25ed1ba 100644
--- a/gcc/doc/tm.texi
+++ b/gcc/doc/tm.texi
@@ -7491,6 +7491,14 @@ the @code{POLY_VALUE_MIN}, @code{POLY_VALUE_MAX} and
implementation returns the lowest possible value of @var{val}.
@end deftypefn
+@deftypefn {Target Hook} poly_uint64 TARGET_POLY_INT_INDETERMINATE_BOUND (void)
+Return a @code{poly_uint64} whose coefficients give conservative
+inclusive upper bounds for the corresponding @code{poly_int}
+indeterminates. The constant coefficient is ignored and should be zero.
+This hook is used for correctness rather than for costing, so the returned
+bounds must be safe for all architecturally-valid runtime values.
+@end deftypefn
+
@deftypefn {Target Hook} bool TARGET_AVOID_STORE_FORWARDING_P (vec<store_fwd_info>, @var{rtx}, @var{int}, @var{bool})
Given a list of stores and a load instruction that reads from the location
of the stores, this hook decides if it's profitable to emit additional code
diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in
index 1a9edd0635d03cc7afbffee8e2ec028d6097a5e1..0d2a76ec6b0b09224aafb7fe68dfa49afb5b12ec 100644
--- a/gcc/doc/tm.texi.in
+++ b/gcc/doc/tm.texi.in
@@ -4794,6 +4794,8 @@ Define this macro if a non-short-circuit operation produced by
@hook TARGET_ESTIMATED_POLY_VALUE
+@hook TARGET_POLY_INT_INDETERMINATE_BOUND
+
@hook TARGET_AVOID_STORE_FORWARDING_P
@node Scheduling
diff --git a/gcc/target.def b/gcc/target.def
index 884fe1bd57e17b3ae92cd052a340f02a2559da31..bf2825e991d619166f281bef006b6f14288e603d 100644
--- a/gcc/target.def
+++ b/gcc/target.def
@@ -4217,6 +4217,16 @@ implementation returns the lowest possible value of @var{val}.",
HOST_WIDE_INT, (poly_int64 val, poly_value_estimate_kind kind),
default_estimated_poly_value)
+DEFHOOK
+(poly_int_indeterminate_bound,
+ "Return a @code{poly_uint64} whose coefficients give conservative\n\
+inclusive upper bounds for the corresponding @code{poly_int}\n\
+indeterminates. The constant coefficient is ignored and should be zero.\n\
+This hook is used for correctness rather than for costing, so the returned\n\
+bounds must be safe for all architecturally-valid runtime values.",
+ poly_uint64, (void),
+ NULL)
+
/* Permit speculative instructions in delay slots during delayed-branch
scheduling. */
DEFHOOK
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/cnt_fold_7.c b/gcc/testsuite/gcc.target/aarch64/sve/cnt_fold_7.c
new file mode 100644
index 0000000000000000000000000000000000000000..d64933527da7e3e85f37dc231355eb4cd30a1422
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/cnt_fold_7.c
@@ -0,0 +1,154 @@
+/* { dg-do compile } */
+/* { dg-options "-O2" } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+#include <arm_sve.h>
+
+/*
+** b_lt_257:
+** mov w0, 1
+** ret
+*/
+int
+b_lt_257 (void)
+{
+ unsigned int vl = svcntb ();
+
+ return vl < 257;
+}
+
+/*
+** h_ge_8:
+** mov w0, 1
+** ret
+*/
+int
+h_ge_8 (void)
+{
+ return svcnth () >= 8;
+}
+
+/*
+** w_le_64:
+** mov w0, 1
+** ret
+*/
+int
+w_le_64 (void)
+{
+ return svcntw () <= 64;
+}
+
+/*
+** d_gt_1:
+** mov w0, 1
+** ret
+*/
+int
+d_gt_1 (void)
+{
+ return svcntd () > 1;
+}
+
+/*
+** b_ne_0:
+** mov w0, 1
+** ret
+*/
+int
+b_ne_0 (void)
+{
+ return svcntb () != 0;
+}
+
+/*
+** b_le_15:
+** mov w0, 0
+** ret
+*/
+int
+b_le_15 (void)
+{
+ return svcntb () <= 15;
+}
+
+/*
+** h_lt_8:
+** mov w0, 0
+** ret
+*/
+int
+h_lt_8 (void)
+{
+ return svcnth () < 8;
+}
+
+/*
+** w_gt_64:
+** mov w0, 0
+** ret
+*/
+int
+w_gt_64 (void)
+{
+ return svcntw () > 64;
+}
+
+/*
+** d_eq_0:
+** mov w0, 0
+** ret
+*/
+int
+d_eq_0 (void)
+{
+ return svcntd () == 0;
+}
+
+/*
+** b_lt_256:
+** cntb x0
+** cmp x0, 256
+** cset w0, cc
+** ret
+*/
+int
+b_lt_256 (void)
+{
+ return svcntb () < 256;
+}
+
+/*
+** w_gt_4:
+** cntw x0
+** cmp x0, 4
+** cset w0, hi
+** ret
+*/
+int
+w_gt_4 (void)
+{
+ return svcntw () > 4;
+}
+
+/*
+** b_pat_all_lt_257:
+** mov w0, 1
+** ret
+*/
+int
+b_pat_all_lt_257 (void)
+{
+ return svcntb_pat (SV_ALL) < 257;
+}
+
+/*
+** w_pat_all_le_64:
+** mov w0, 1
+** ret
+*/
+int
+w_pat_all_le_64 (void)
+{
+ return svcntw_pat (SV_ALL) <= 64;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/cnt_fold_7_run.c b/gcc/testsuite/gcc.target/aarch64/sve/cnt_fold_7_run.c
new file mode 100644
index 0000000000000000000000000000000000000000..59217ab343ec03509bccc6d12098e6264adaa8b1
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/cnt_fold_7_run.c
@@ -0,0 +1,47 @@
+/* { dg-do run { target aarch64_sve_hw } } */
+/* { dg-options "-O2" } */
+
+#include <arm_sve.h>
+
+#define CHECK(EXPR) \
+ do \
+ { \
+ if (!(EXPR)) \
+ __builtin_abort (); \
+ } \
+ while (0)
+
+int
+main (void)
+{
+ unsigned int b = svcntb ();
+ unsigned int h = svcnth ();
+ unsigned int w = svcntw ();
+ unsigned int d = svcntd ();
+
+ CHECK (b >= 16 && b <= 256);
+ CHECK (h >= 8 && h <= 128);
+ CHECK (w >= 4 && w <= 64);
+ CHECK (d >= 2 && d <= 32);
+
+ CHECK (b < 257);
+ CHECK (h >= 8);
+ CHECK (w <= 64);
+ CHECK (d > 1);
+ CHECK (b != 0);
+
+ CHECK (!(b <= 15));
+ CHECK (!(h < 8));
+ CHECK (!(w > 64));
+ CHECK (!(d == 0));
+
+ CHECK ((svcntb () < 256) == (b < 256));
+ CHECK ((svcntw () > 4) == (w > 4));
+
+ CHECK (svcntb_pat (SV_ALL) == b);
+ CHECK (svcntw_pat (SV_ALL) == w);
+ CHECK (svcntb_pat (SV_ALL) < 257);
+ CHECK (svcntw_pat (SV_ALL) <= 64);
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_12.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_12.c
index 6ba4c9651b67d7e2495b58ffc0ed01f563c428b6..c81845f2c87f9e68e7933cded9868d1a732a8854 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/slp_12.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_12.c
@@ -47,14 +47,12 @@ TEST_ALL (VEC_PERM)
/* We should use WHILEs for all accesses. */
/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.b} 20 } } */
-/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.h} 18 } } */
-/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.s} 27 } } */
-/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.d} 24 } } */
+/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.h} 16 } } */
+/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.s} 21 } } */
+/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7]\.d} 15 } } */
/* 6 for the 8-bit types and 2 for the 16-bit types. */
/* { dg-final { scan-assembler-times {\tuqdecb\t} 8 } } */
-/* 4 for the 16-bit types and 3 for the 32-bit types. */
-/* { dg-final { scan-assembler-times {\tuqdech\t} 7 } } */
-/* 6 for the 32-bit types and 3 for the 64-bit types. */
-/* { dg-final { scan-assembler-times {\tuqdecw\t} 9 } } */
-/* { dg-final { scan-assembler-times {\tuqdecd\t} 6 } } */
+/* { dg-final { scan-assembler-times {\tuqdech\t} 2 } } */
+/* { dg-final { scan-assembler-times {\tuqdecw\t} 3 } } */
+/* { dg-final { scan-assembler-not {\tuqdecd\t} } } */
diff --git a/gcc/value-query.cc b/gcc/value-query.cc
index decf4e4d0ac0eddb2dd6657072bf7c62c2e2ed00..53728addefc84b43fc08f1cff680a9affa7c041b 100644
--- a/gcc/value-query.cc
+++ b/gcc/value-query.cc
@@ -27,11 +27,13 @@ along with GCC; see the file COPYING3. If not see
#include "gimple.h"
#include "ssa.h"
#include "tree-pretty-print.h"
+#include "tree-ssanames.h"
#include "fold-const.h"
#include "value-query.h"
#include "alloc-pool.h"
#include "gimple-range.h"
#include "value-range-storage.h"
+#include "target.h"
// range_query default methods.
@@ -404,8 +406,98 @@ range_query::get_tree_range (vrange &r, tree expr, gimple *stmt,
if (POLY_INT_CST_P (expr))
{
unsigned int precision = TYPE_PRECISION (type);
- r.set_varying (type);
- r.update_bitmask ({ wi::zero (precision), get_nonzero_bits (expr) });
+ signop sign = TYPE_SIGN (type);
+ bool have_poly_bound = targetm.poly_int_indeterminate_bound;
+ poly_uint64 indeterminate_bound;
+
+ if (have_poly_bound)
+ indeterminate_bound = targetm.poly_int_indeterminate_bound ();
+
+ poly_wide_int val = wi::to_poly_wide (expr);
+ wide_int type_min = wi::to_wide (TYPE_MIN_VALUE (type));
+ wide_int type_max = wi::to_wide (TYPE_MAX_VALUE (type));
+
+ /* Start with the invariant part of the poly-int, then account
+ for each coefficient below.
+
+ The target hook gives a per-coefficient upper bound for the
+ indeterminate. Since those indeterminates are unsigned and
+ nonnegative, a positive coefficient can only increase the upper
+ bound and a negative coefficient can only decrease the lower
+ bound. The opposite bound is unaffected by that coefficient:
+
+ [A, +C] with C >= 0 => max += C * bound
+ [A, -C] with C >= 0 => min -= C * bound. */
+ wide_int bounds[2]
+ = { wide_int::from (val.coeffs[0], precision, sign),
+ wide_int::from (val.coeffs[0], precision, sign) };
+ bool ovf[2] = { false, false };
+
+ for (unsigned int i = 1; i < NUM_POLY_INT_COEFFS; ++i)
+ {
+ wide_int coeff = wide_int::from (val.coeffs[i], precision, sign);
+ if (wi::eq_p (coeff, 0))
+ continue;
+
+ /* Select the only bound affected by this coefficient. A
+ negative coefficient contributes to the minimum and a positive
+ coefficient contributes to the maximum. */
+ bool coeff_neg = wi::neg_p (coeff, sign);
+ wide_int &bound = bounds[coeff_neg ? 0 : 1];
+ bool &bound_ovf = ovf[coeff_neg ? 0 : 1];
+
+ if (bound_ovf)
+ continue;
+
+ /* A missing hook, or a -1 bound for this coefficient, means the
+ indeterminate has no finite target-specific limit. Treat that
+ like an overflow of the affected bound. */
+ if (!have_poly_bound
+ || indeterminate_bound.coeffs[i] == HOST_WIDE_INT_M1U)
+ bound_ovf = true;
+ else
+ {
+ wide_int indeterminate
+ = wi::uhwi (indeterminate_bound.coeffs[i], precision);
+ wi::overflow_type mul_ovf = wi::OVF_NONE;
+ wide_int term = wi::mul (coeff, indeterminate, sign,
+ &mul_ovf);
+ wi::overflow_type add_ovf = wi::OVF_NONE;
+ bound = wi::add (bound, term, sign, &add_ovf);
+ bound_ovf = (mul_ovf != wi::OVF_NONE
+ || add_ovf != wi::OVF_NONE);
+ }
+
+ if (TYPE_OVERFLOW_WRAPS (type) && bound_ovf)
+ {
+ r.set_varying (type);
+ return true;
+ }
+
+ if (bound_ovf)
+ {
+ if (coeff_neg)
+ bounds[0] = type_min;
+ else
+ bounds[1] = type_max;
+ }
+ }
+
+ if (!wi::le_p (bounds[0], bounds[1], sign))
+ {
+ r.set_varying (type);
+ return true;
+ }
+
+ irange &ir = as_a <irange> (r);
+ ir.set (type, bounds[0], bounds[1]);
+
+ /* Preserve alignment/step information that is not visible in the
+ intervals. For example, a poly-int like [8, 8] can
+ only produce multiples of 8, but the interval range might be
+ [8, 136], which also contains values with low bits set. */
+ ir.update_bitmask (irange_bitmask (wi::zero (precision),
+ get_nonzero_bits (expr)));
return true;
}
break;