[PATCH v2 4/5] vect: Request no-wrap bounds from DR analysis.
Robin Dapp <[email protected]>
| Newsgroups | gmane.comp.gcc.patches |
|---|---|
| Message-ID | <[email protected]> |
From: Robin Dapp <[email protected]> This patch makes use of the preparational patches and activates non-wrapping DR analysis. The resulting nowrap bounds for each DR are collected by taking the minimum and a final new assumption niter < bounds_min is ANDed into the niter assumptions. We cannot yet fully get rid of LOOP_C_FINITE but there is not a lot missing. I hope to finish that after this series. I needed to move DR analysis a bit earlier in order to re-use niter checks and dumps. We now retrieve the DRs directly after vect_analyze_loop_form, then build the assumptions and check them. Afterwards, the DRs are finalized. The vect tests are unchanged from my last patch attempt. Test 1-3 are based on 557.xz's hot loop. PR tree-optimization/121908 gcc/ChangeLog: * tree-vect-data-refs.cc (vect_find_stmt_data_reference): Allow non-wrapping DR analysis. * tree-vect-loop.cc (vect_analyze_loop_form): Defer assumption checking. (vect_get_loop_datarefs): New function. (vect_build_assumptions): New function. (vect_analyze_loop): Use new functions. gcc/testsuite/ChangeLog: * gcc.dg/tree-prof/update-loopch.c: Disable vectorization. * gcc.dg/tree-ssa/scev-12.c: Ditto. * gcc.dg/tree-ssa/scev-8.c: Ditto. * gcc.target/aarch64/sve2/niter-convert-range.c: Expect one more dump instance. * gcc.dg/vect/vect-unsigned-assump-1.c: New test. * gcc.dg/vect/vect-unsigned-assump-2.c: New test. * gcc.dg/vect/vect-unsigned-assump-3.c: New test. * gcc.dg/vect/vect-unsigned-assump-4.c: New test. * gcc.dg/vect/vect-unsigned-assump-5.c: New test. * gcc.dg/vect/vect-unsigned-assump-6.c: New test. * gcc.dg/vect/vect-unsigned-assump-7.c: New test. * gcc.dg/vect/vect-unsigned-assump-8.c: New test. --- .../gcc.dg/tree-prof/update-loopch.c | 2 + gcc/testsuite/gcc.dg/tree-ssa/scev-12.c | 2 +- gcc/testsuite/gcc.dg/tree-ssa/scev-8.c | 2 +- .../gcc.dg/vect/vect-unsigned-assump-1.c | 22 ++ .../gcc.dg/vect/vect-unsigned-assump-2.c | 24 ++ .../gcc.dg/vect/vect-unsigned-assump-3.c | 20 ++ .../gcc.dg/vect/vect-unsigned-assump-4.c | 13 ++ .../gcc.dg/vect/vect-unsigned-assump-5.c | 14 ++ .../gcc.dg/vect/vect-unsigned-assump-6.c | 13 ++ .../gcc.dg/vect/vect-unsigned-assump-7.c | 17 ++ .../gcc.dg/vect/vect-unsigned-assump-8.c | 13 ++ .../aarch64/sve2/niter-convert-range.c | 2 +- gcc/tree-vect-data-refs.cc | 5 +- gcc/tree-vect-loop.cc | 218 ++++++++++++------ gcc/tree-vectorizer.h | 4 +- 15 files changed, 296 insertions(+), 75 deletions(-) create mode 100644 gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-1.c create mode 100644 gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-2.c create mode 100644 gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-3.c create mode 100644 gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-4.c create mode 100644 gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-5.c create mode 100644 gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-6.c create mode 100644 gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-7.c create mode 100644 gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-8.c diff --git a/gcc/testsuite/gcc.dg/tree-prof/update-loopch.c b/gcc/testsuite/gcc.dg/tree-prof/update-loopch.c index a30b895bb67..efb92b2c4fc 100644 --- a/gcc/testsuite/gcc.dg/tree-prof/update-loopch.c +++ b/gcc/testsuite/gcc.dg/tree-prof/update-loopch.c @@ -5,6 +5,8 @@ int main () { int i; +/* If we vectorize this, the block count changes which defies the test. */ +#pragma GCC novector for (i = 0; i < max; i++) { a[i % 8]++; diff --git a/gcc/testsuite/gcc.dg/tree-ssa/scev-12.c b/gcc/testsuite/gcc.dg/tree-ssa/scev-12.c index e2ded46047e..b0feee911c1 100644 --- a/gcc/testsuite/gcc.dg/tree-ssa/scev-12.c +++ b/gcc/testsuite/gcc.dg/tree-ssa/scev-12.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -fdump-tree-ivopts-details" } */ +/* { dg-options "-O2 -fno-tree-vectorize -fdump-tree-ivopts-details" } */ int a[128]; extern int b[]; diff --git a/gcc/testsuite/gcc.dg/tree-ssa/scev-8.c b/gcc/testsuite/gcc.dg/tree-ssa/scev-8.c index a5b2ff71958..abdbf0250dd 100644 --- a/gcc/testsuite/gcc.dg/tree-ssa/scev-8.c +++ b/gcc/testsuite/gcc.dg/tree-ssa/scev-8.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -fdump-tree-ivopts-details" } */ +/* { dg-options "-O2 -fno-tree-vectorize -fdump-tree-ivopts-details" } */ int *a; diff --git a/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-1.c b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-1.c new file mode 100644 index 00000000000..2c64eb890fd --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-1.c @@ -0,0 +1,22 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target vect_int } */ +/* { dg-require-effective-target vect_early_break } */ + +#define uint8_t unsigned char +#define uint32_t unsigned int + +int foo (const uint8_t *const cur, uint32_t len, uint32_t len_limit, + uint32_t pos, uint32_t cur_match) +{ + const uint32_t delta = pos - cur_match; + const uint8_t *pb = cur - delta; + + while (++len != len_limit) + if (pb[len] != cur[len]) + break; + + return len; +} + +/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */ +/* { dg-final { scan-tree-dump "adding no-wrap assumption" "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-2.c b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-2.c new file mode 100644 index 00000000000..a071ec399f0 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-2.c @@ -0,0 +1,24 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target vect_int } */ +/* { dg-require-effective-target vect_early_break } */ +/* { dg-require-effective-target vect_load_lanes } */ + +#define uint8_t unsigned char +#define uint32_t unsigned int + +int foo (const uint8_t *const cur, uint32_t len, uint32_t len_limit, + uint32_t pos, uint32_t cur_match) +{ + const uint32_t delta = pos - cur_match; + const uint8_t *pb = cur - delta; + + /* We vectorize this with struct loads right now. */ + while ((len += 2) != len_limit) + if (pb[len] != cur[len]) + break; + + return len; +} + +/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */ +/* { dg-final { scan-tree-dump "adding no-wrap assumption" "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-3.c b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-3.c new file mode 100644 index 00000000000..c9f6c2b0ca7 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-3.c @@ -0,0 +1,20 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target vect_int } */ +/* { dg-require-effective-target vect_early_break } */ + +#define uint8_t unsigned char +#define uint32_t unsigned int + +int foo (const uint8_t *const cur, uint32_t n) +{ + uint32_t i = 15; + + while (i++ != n) + if (cur[i - 15] != cur[i]) + break; + + return i; +} + +/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */ +/* { dg-final { scan-tree-dump "adding no-wrap assumption" "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-4.c b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-4.c new file mode 100644 index 00000000000..8ba7ce435f9 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-4.c @@ -0,0 +1,13 @@ +/* { dg-do compile } */ +/* { dg-additional-options "-fno-builtin" } */ +/* { dg-require-effective-target vect_int } */ + +/* We should vectorize this without additional assumptions. + The data-ref uses a pointer-based access which never wraps. */ + +void f4 (int *p, unsigned start, unsigned end) { + for (unsigned i = start; i != end; i++) + *p++ = 0; +} + +/* { dg-final { scan-tree-dump-not "adding no-wrap assumption" "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-5.c b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-5.c new file mode 100644 index 00000000000..2922fd1c114 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-5.c @@ -0,0 +1,14 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target vect_int } */ +/* { dg-require-effective-target vect_gather_load_ifn } */ + +/* This should be vectorized without gather/scatter. */ + +void f5 (int *__restrict a, int *__restrict b, unsigned start, unsigned end) { + for (unsigned i = start; i != end; i++) + a[i] = b[i]; +} + +/* { dg-final { scan-tree-dump-not "gather" "vect" } } */ +/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */ +/* { dg-final { scan-tree-dump "adding no-wrap assumption" "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-6.c b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-6.c new file mode 100644 index 00000000000..e869e8095ea --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-6.c @@ -0,0 +1,13 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target vect_int } */ +/* { dg-require-effective-target vect_gather_load_ifn } */ + +/* This should be vectorized without gather/scatter. */ + +void f6 (int *__restrict a, int *__restrict b, unsigned start, unsigned end) { + for (unsigned i = start; i <= end; i++) + a[i] = b[i]; +} + +/* { dg-final { scan-tree-dump-not "gather" "vect" } } */ +/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-7.c b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-7.c new file mode 100644 index 00000000000..6d7df016a13 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-7.c @@ -0,0 +1,17 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target vect_int } */ + +/* This should be vectorized without gather/scatter. */ + +void f7 (int *__restrict dst, unsigned n) { + unsigned i = 1; + while (i != n) + { + dst[i] = i; + i++; + } +} + +/* { dg-final { scan-tree-dump-not "gather" "vect" } } */ +/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */ +/* { dg-final { scan-tree-dump "adding no-wrap assumption" "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-8.c b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-8.c new file mode 100644 index 00000000000..37d76304274 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-unsigned-assump-8.c @@ -0,0 +1,13 @@ +/* { dg-do compile } */ +/* { dg-additional-options "-fno-builtin" } */ +/* { dg-require-effective-target vect_int } */ + +/* We should vectorize this without additional assumptions. + The data-ref uses a pointer-based access which never wraps. */ + +void f8 (int *__restrict a, unsigned start, unsigned n) { + for (unsigned i = start; i != n; i++) + a[i - start] = 0; +} + +/* { dg-final { scan-tree-dump-not "adding no-wrap assumption" "vect" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve2/niter-convert-range.c b/gcc/testsuite/gcc.target/aarch64/sve2/niter-convert-range.c index c96b2f3999c..58b946782cb 100644 --- a/gcc/testsuite/gcc.target/aarch64/sve2/niter-convert-range.c +++ b/gcc/testsuite/gcc.target/aarch64/sve2/niter-convert-range.c @@ -64,5 +64,5 @@ niter_convert_range_wrap (unsigned char *buf, uint8_t len) /* { dg-final { scan-tree-dump-times {bounds on difference of bases: 0 [.][.][.] 126} 3 "vect" } } */ /* { dg-final { scan-tree-dump-times {bounds on difference of bases: 0 [.][.][.] 254} 9 "vect" } } */ -/* { dg-final { scan-tree-dump-times {bounds on difference of bases: -1 [.][.][.] 4294967294} 3 "vect" } } */ +/* { dg-final { scan-tree-dump-times {bounds on difference of bases: -1 [.][.][.] 4294967294} 4 "vect" } } */ /* { dg-final { scan-tree-dump-times "loop vectorized using variable length vectors" 5 "vect" } } */ diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc index a50e23d7672..ed69539b1b6 100644 --- a/gcc/tree-vect-data-refs.cc +++ b/gcc/tree-vect-data-refs.cc @@ -5193,7 +5193,7 @@ vect_find_stmt_data_reference (loop_p loop, gimple *stmt, stmt); auto_vec<data_reference_p, 2> refs; - opt_result res = find_data_references_in_stmt (loop, stmt, &refs); + opt_result res = find_data_references_in_stmt (loop, stmt, &refs, true); if (!res) return res; @@ -5250,7 +5250,8 @@ vect_find_stmt_data_reference (loop_p loop, gimple *stmt, { struct data_reference *newdr = create_data_ref (NULL, loop_containing_stmt (stmt), DR_REF (dr), stmt, - DR_IS_READ (dr), DR_IS_CONDITIONAL_IN_STMT (dr)); + DR_IS_READ (dr), DR_IS_CONDITIONAL_IN_STMT (dr), + true); if (DR_BASE_ADDRESS (newdr) && DR_OFFSET (newdr) && DR_INIT (newdr) diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 0a09ce8bd85..54f8912b894 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -717,11 +717,28 @@ bb_in_loop_p (const_basic_block bb, const void *data) return false; } +/* Get and return the basic blocks of LOOP in DFS order. Store the number of + BBs in NBBS. */ + +static basic_block * +vect_get_loop_dfs_bbs (class loop *loop, unsigned int *nbbs) +{ + /* CHECKME: We want to visit all BBs before their successors (except for + latch blocks, for which this assertion wouldn't hold). In the simple + case of the loop forms we allow, a dfs order of the BBs would the same + as reversed postorder traversal, so we are safe. */ + basic_block *bbs = XCNEWVEC (basic_block, loop->num_nodes); + *nbbs = dfs_enumerate_from (loop->header, 0, bb_in_loop_p, bbs, + loop->num_nodes, loop); + gcc_assert (*nbbs == loop->num_nodes); + return bbs; +} /* Create and initialize a new loop_vec_info struct for LOOP_IN, as well as stmt_vec_info structs for all the stmts in LOOP_IN. */ -_loop_vec_info::_loop_vec_info (class loop *loop_in, vec_info_shared *shared) +_loop_vec_info::_loop_vec_info (class loop *loop_in, vec_info_shared *shared, + basic_block *bbs_in) : vec_info (vec_info::loop, shared), loop (loop_in), num_itersm1 (NULL_TREE), @@ -779,15 +796,9 @@ _loop_vec_info::_loop_vec_info (class loop *loop_in, vec_info_shared *shared) early_break_needs_epilogue (false), early_break_niters_var (NULL) { - /* CHECKME: We want to visit all BBs before their successors (except for - latch blocks, for which this assertion wouldn't hold). In the simple - case of the loop forms we allow, a dfs order of the BBs would the same - as reversed postorder traversal, so we are safe. */ - - bbs = XCNEWVEC (basic_block, loop->num_nodes); - nbbs = dfs_enumerate_from (loop->header, 0, bb_in_loop_p, bbs, - loop->num_nodes, loop); - gcc_assert (nbbs == loop->num_nodes); + nbbs = loop->num_nodes; + bbs = XNEWVEC (basic_block, nbbs); + memcpy (bbs, bbs_in, nbbs * sizeof (basic_block)); for (unsigned int i = 0; i < nbbs; i++) { @@ -1466,22 +1477,21 @@ vect_analyze_loop_form (class loop *loop, gimple *loop_vectorized_call, exit_e->src->index, exit_e->dest->index, exit_e->aux); /* Check if we have any control flow that doesn't leave the loop. */ - basic_block *bbs = get_loop_body (loop); - for (unsigned i = 0; i < loop->num_nodes; i++) - if (EDGE_COUNT (bbs[i]->succs) != 1 - && (EDGE_COUNT (bbs[i]->succs) != 2 - || !loop_exits_from_bb_p (bbs[i]->loop_father, bbs[i]))) - { - free (bbs); - return opt_result::failure_at (vect_location, - "not vectorized:" - " unsupported control flow in loop.\n"); - } + unsigned int nbbs; + info->bbs = vect_get_loop_dfs_bbs (loop, &nbbs); + for (unsigned i = 0; i < nbbs; i++) + if (EDGE_COUNT (info->bbs[i]->succs) != 1 + && (EDGE_COUNT (info->bbs[i]->succs) != 2 + || !loop_exits_from_bb_p (info->bbs[i]->loop_father, + info->bbs[i]))) + return opt_result::failure_at (vect_location, + "not vectorized:" + " unsupported control flow in loop.\n"); /* Check if we have any control flow that doesn't leave the loop. */ bool has_phi = false; - for (unsigned i = 0; i < loop->num_nodes; i++) - if (!gimple_seq_empty_p (phi_nodes (bbs[i]))) + for (unsigned i = 0; i < nbbs; i++) + if (!gimple_seq_empty_p (phi_nodes (info->bbs[i]))) { has_phi = true; break; @@ -1491,8 +1501,6 @@ vect_analyze_loop_form (class loop *loop, gimple *loop_vectorized_call, "not vectorized:" " no scalar evolution detected in loop.\n"); - free (bbs); - /* Different restrictions apply when we are considering an inner-most loop, vs. an outer (nested) loop. (FORNOW. May want to relax some of these restrictions in the future). */ @@ -1651,17 +1659,6 @@ vect_analyze_loop_form (class loop *loop, gimple *loop_vectorized_call, } } - if (!integer_onep (info->assumptions)) - { - if (dump_enabled_p ()) - { - dump_printf_loc (MSG_NOTE, vect_location, - "Loop to be versioned with niter assumption "); - dump_generic_expr (MSG_NOTE, TDF_SLIM, info->assumptions); - dump_printf (MSG_NOTE, "\n"); - } - } - return opt_result::success (); } @@ -1673,7 +1670,7 @@ vect_create_loop_vinfo (class loop *loop, vec_info_shared *shared, const vect_loop_form_info *info, loop_vec_info orig_loop_info) { - loop_vec_info loop_vinfo = new _loop_vec_info (loop, shared); + loop_vec_info loop_vinfo = new _loop_vec_info (loop, shared, info->bbs); LOOP_VINFO_NITERSM1 (loop_vinfo) = info->number_of_iterationsm1; LOOP_VINFO_NITERS (loop_vinfo) = info->number_of_iterations; LOOP_VINFO_NITERS_UNCHANGED (loop_vinfo) = info->number_of_iterations; @@ -2053,6 +2050,104 @@ vect_get_datarefs_in_loop (loop_p loop, basic_block *bbs, return opt_result::success (); } +/* Gather no-wrap bounds from SHARED's datarefs and accumulate them in + one assumption. Validate this assumption, as well as INFO's niter + assumption and return an error on failure. + If the assumptions are valid, reset loop properties as well as + scev's hashtable and return success. */ + +static opt_result +vect_build_assumptions (class loop *loop, vec_info_shared *shared, + vect_loop_form_info *info) +{ + tree niter = info->number_of_iterations; + bool niter_known = niter && !chrec_contains_undetermined (niter); + + /* Accumulate the surviving nowrap predicates separately so we can report + just them, rather than the union with the pre-existing niter + assumptions. */ + tree nowrap_assum = NULL_TREE; + tree bound_min = NULL_TREE; + + unsigned int i; + struct data_reference *dr; + + /* Build the minimum over all nowrap bounds. */ + FOR_EACH_VEC_ELT (shared->datarefs, i, dr) + { + tree bound = DR_NOWRAP_BOUND (dr); + if (!bound) + continue; + + /* Use scev's function to MIN two bounds. */ + scev_combine_nowrap_bounds (&bound_min, bound); + } + + /* Build "NITER < BOUND_MIN". */ + if (niter_known && bound_min) + { + unsigned prec = MAX (TYPE_PRECISION (TREE_TYPE (niter)), + TYPE_PRECISION (TREE_TYPE (bound_min))); + tree t = build_nonstandard_integer_type (prec, 1); + nowrap_assum = fold_build2 (LE_EXPR, boolean_type_node, + fold_convert (t, niter), + fold_convert (t, bound_min)); + + /* Try to simplify the assumption with ranger. */ + value_range vr (TREE_TYPE (nowrap_assum)); + tree better_assum; + if (get_range_query (cfun)->range_on_edge + (vr, loop_preheader_edge (loop), nowrap_assum) + && vr.singleton_p (&better_assum)) + nowrap_assum = better_assum; + } + + /* Combine nowrap assumptions and niter assumption. */ + if (nowrap_assum) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "adding no-wrap assumption: %T\n", nowrap_assum); + + if (info->assumptions && !integer_onep (info->assumptions)) + nowrap_assum = fold_build2 (TRUTH_AND_EXPR, boolean_type_node, + info->assumptions, nowrap_assum); + info->assumptions = nowrap_assum; + } + + if (integer_zerop (info->assumptions)) + return opt_result::failure_at + (vect_location, + "not vectorized: number of iterations cannot be computed.\n"); + + if (!integer_onep (info->assumptions)) + { + if (dump_enabled_p ()) + { + dump_printf_loc (MSG_NOTE, vect_location, + "Loop to be versioned with assumption "); + dump_generic_expr (MSG_NOTE, TDF_SLIM, info->assumptions); + dump_printf (MSG_NOTE, "\n"); + } + + /* We consider to vectorize this loop by versioning it under + some assumptions. In order to do this, we need to clear + existing information computed by scev and niter analyzer. */ + scev_reset_htab (); + free_numbers_of_iterations_estimates (loop); + /* Also set flag for this loop so that following scev and niter + analysis are done under the assumptions. */ + loop_constraint_set (loop, LOOP_C_FINITE); + } + else + /* Clear the existing niter information to make sure the nonwrapping flag + will be calculated and set propriately. */ + free_numbers_of_iterations_estimates (loop); + + return opt_result::success (); +} + + /* Determine if operating on full vectors for LOOP_VINFO might leave some scalar iterations still to do. If so, decide how we should handle those scalar iterations. The possibilities are: @@ -2185,25 +2280,8 @@ vect_analyze_loop_2 (loop_vec_info loop_vinfo, int masked_p, bool &fatal, loop_p loop = LOOP_VINFO_LOOP (loop_vinfo); - /* Gather the data references. */ - if (!LOOP_VINFO_DATAREFS (loop_vinfo).exists ()) - { - opt_result res - = vect_get_datarefs_in_loop (loop, LOOP_VINFO_BBS (loop_vinfo), - &LOOP_VINFO_DATAREFS (loop_vinfo)); - if (!res) - { - if (dump_enabled_p ()) - dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, - "not vectorized: loop contains function " - "calls or data references that cannot " - "be analyzed\n"); - return res; - } - loop_vinfo->shared->save_datarefs (); - } - else - loop_vinfo->shared->check_datarefs (); + /* Check the data references. */ + loop_vinfo->shared->check_datarefs (); /* Analyze the data references and also adjust the minimal vectorization factor according to the loads and stores. */ @@ -2952,21 +3030,23 @@ vect_analyze_loop (class loop *loop, gimple *loop_vectorized_call, "bad loop form.\n"); return opt_loop_vec_info::propagate_failure (res); } - if (!integer_onep (loop_form_info.assumptions)) + + res = vect_get_datarefs_in_loop (loop, loop_form_info.bbs, + &shared->datarefs); + if (!res) { - /* We consider to vectorize this loop by versioning it under - some assumptions. In order to do this, we need to clear - existing information computed by scev and niter analyzer. */ - scev_reset_htab (); - free_numbers_of_iterations_estimates (loop); - /* Also set flag for this loop so that following scev and niter - analysis are done under the assumptions. */ - loop_constraint_set (loop, LOOP_C_FINITE); + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "not vectorized: loop contains function " + "calls or data references that cannot " + "be analyzed\n"); + return opt_loop_vec_info::propagate_failure (res); } - else - /* Clear the existing niter information to make sure the nonwrapping flag - will be calculated and set propriately. */ - free_numbers_of_iterations_estimates (loop); + shared->save_datarefs (); + + res = vect_build_assumptions (loop, shared, &loop_form_info); + if (!res) + return opt_loop_vec_info::propagate_failure (res); auto_vector_modes vector_modes; /* Autodetect first vector size we try. */ diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 65de4c0fca4..527aef2f3d7 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -945,7 +945,7 @@ struct vect_reusable_accumulator { /*-----------------------------------------------------------------*/ typedef class _loop_vec_info : public vec_info { public: - _loop_vec_info (class loop *, vec_info_shared *); + _loop_vec_info (class loop *, vec_info_shared *, basic_block *); ~_loop_vec_info (); /* The loop to which this info struct refers to. */ @@ -2727,6 +2727,8 @@ struct vect_loop_form_info auto_vec<gcond *> conds; gcond *inner_loop_cond; edge loop_exit; + basic_block *bbs = nullptr; + ~vect_loop_form_info () { free (bbs); } }; extern opt_result vect_analyze_loop_form (class loop *, gimple *, vect_loop_form_info *); -- 2.54.0