[gcc r17-3220] tree-optimization/126028 - vector placement and live lane extracts
Richard Biener via Gcc-cvs <[email protected]>
| Newsgroups | gmane.comp.gcc.cvs |
|---|---|
| Message-ID | <[email protected]> |
https://gcc.gnu.org/g:5e63fb58f84e3ce19cb336710f69094ecbd4b63d commit r17-3220-g5e63fb58f84e3ce19cb336710f69094ecbd4b63d Author: Richard Biener <[email protected]> Date: Wed Jul 15 15:33:31 2026 +0200 tree-optimization/126028 - vector placement and live lane extracts We currently conservatively assume vector stmt placement when determining whether we can place a lane extraction for a live scalar value. The following attempts to improve this by pre-computing an insertion place for all nodes and using that to improve live lane extract feasibility computation. While we'd like to re-use such computation for the actual insertion there are issues preventing that such as scalar stmts eventually getting elided and thus iterators invalidated. So the following re-computes the scheduling again but at least validates the earlier schedule against it. For PR126099 I have to turn this into schedule validation, but not as part of this initial work. The major pain-point of chosing a gimple * as scheduling anchor are that for placement in empty BBs that is NULL (this hits us with region boundaries), a gsi would be able to handle this case. But then, as you can see in the blob that attempts to verify the early schedule constraints hold at transform time there's issues like the scalar stmts vanishing and statements being inserted. As most viable and ugly option I briefly considered inserting GIMPLE_NOPs as anchors. Instead of vect_schedule_slp_node computing the insertion place this could be also handled (or just overridden?) by vectorizable_* which sometimes has its own idea where to place. PR tree-optimization/126028 * tree-vectorizer.h (_slp_tree::si): New member. (vect_schedule_slp): Adjust prototype. * tree-vect-loop.cc (vect_transform_loop): Adjust. * tree-vect-slp.cc (_slp_tree::_slp_tree): Initialize si. (vect_bb_slp_mark_live_stmts): Use vector stmt placement constraint computed by pre-scheduling of SLP nodes. (vect_slp_analyze_bb_1): Pre-schedule SLP nodes before computing live lane extraction points. (vect_slp_region): Adjust. (vect_schedule_slp_node): Add parameter indicating to whether we should perform pre-scehduling or not. Compute the insertion place suitable for dominance checks when pre-scheduling. (vect_schedule_scc): Adjust. (vect_schedule_slp): Likewise. * gcc.dg/vect/bb-slp-46.c: Un-XFAIL. * gcc.dg/vect/bb-slp-pr126053.c: New testcase. * gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c: Likewise. Diff: --- gcc/testsuite/gcc.dg/vect/bb-slp-46.c | 12 +- gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c | 63 ++++++++ .../vect/costmodel/x86_64/costmodel-pr126028.c | 63 ++++++++ gcc/tree-vect-loop.cc | 3 +- gcc/tree-vect-slp.cc | 166 ++++++++++++++++++--- gcc/tree-vectorizer.h | 5 +- 6 files changed, 280 insertions(+), 32 deletions(-) diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-46.c b/gcc/testsuite/gcc.dg/vect/bb-slp-46.c index 4eceea44efcf..ff4a64a011b9 100644 --- a/gcc/testsuite/gcc.dg/vect/bb-slp-46.c +++ b/gcc/testsuite/gcc.dg/vect/bb-slp-46.c @@ -6,7 +6,7 @@ int a[4], b[4]; int foo () { int tem0 = a[0] + b[0]; - int temx = tem0 * 17; /* this fails without a real need */ + int temx = tem0 * 17; int tem1 = a[1] + b[1]; int tem2 = a[2] + b[2]; int tem3 = a[3] + b[3]; @@ -18,11 +18,9 @@ int foo () return temx / temy; } -/* We should extract the live lane from the vectorized add rather than - keeping the original scalar add. - ??? Because of a too conservative check we fail for temx here. */ +/* We should extract the live lanes from the vectorized mul rather than + keeping the original scalar add. */ /* { dg-final { scan-tree-dump "optimized: basic block" "slp2" } } */ -/* { dg-final { scan-tree-dump "extracting lane for live stmt" "slp2" } } */ -/* { dg-final { scan-tree-dump-times "extracting lane for live stmt" 2 "slp2" { xfail *-*-* } } } */ +/* { dg-final { scan-tree-dump-times "extracting lane for live stmt" 2 "slp2" } } */ /* { dg-final { scan-tree-dump-not "tem3_\[0-9\]\+ = " "optimized" } } */ -/* { dg-final { scan-tree-dump-not "tem0_\[0-9\]\+ = " "optimized" { xfail *-*-* } } } */ +/* { dg-final { scan-tree-dump-not "tem0_\[0-9\]\+ = " "optimized" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c b/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c new file mode 100644 index 000000000000..018810d37b8f --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c @@ -0,0 +1,63 @@ +/* { dg-do compile } */ + +typedef double BigReal; + +struct Position { BigReal x, y, z; }; + +struct CompAtom { + struct Position position; + float charge; + short vdwType; + unsigned char partition; + unsigned char nonbondedGroupSize; +}; + +void cull_goodgroups(const struct CompAtom * __restrict p_1, + const int * __restrict glist, + int gu, + BigReal p_i_x, BigReal p_i_y, BigReal p_i_z, + BigReal groupplcutoff2, + int * __restrict goodglist) +{ + int hu = 0; + int g = 0; + int jprev0 = glist[0]; + int jprev1 = glist[1]; + int j0, j1; + + BigReal pj_x_0, pj_x_1, pj_y_0, pj_y_1, pj_z_0, pj_z_1; + BigReal t_0, t_1, r2_0, r2_1; + + pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x; + pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y; + pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z; + g += 2; + + for ( ; g < gu - 2; g += 2 ) { + j0 = jprev0; + j1 = jprev1; + + t_0 = p_i_x - pj_x_0; t_1 = p_i_x - pj_x_1; + r2_0 = t_0 * t_0; r2_1 = t_1 * t_1; + t_0 = p_i_y - pj_y_0; t_1 = p_i_y - pj_y_1; + r2_0 += t_0 * t_0; r2_1 += t_1 * t_1; + t_0 = p_i_z - pj_z_0; t_1 = p_i_z - pj_z_1; + r2_0 += t_0 * t_0; r2_1 += t_1 * t_1; + + jprev0 = glist[g]; + jprev1 = glist[g + 1]; + pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x; + pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y; + pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z; + + bool test0 = (r2_0 < groupplcutoff2); + bool test1 = (r2_1 < groupplcutoff2); + + goodglist[hu] = j0; + goodglist[hu + test0] = j1; + hu += test0 + test1; + } +} + +/* { dg-final { scan-tree-dump "58:12: optimized: basic block part vectorized" "slp1" { target x86_64-*-* } } } */ +/* { dg-final { scan-tree-dump-not "Cannot determine insertion place" "slp1" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c b/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c new file mode 100644 index 000000000000..15db1d390d9b --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c @@ -0,0 +1,63 @@ +/* { dg-do compile } */ +/* { dg-additional-options "-fdump-tree-slp1 -fopt-info-vec" } */ + +typedef double BigReal; + +struct Position { BigReal x, y, z; }; + +struct CompAtom { + struct Position position; + float charge; + short vdwType; + unsigned char partition; + unsigned char nonbondedGroupSize; +}; + +void cull_goodgroups(const struct CompAtom * __restrict p_1, + const int * __restrict glist, + int gu, + BigReal p_i_x, BigReal p_i_y, BigReal p_i_z, + BigReal groupplcutoff2, + int * __restrict goodglist) +{ + int hu = 0; + int g = 0; + int jprev0 = glist[0]; + int jprev1 = glist[1]; + int j0, j1; + + BigReal pj_x_0, pj_x_1, pj_y_0, pj_y_1, pj_z_0, pj_z_1; + BigReal t_0, t_1, r2_0, r2_1; + + pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x; + pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y; + pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z; + g += 2; + + for ( ; g < gu - 2; g += 2 ) { + j0 = jprev0; + j1 = jprev1; + + t_0 = p_i_x - pj_x_0; t_1 = p_i_x - pj_x_1; + r2_0 = t_0 * t_0; r2_1 = t_1 * t_1; + t_0 = p_i_y - pj_y_0; t_1 = p_i_y - pj_y_1; + r2_0 += t_0 * t_0; r2_1 += t_1 * t_1; + t_0 = p_i_z - pj_z_0; t_1 = p_i_z - pj_z_1; + r2_0 += t_0 * t_0; r2_1 += t_1 * t_1; + + jprev0 = glist[g]; + jprev1 = glist[g + 1]; + pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x; + pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y; + pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z; + + bool test0 = (r2_0 < groupplcutoff2); + bool test1 = (r2_1 < groupplcutoff2); + + goodglist[hu] = j0; + goodglist[hu + test0] = j1; + hu += test0 + test1; /* { dg-optimized "basic block part vectorized using 8 byte vectors" } */ + } +} + +/* { dg-final { scan-tree-dump-times " = vect_t\[^ \]* \\\* vect_t" 3 "slp1" } } */ diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 0a09ce8bd85c..a1e60a31ecb7 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -11405,7 +11405,8 @@ vect_transform_loop (loop_vec_info loop_vinfo, gimple *loop_vectorized_call) if (!loop_vinfo->slp_instances.is_empty ()) { DUMP_VECT_SCOPE ("scheduling SLP instances"); - vect_schedule_slp (loop_vinfo, LOOP_VINFO_SLP_INSTANCES (loop_vinfo)); + vect_schedule_slp (loop_vinfo, LOOP_VINFO_SLP_INSTANCES (loop_vinfo), + false); } /* Generate the loop invariant statements. */ diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index b4eb9c204cbb..fa229ce1547e 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -135,6 +135,7 @@ _slp_tree::_slp_tree () this->lanes = 0; SLP_TREE_TYPE (this) = undef_vec_info_type; this->data = NULL; + this->si = NULL; } /* Tear down a SLP node. */ @@ -9073,7 +9074,7 @@ vect_bb_slp_mark_live_stmts (bb_vec_info bb_vinfo, slp_tree node, unsigned i; stmt_vec_info stmt_info; - stmt_vec_info last_stmt = vect_find_last_scalar_stmt_in_slp (node); + gimple *last_stmt = NULL; FOR_EACH_VEC_ELT (SLP_TREE_SCALAR_STMTS (node), i, stmt_info) { if (!stmt_info || svisited.contains (stmt_info)) @@ -9125,7 +9126,11 @@ vect_bb_slp_mark_live_stmts (bb_vec_info bb_vinfo, slp_tree node, || !PURE_SLP_STMT (use_stmt_info))) { live_p = true; - if (!vect_stmt_dominates_stmt_p (last_stmt->stmt, use_stmt)) + if (!last_stmt) + last_stmt + = (node->si ? node->si + : vect_find_last_scalar_stmt_in_slp (node)->stmt); + if (!vect_stmt_dominates_stmt_p (last_stmt, use_stmt)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -10244,6 +10249,9 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo, int n_stmts, bool &fatal, /* Mark all the statements that we vectorize. */ vect_bb_slp_mark_stmts_vectorized (bb_vinfo); + /* Compute vector stmt placement. */ + vect_schedule_slp (bb_vinfo, BB_VINFO_SLP_INSTANCES (bb_vinfo), true); + /* Compute vectorizable live stmts. */ vect_bb_slp_mark_live_stmts (bb_vinfo); @@ -10398,12 +10406,11 @@ vect_slp_region (vec<basic_block> bbs, vec<data_reference_p> datarefs, dump_user_location_t saved_vect_location = vect_location; vect_location = instance->location (); - vect_schedule_slp (bb_vinfo, instance->subgraph_entries); + vect_schedule_slp (bb_vinfo, instance->subgraph_entries, false); vect_location = saved_vect_location; } - /* Generate the invariant statements. */ if (!gimple_seq_empty_p (bb_vinfo->inv_pattern_def_seq)) { @@ -11845,13 +11852,13 @@ vectorizable_slp_permutation (vec_info *vinfo, gimple_stmt_iterator *gsi, return true; } -/* Vectorize SLP NODE. */ +/* Vectorize SLP NODE. Only compute the vector insertion places when + PLACE_ONLY is true. */ static void vect_schedule_slp_node (vec_info *vinfo, - slp_tree node, slp_instance instance) + slp_tree node, slp_instance instance, bool place_only) { - gimple_stmt_iterator si; int i; slp_tree child; @@ -11859,6 +11866,33 @@ vect_schedule_slp_node (vec_info *vinfo, if (SLP_TREE_DEF_TYPE (node) == vect_constant_def || SLP_TREE_DEF_TYPE (node) == vect_external_def) { + if (place_only) + { + if (SLP_TREE_DEF_TYPE (node) == vect_constant_def) + return; + gimple *last_stmt = NULL; + vec<tree> &defs = (!SLP_TREE_SCALAR_OPS (node).is_empty () + ? SLP_TREE_SCALAR_OPS (node) + : SLP_TREE_VEC_DEFS (node)); + for (tree def : defs) + /* If the stmt is not inside the region do not + use it as possible insertion point. */ + if (auto stmt_info = vinfo->lookup_def (def)) + { + gimple *stmt = stmt_info->stmt; + if (!last_stmt) + last_stmt = stmt; + else if (vect_stmt_dominates_stmt_p (last_stmt, stmt)) + last_stmt = stmt; + else if (vect_stmt_dominates_stmt_p (stmt, last_stmt)) + ; + else + gcc_unreachable (); + } + node->si = last_stmt; + return; + } + /* ??? vectorizable_shift can end up using a scalar operand which is currently denoted as !SLP_TREE_VECTYPE. No need to vectorize the node in this case. */ @@ -11878,9 +11912,15 @@ vect_schedule_slp_node (vec_info *vinfo, stmt_vec_info stmt_info = SLP_TREE_REPRESENTATIVE (node); gcc_assert (SLP_TREE_VEC_DEFS (node).is_empty ()); - if (SLP_TREE_VECTYPE (node)) + if (!place_only && SLP_TREE_VECTYPE (node)) SLP_TREE_VEC_DEFS (node).create (vect_get_num_copies (vinfo, node)); + gimple *last_stmt; + gimple_stmt_iterator si; + /* ??? When !place_only we'd like to re-use place_only computed info, + but this is a bit awkward due to using gsi_insert_before and the + requirement to insert after vector defs. So we compute last_stmt + during pre-scheduling and si during scheduling. */ if (!SLP_TREE_PERMUTE_P (node) && STMT_VINFO_DATA_REF (stmt_info)) { /* Vectorized loads go before the first scalar load to make it @@ -11891,7 +11931,8 @@ vect_schedule_slp_node (vec_info *vinfo, last_stmt_info = vect_find_first_scalar_stmt_in_slp (node); else /* DR_IS_WRITE */ last_stmt_info = vect_find_last_scalar_stmt_in_slp (node); - si = gsi_for_stmt (last_stmt_info->stmt); + last_stmt = last_stmt_info->stmt; + si = gsi_for_stmt (last_stmt); } else if (!SLP_TREE_PERMUTE_P (node) && (SLP_TREE_TYPE (node) == cycle_phi_info_type @@ -11899,15 +11940,37 @@ vect_schedule_slp_node (vec_info *vinfo, || SLP_TREE_TYPE (node) == phi_info_type)) { /* For PHI node vectorization we do not use the insertion iterator. */ + last_stmt = SLP_TREE_SCALAR_STMTS (node)[0]->stmt; si = gsi_none (); } else { /* Emit other stmts after the children vectorized defs which is earliest possible. */ - gimple *last_stmt = NULL; + last_stmt = NULL; FOR_EACH_VEC_ELT (SLP_TREE_CHILDREN (node), i, child) - if (SLP_TREE_DEF_TYPE (child) == vect_internal_def) + if (place_only) + { + gimple *vstmt = child->si; + if (!vstmt) + { + /* vect_constant_def and defs at region boundary do not + constrain placement. */ + gcc_assert (SLP_TREE_DEF_TYPE (child) == vect_constant_def + /* ??? Region boundary is not representated + by a NULL stmt. */ + || true); + } + else if (!last_stmt) + last_stmt = vstmt; + else if (vect_stmt_dominates_stmt_p (last_stmt, vstmt)) + last_stmt = vstmt; + else if (vect_stmt_dominates_stmt_p (vstmt, last_stmt)) + ; + else + gcc_unreachable (); + } + else if (SLP_TREE_DEF_TYPE (child) == vect_internal_def) { /* For fold-left reductions we are retaining the scalar reduction PHI but we still have SLP_TREE_NUM_VEC_STMTS @@ -12021,11 +12084,16 @@ vect_schedule_slp_node (vec_info *vinfo, gimple_bb (stmt_info->stmt), gimple_bb (last_stmt))); si = gsi_after_labels (gimple_bb (stmt_info->stmt)); + last_stmt = gsi_stmt (si); } /* When there is no in-region child def to guide placement, insert at region boundary. */ else if (!last_stmt) - si = gsi_after_labels (vinfo->bbs[0]); + { + si = gsi_after_labels (vinfo->bbs[0]); + /* ??? last_stmt can be NULL if the block is empty. */ + last_stmt = gsi_stmt (si); + } else if (is_a <gphi *> (last_stmt)) si = gsi_after_labels (gimple_bb (last_stmt)); else @@ -12037,6 +12105,8 @@ vect_schedule_slp_node (vec_info *vinfo, if (auto loop_vinfo = dyn_cast <loop_vec_info> (vinfo)) { + /* We'll have to fix this up for loop vect. */ + gcc_assert (!place_only); /* Avoid scheduling stmts to random places in the CFG, any stmt dominance check we performed is possibly wrong as UIDs are not initialized for all of the function for loop @@ -12070,6 +12140,46 @@ vect_schedule_slp_node (vec_info *vinfo, } } + if (place_only) + { + if (dump_enabled_p () && last_stmt) + dump_printf_loc (MSG_NOTE, vect_location, + "placing node %p at %G:", (void *)node, last_stmt); + /* Verify we either get a stmt anchor or region start. */ + gcc_assert ((last_stmt && gimple_bb (last_stmt)) + || (!last_stmt && gsi_bb (si))); + node->si = last_stmt; + return; + } + + /* ??? Asserting vect_stmt_dominates_stmt_p (gsi_stmt (si), node->si) + does not work because in some cases we advance si from last_stmt (as + we want to insert after vector stmts) and because vector stmts of + children have been inserted possibly at the same location constraint, + moving si even further. */ + if (flag_checking && node->si && gimple_bb (node->si) && !gsi_end_p (si)) + { + auto gsi2 = si; + while (1) + { + if (vect_stmt_dominates_stmt_p (gsi_stmt (gsi2), node->si)) + break; + /* As we have possibly advanced si it might now point to the + scalar stmt immediately following node->si. That's OK. */ + if (gsi_stmt (gsi2) != gsi_stmt (si) + && gimple_uid (gsi_stmt (gsi2)) != 0) + gcc_unreachable (); + gsi_prev (&gsi2); + if (gsi_end_p (gsi2)) + { + if (is_a <gphi *> (node->si) + && gimple_bb (node->si) == gsi_bb (si)) + break; + gcc_unreachable (); + } + } + } + if (dump_enabled_p ()) { if (stmt_info) @@ -12295,7 +12405,7 @@ struct slp_scc_info static void vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance, hash_map<slp_tree, slp_scc_info> &scc_info, - int &maxdfs, vec<slp_tree> &stack) + int &maxdfs, vec<slp_tree> &stack, bool place_only) { bool existed_p; slp_scc_info *info = &scc_info.get_or_insert (node, &existed_p); @@ -12308,7 +12418,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance, if (SLP_TREE_DEF_TYPE (node) != vect_internal_def) { info->on_stack = false; - vect_schedule_slp_node (vinfo, node, instance); + vect_schedule_slp_node (vinfo, node, instance, place_only); return; } @@ -12325,7 +12435,8 @@ vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance, slp_scc_info *child_info = scc_info.get (child); if (!child_info) { - vect_schedule_scc (vinfo, child, instance, scc_info, maxdfs, stack); + vect_schedule_scc (vinfo, child, instance, scc_info, maxdfs, stack, + place_only); /* Recursion might have re-allocated the node. */ info = scc_info.get (node); child_info = scc_info.get (child); @@ -12344,7 +12455,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance, { stack.pop (); info->on_stack = false; - vect_schedule_slp_node (vinfo, node, instance); + vect_schedule_slp_node (vinfo, node, instance, place_only); if (!SLP_TREE_PERMUTE_P (node) && is_a <gphi *> (SLP_TREE_REPRESENTATIVE (node)->stmt)) phis_to_fixup.quick_push (node); @@ -12396,7 +12507,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance, } if (ready) { - vect_schedule_slp_node (vinfo, entry, instance); + vect_schedule_slp_node (vinfo, entry, instance, place_only); scc_info.get (entry)->on_stack = false; stack[idx] = NULL; todo--; @@ -12411,6 +12522,9 @@ vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance, stack.truncate (last_idx); } + if (place_only) + return; + /* Now fixup the backedge def of the vectorized PHIs in this SCC. */ slp_tree phi_node; FOR_EACH_VEC_ELT (phis_to_fixup, i, phi_node) @@ -12461,10 +12575,12 @@ vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance, } } -/* Generate vector code for SLP_INSTANCES in the loop/basic block. */ +/* Generate vector code for SLP_INSTANCES in the loop/basic block. Perform + vector stmt placement only when PLACE_ONLY is true. */ void -vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances) +vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances, + bool place_only) { slp_instance instance; unsigned int i; @@ -12474,7 +12590,7 @@ vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances) FOR_EACH_VEC_ELT (slp_instances, i, instance) { slp_tree node = SLP_INSTANCE_TREE (instance); - if (dump_enabled_p ()) + if (!place_only && dump_enabled_p ()) { dump_printf_loc (MSG_NOTE, vect_location, "Vectorizing SLP tree:\n"); @@ -12489,16 +12605,20 @@ vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances) have a PHI be the node breaking the cycle. */ auto_vec<slp_tree> stack; if (!scc_info.get (node)) - vect_schedule_scc (vinfo, node, instance, scc_info, maxdfs, stack); + vect_schedule_scc (vinfo, node, instance, scc_info, maxdfs, stack, + place_only); - if (!SLP_INSTANCE_ROOT_STMTS (instance).is_empty ()) + if (!place_only && !SLP_INSTANCE_ROOT_STMTS (instance).is_empty ()) vectorize_slp_instance_root_stmt (vinfo, node, instance); - if (dump_enabled_p ()) + if (!place_only && dump_enabled_p ()) dump_printf_loc (MSG_NOTE, vect_location, "vectorizing stmts using SLP.\n"); } + if (place_only) + return; + FOR_EACH_VEC_ELT (slp_instances, i, instance) { slp_tree root = SLP_INSTANCE_TREE (instance); diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 869b84977807..9cf4710fb26f 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -358,6 +358,9 @@ struct _slp_tree { tree vectype; /* Vectorized defs. */ vec<tree> vec_defs; + /* Insertion place for verification purposes. Only set for + BB vectorization. NULL denotes region entry. */ + gimple *si; /* Reference count in the SLP graph. */ unsigned int refcnt; @@ -2777,7 +2780,7 @@ extern bool vect_transform_slp_perm_load (vec_info *, slp_tree, const vec<tree> extern bool vectorizable_slp_permutation (vec_info *, gimple_stmt_iterator *, slp_tree, stmt_vector_for_cost *); extern bool vect_slp_analyze_operations (vec_info *); -extern void vect_schedule_slp (vec_info *, const vec<slp_instance> &); +extern void vect_schedule_slp (vec_info *, const vec<slp_instance> &, bool); extern opt_result vect_analyze_slp (vec_info *, unsigned, bool); extern bool vect_make_slp_decision (loop_vec_info); extern bool vect_detect_hybrid_slp (loop_vec_info);