RE: [PATCH] tree-optimization/126028 - vector placement and live lane extracts
Richard Biener <[email protected]> Thu, 6 Aug 2026 14:06:12 +0200 (CEST)
| Newsgroups | gmane.comp.gcc.patches |
|---|---|
| Message-ID | <[email protected]> |
On Thu, 6 Aug 2026, Tamar Christina wrote: > > -----Original Message----- > > From: Richard Biener <[email protected]> > > Sent: 03 August 2026 13:47 > > To: [email protected] > > Cc: Tamar Christina <[email protected]>; > > [email protected] > > Subject: [PATCH] tree-optimization/126028 - vector placement and live lane > > extracts > > > > We currently conservatively assume vector stmt placement when > > determining whether we can place a lane extraction for a live > > scalar value. The following attempts to improve this by > > pre-computing an insertion place for all nodes and using that > > to improve live lane extract feasibility computation. > > > > While we'd like to re-use such computation for the actual insertion > > there are issues preventing that such as scalar stmts eventually > > getting elided and thus iterators invalidated. So the following > > re-computes the scheduling again but at least validates the earlier > > schedule against it. > > > > For PR126099 I have to turn this into schedule validation, but > > not as part of this initial work. > > > > What I've been trying to figure out here is why the insertion point > isn't just at the end of the SLP region of the tree with the lane extract. > Since after scheduling, we should know the dependency order, so for a live > lane it feels like placing the extract immediately after the vector stmt for that node > would be the natural/latest-needed point. At least for bb-slp-46.c I don't immediately > see why a more global/conservative placement is needed. This is because for bb-slp-46.c we can insert the vector operation for the plus at the earliest scalar operation. That's not always possible, in most cases because of externs (aka from-scalar vector CTOR) which have to be placed after the last scalar def. That can then trivially interleave: x = ... tem0 = a[0] + x; ... = tem0; y = ... tem1 = a[1] + y; so a vector plus would be inserted after y = ..., which is after the use of tem0. > So I'm pretty sure I'm missing something. Other than that the patch looks > sensible to me. Thanks. > > The major pain-point of chosing a gimple * as scheduling anchor > > are that for placement in empty BBs that is NULL (this hits us > > with region boundaries), a gsi would be able to handle this > > You could store a tuple of bb and gimple* maybe? The bb would > Be the fallback if insertion point is empty at the time and so you > can't get the gimple *. So I've (for now) settled with documenting that a NULL gimple * means insertion at region start. I have an updated series I'll post shortly Richard. > Thanks, > Tamar > > > case. But then, as you can see in the blob that attempts to > > verify the early schedule constraints hold at transform time > > there's issues like the scalar stmts vanishing and statements > > being inserted. As most viable and ugly option I briefly > > considered inserting GIMPLE_NOPs as anchors. Instead of > > vect_schedule_slp_node computing the insertion place this > > could be also handled (or just overridden?) by vectorizable_* > > which sometimes has its own idea where to place. > > > > Bootstrapped and tested on x86_64-unknown-linux-gnu. > > > > Any comments? > > > > Thanks, > > Richard. > > > > PR tree-optimization/126028 > > * tree-vectorizer.h (_slp_tree::si): New member. > > (vect_schedule_slp): Adjust prototype. > > * tree-vect-loop.cc (vect_transform_loop): Adjust. > > * tree-vect-slp.cc (_slp_tree::_slp_tree): Initialize si. > > (vect_bb_slp_mark_live_stmts): Use vector stmt placement > > constraint computed by pre-scheduling of SLP nodes. > > (vect_slp_analyze_bb_1): Pre-schedule SLP nodes before > > computing live lane extraction points. > > (vect_slp_region): Adjust. > > (vect_schedule_slp_node): Add parameter indicating to whether > > we should perform pre-scehduling or not. Compute the > > insertion place suitable for dominance checks when > > pre-scheduling. > > (vect_schedule_scc): Adjust. > > (vect_schedule_slp): Likewise. > > > > * gcc.dg/vect/bb-slp-46.c: Un-XFAIL. > > * gcc.dg/vect/bb-slp-pr126053.c: New testcase. > > * gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c: Likewise. > > --- > > gcc/testsuite/gcc.dg/vect/bb-slp-46.c | 12 +- > > gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c | 63 +++++++ > > .../costmodel/x86_64/costmodel-pr126028.c | 63 +++++++ > > gcc/tree-vect-loop.cc | 3 +- > > gcc/tree-vect-slp.cc | 162 +++++++++++++++--- > > gcc/tree-vectorizer.h | 4 +- > > 6 files changed, 275 insertions(+), 32 deletions(-) > > create mode 100644 gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c > > create mode 100644 > > gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c > > > > diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-46.c > > b/gcc/testsuite/gcc.dg/vect/bb-slp-46.c > > index 4eceea44efc..ff4a64a011b 100644 > > --- a/gcc/testsuite/gcc.dg/vect/bb-slp-46.c > > +++ b/gcc/testsuite/gcc.dg/vect/bb-slp-46.c > > @@ -6,7 +6,7 @@ int a[4], b[4]; > > int foo () > > { > > int tem0 = a[0] + b[0]; > > - int temx = tem0 * 17; /* this fails without a real need */ > > + int temx = tem0 * 17; > > int tem1 = a[1] + b[1]; > > int tem2 = a[2] + b[2]; > > int tem3 = a[3] + b[3]; > > @@ -18,11 +18,9 @@ int foo () > > return temx / temy; > > } > > > > -/* We should extract the live lane from the vectorized add rather than > > - keeping the original scalar add. > > - ??? Because of a too conservative check we fail for temx here. */ > > +/* We should extract the live lanes from the vectorized mul rather than > > + keeping the original scalar add. */ > > /* { dg-final { scan-tree-dump "optimized: basic block" "slp2" } } */ > > -/* { dg-final { scan-tree-dump "extracting lane for live stmt" "slp2" } } */ > > -/* { dg-final { scan-tree-dump-times "extracting lane for live stmt" 2 "slp2" { > > xfail *-*-* } } } */ > > +/* { dg-final { scan-tree-dump-times "extracting lane for live stmt" 2 "slp2" } } > > */ > > /* { dg-final { scan-tree-dump-not "tem3_\[0-9\]\+ = " "optimized" } } */ > > -/* { dg-final { scan-tree-dump-not "tem0_\[0-9\]\+ = " "optimized" { xfail *-*- > > * } } } */ > > +/* { dg-final { scan-tree-dump-not "tem0_\[0-9\]\+ = " "optimized" } } */ > > diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c > > b/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c > > new file mode 100644 > > index 00000000000..018810d37b8 > > --- /dev/null > > +++ b/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c > > @@ -0,0 +1,63 @@ > > +/* { dg-do compile } */ > > + > > +typedef double BigReal; > > + > > +struct Position { BigReal x, y, z; }; > > + > > +struct CompAtom { > > + struct Position position; > > + float charge; > > + short vdwType; > > + unsigned char partition; > > + unsigned char nonbondedGroupSize; > > +}; > > + > > +void cull_goodgroups(const struct CompAtom * __restrict p_1, > > + const int * __restrict glist, > > + int gu, > > + BigReal p_i_x, BigReal p_i_y, BigReal p_i_z, > > + BigReal groupplcutoff2, > > + int * __restrict goodglist) > > +{ > > + int hu = 0; > > + int g = 0; > > + int jprev0 = glist[0]; > > + int jprev1 = glist[1]; > > + int j0, j1; > > + > > + BigReal pj_x_0, pj_x_1, pj_y_0, pj_y_1, pj_z_0, pj_z_1; > > + BigReal t_0, t_1, r2_0, r2_1; > > + > > + pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x; > > + pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y; > > + pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z; > > + g += 2; > > + > > + for ( ; g < gu - 2; g += 2 ) { > > + j0 = jprev0; > > + j1 = jprev1; > > + > > + t_0 = p_i_x - pj_x_0; t_1 = p_i_x - pj_x_1; > > + r2_0 = t_0 * t_0; r2_1 = t_1 * t_1; > > + t_0 = p_i_y - pj_y_0; t_1 = p_i_y - pj_y_1; > > + r2_0 += t_0 * t_0; r2_1 += t_1 * t_1; > > + t_0 = p_i_z - pj_z_0; t_1 = p_i_z - pj_z_1; > > + r2_0 += t_0 * t_0; r2_1 += t_1 * t_1; > > + > > + jprev0 = glist[g]; > > + jprev1 = glist[g + 1]; > > + pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x; > > + pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y; > > + pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z; > > + > > + bool test0 = (r2_0 < groupplcutoff2); > > + bool test1 = (r2_1 < groupplcutoff2); > > + > > + goodglist[hu] = j0; > > + goodglist[hu + test0] = j1; > > + hu += test0 + test1; > > + } > > +} > > + > > +/* { dg-final { scan-tree-dump "58:12: optimized: basic block part vectorized" > > "slp1" { target x86_64-*-* } } } */ > > +/* { dg-final { scan-tree-dump-not "Cannot determine insertion place" "slp1" > > } } */ > > diff --git a/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel- > > pr126028.c b/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel- > > pr126028.c > > new file mode 100644 > > index 00000000000..15db1d390d9 > > --- /dev/null > > +++ b/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c > > @@ -0,0 +1,63 @@ > > +/* { dg-do compile } */ > > +/* { dg-additional-options "-fdump-tree-slp1 -fopt-info-vec" } */ > > + > > +typedef double BigReal; > > + > > +struct Position { BigReal x, y, z; }; > > + > > +struct CompAtom { > > + struct Position position; > > + float charge; > > + short vdwType; > > + unsigned char partition; > > + unsigned char nonbondedGroupSize; > > +}; > > + > > +void cull_goodgroups(const struct CompAtom * __restrict p_1, > > + const int * __restrict glist, > > + int gu, > > + BigReal p_i_x, BigReal p_i_y, BigReal p_i_z, > > + BigReal groupplcutoff2, > > + int * __restrict goodglist) > > +{ > > + int hu = 0; > > + int g = 0; > > + int jprev0 = glist[0]; > > + int jprev1 = glist[1]; > > + int j0, j1; > > + > > + BigReal pj_x_0, pj_x_1, pj_y_0, pj_y_1, pj_z_0, pj_z_1; > > + BigReal t_0, t_1, r2_0, r2_1; > > + > > + pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x; > > + pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y; > > + pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z; > > + g += 2; > > + > > + for ( ; g < gu - 2; g += 2 ) { > > + j0 = jprev0; > > + j1 = jprev1; > > + > > + t_0 = p_i_x - pj_x_0; t_1 = p_i_x - pj_x_1; > > + r2_0 = t_0 * t_0; r2_1 = t_1 * t_1; > > + t_0 = p_i_y - pj_y_0; t_1 = p_i_y - pj_y_1; > > + r2_0 += t_0 * t_0; r2_1 += t_1 * t_1; > > + t_0 = p_i_z - pj_z_0; t_1 = p_i_z - pj_z_1; > > + r2_0 += t_0 * t_0; r2_1 += t_1 * t_1; > > + > > + jprev0 = glist[g]; > > + jprev1 = glist[g + 1]; > > + pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x; > > + pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y; > > + pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z; > > + > > + bool test0 = (r2_0 < groupplcutoff2); > > + bool test1 = (r2_1 < groupplcutoff2); > > + > > + goodglist[hu] = j0; > > + goodglist[hu + test0] = j1; > > + hu += test0 + test1; /* { dg-optimized "basic block part vectorized using 8 > > byte vectors" } */ > > + } > > +} > > + > > +/* { dg-final { scan-tree-dump-times " = vect_t\[^ \]* \\\* vect_t" 3 "slp1" } } > > */ > > diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc > > index 31de29c036b..a336df81bc7 100644 > > --- a/gcc/tree-vect-loop.cc > > +++ b/gcc/tree-vect-loop.cc > > @@ -11406,7 +11406,8 @@ vect_transform_loop (loop_vec_info loop_vinfo, > > gimple *loop_vectorized_call) > > if (!loop_vinfo->slp_instances.is_empty ()) > > { > > DUMP_VECT_SCOPE ("scheduling SLP instances"); > > - vect_schedule_slp (loop_vinfo, LOOP_VINFO_SLP_INSTANCES > > (loop_vinfo)); > > + vect_schedule_slp (loop_vinfo, LOOP_VINFO_SLP_INSTANCES > > (loop_vinfo), > > + false); > > } > > > > /* Generate the loop invariant statements. */ > > diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc > > index 8ed7a0da377..dda9e046ade 100644 > > --- a/gcc/tree-vect-slp.cc > > +++ b/gcc/tree-vect-slp.cc > > @@ -135,6 +135,7 @@ _slp_tree::_slp_tree () > > this->lanes = 0; > > SLP_TREE_TYPE (this) = undef_vec_info_type; > > this->data = NULL; > > + this->si = NULL; > > } > > > > /* Tear down a SLP node. */ > > @@ -9277,7 +9278,7 @@ vect_bb_slp_mark_live_stmts (bb_vec_info > > bb_vinfo, slp_tree node, > > > > unsigned i; > > stmt_vec_info stmt_info; > > - stmt_vec_info last_stmt = vect_find_last_scalar_stmt_in_slp (node); > > + gimple *last_stmt = NULL; > > FOR_EACH_VEC_ELT (SLP_TREE_SCALAR_STMTS (node), i, stmt_info) > > { > > if (!stmt_info || svisited.contains (stmt_info)) > > @@ -9329,7 +9330,11 @@ vect_bb_slp_mark_live_stmts (bb_vec_info > > bb_vinfo, slp_tree node, > > || !PURE_SLP_STMT (use_stmt_info))) > > { > > live_p = true; > > - if (!vect_stmt_dominates_stmt_p (last_stmt->stmt, > > use_stmt)) > > + if (!last_stmt) > > + last_stmt > > + = (node->si ? node->si > > + : vect_find_last_scalar_stmt_in_slp (node)->stmt); > > + if (!vect_stmt_dominates_stmt_p (last_stmt, use_stmt)) > > { > > if (dump_enabled_p ()) > > dump_printf_loc (MSG_MISSED_OPTIMIZATION, > > vect_location, > > @@ -10448,6 +10453,9 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo, > > int n_stmts, bool &fatal, > > /* Mark all the statements that we vectorize. */ > > vect_bb_slp_mark_stmts_vectorized (bb_vinfo); > > > > + /* Compute vector stmt placement. */ > > + vect_schedule_slp (bb_vinfo, BB_VINFO_SLP_INSTANCES (bb_vinfo), true); > > + > > /* Compute vectorizable live stmts. */ > > vect_bb_slp_mark_live_stmts (bb_vinfo); > > > > @@ -10602,12 +10610,11 @@ vect_slp_region (vec<basic_block> bbs, > > vec<data_reference_p> datarefs, > > dump_user_location_t saved_vect_location = vect_location; > > vect_location = instance->location (); > > > > - vect_schedule_slp (bb_vinfo, instance->subgraph_entries); > > + vect_schedule_slp (bb_vinfo, instance->subgraph_entries, false); > > > > vect_location = saved_vect_location; > > } > > > > - > > /* Generate the invariant statements. */ > > if (!gimple_seq_empty_p (bb_vinfo->inv_pattern_def_seq)) > > { > > @@ -12049,13 +12056,13 @@ vectorizable_slp_permutation (vec_info > > *vinfo, gimple_stmt_iterator *gsi, > > return true; > > } > > > > -/* Vectorize SLP NODE. */ > > +/* Vectorize SLP NODE. Only compute the vector insertion places when > > + PLACE_ONLY is true. */ > > > > static void > > vect_schedule_slp_node (vec_info *vinfo, > > - slp_tree node, slp_instance instance) > > + slp_tree node, slp_instance instance, bool place_only) > > { > > - gimple_stmt_iterator si; > > int i; > > slp_tree child; > > > > @@ -12063,6 +12070,9 @@ vect_schedule_slp_node (vec_info *vinfo, > > if (SLP_TREE_DEF_TYPE (node) == vect_constant_def > > || SLP_TREE_DEF_TYPE (node) == vect_external_def) > > { > > + if (place_only) > > + return; > > + > > /* ??? vectorizable_shift can end up using a scalar operand which is > > currently denoted as !SLP_TREE_VECTYPE. No need to vectorize the > > node in this case. */ > > @@ -12082,9 +12092,15 @@ vect_schedule_slp_node (vec_info *vinfo, > > stmt_vec_info stmt_info = SLP_TREE_REPRESENTATIVE (node); > > > > gcc_assert (SLP_TREE_VEC_DEFS (node).is_empty ()); > > - if (SLP_TREE_VECTYPE (node)) > > + if (!place_only && SLP_TREE_VECTYPE (node)) > > SLP_TREE_VEC_DEFS (node).create (vect_get_num_copies (vinfo, node)); > > > > + gimple *last_stmt; > > + gimple_stmt_iterator si; > > + /* ??? When !place_only we'd like to re-use place_only computed info, > > + but this is a bit awkward due to using gsi_insert_before and the > > + requirement to insert after vector defs. So we compute last_stmt > > + during pre-scheduling and si during scheduling. */ > > if (!SLP_TREE_PERMUTE_P (node) && STMT_VINFO_DATA_REF (stmt_info)) > > { > > /* Vectorized loads go before the first scalar load to make it > > @@ -12095,7 +12111,8 @@ vect_schedule_slp_node (vec_info *vinfo, > > last_stmt_info = vect_find_first_scalar_stmt_in_slp (node); > > else /* DR_IS_WRITE */ > > last_stmt_info = vect_find_last_scalar_stmt_in_slp (node); > > - si = gsi_for_stmt (last_stmt_info->stmt); > > + last_stmt = last_stmt_info->stmt; > > + si = gsi_for_stmt (last_stmt); > > } > > else if (!SLP_TREE_PERMUTE_P (node) > > && (SLP_TREE_TYPE (node) == cycle_phi_info_type > > @@ -12103,15 +12120,59 @@ vect_schedule_slp_node (vec_info *vinfo, > > || SLP_TREE_TYPE (node) == phi_info_type)) > > { > > /* For PHI node vectorization we do not use the insertion iterator. */ > > + last_stmt = SLP_TREE_SCALAR_STMTS (node)[0]->stmt; > > si = gsi_none (); > > } > > else > > { > > /* Emit other stmts after the children vectorized defs which is > > earliest possible. */ > > - gimple *last_stmt = NULL; > > + last_stmt = NULL; > > FOR_EACH_VEC_ELT (SLP_TREE_CHILDREN (node), i, child) > > - if (SLP_TREE_DEF_TYPE (child) == vect_internal_def) > > + if (place_only) > > + { > > + gimple *vstmt = child->si; > > + if (!vstmt) > > + { > > + if (SLP_TREE_DEF_TYPE (child) == vect_external_def) > > + { > > + vec<tree> &defs > > + = (!SLP_TREE_SCALAR_OPS (child).is_empty () > > + ? SLP_TREE_SCALAR_OPS (child) > > + : SLP_TREE_VEC_DEFS (child)); > > + for (tree def : defs) > > + /* If the stmt is not inside the region do not > > + use it as possible insertion point. */ > > + if (auto stmt_info = vinfo->lookup_def (def)) > > + { > > + gimple *stmt = stmt_info->stmt; > > + if (!last_stmt) > > + last_stmt = stmt; > > + else if (vect_stmt_dominates_stmt_p (last_stmt, > > stmt)) > > + last_stmt = stmt; > > + else if (vect_stmt_dominates_stmt_p (stmt, > > last_stmt)) > > + ; > > + else > > + gcc_unreachable (); > > + } > > + } > > + else > > + /* vect_constant_def and defs at region boundary do not > > + constrain placement. */ > > + gcc_assert (SLP_TREE_DEF_TYPE (child) == vect_constant_def > > + /* ??? Region boundary is not representated. */ > > + || true); > > + } > > + else if (!last_stmt) > > + last_stmt = vstmt; > > + else if (vect_stmt_dominates_stmt_p (last_stmt, vstmt)) > > + last_stmt = vstmt; > > + else if (vect_stmt_dominates_stmt_p (vstmt, last_stmt)) > > + ; > > + else > > + gcc_unreachable (); > > + } > > + else if (SLP_TREE_DEF_TYPE (child) == vect_internal_def) > > { > > /* For fold-left reductions we are retaining the scalar > > reduction PHI but we still have SLP_TREE_NUM_VEC_STMTS > > @@ -12225,11 +12286,16 @@ vect_schedule_slp_node (vec_info *vinfo, > > gimple_bb (stmt_info->stmt), > > gimple_bb (last_stmt))); > > si = gsi_after_labels (gimple_bb (stmt_info->stmt)); > > + last_stmt = gsi_stmt (si); > > } > > /* When there is no in-region child def to guide placement, insert > > at region boundary. */ > > else if (!last_stmt) > > - si = gsi_after_labels (vinfo->bbs[0]); > > + { > > + si = gsi_after_labels (vinfo->bbs[0]); > > + /* ??? last_stmt can be NULL if the block is empty. */ > > + last_stmt = gsi_stmt (si); > > + } > > else if (is_a <gphi *> (last_stmt)) > > si = gsi_after_labels (gimple_bb (last_stmt)); > > else > > @@ -12241,6 +12307,8 @@ vect_schedule_slp_node (vec_info *vinfo, > > > > if (auto loop_vinfo = dyn_cast <loop_vec_info> (vinfo)) > > { > > + /* We'll have to fix this up for loop vect. */ > > + gcc_assert (!place_only); > > /* Avoid scheduling stmts to random places in the CFG, any > > stmt dominance check we performed is possibly wrong as > > UIDs > > are not initialized for all of the function for loop > > @@ -12274,6 +12342,44 @@ vect_schedule_slp_node (vec_info *vinfo, > > } > > } > > > > + if (place_only) > > + { > > + if (dump_enabled_p () && last_stmt) > > + dump_printf_loc (MSG_NOTE, vect_location, > > + "placing node %p at %G:", (void *)node, last_stmt); > > + gcc_assert ((last_stmt && gimple_bb (last_stmt)) || gsi_bb (si)); > > + node->si = last_stmt; > > + return; > > + } > > + > > + /* ??? Asserting vect_stmt_dominates_stmt_p (gsi_stmt (si), node->si) > > + does not work because in some cases we advance si from last_stmt (as > > + we want to insert after vector stmts) and because vector stmts of > > + children have been inserted possibly at the same location constraint, > > + moving si even further. */ > > + if (flag_checking && node->si && gimple_bb (node->si) && !gsi_end_p (si)) > > + { > > + auto gsi2 = si; > > + while (1) > > + { > > + if (vect_stmt_dominates_stmt_p (gsi_stmt (gsi2), node->si)) > > + break; > > + /* As we have possibly advanced si it might now point to the > > + scalar stmt immediately following node->si. That's OK. */ > > + if (gsi_stmt (gsi2) != gsi_stmt (si) > > + && gimple_uid (gsi_stmt (gsi2)) != 0) > > + gcc_unreachable (); > > + gsi_prev (&gsi2); > > + if (gsi_end_p (gsi2)) > > + { > > + if (is_a <gphi *> (node->si) > > + && gimple_bb (node->si) == gsi_bb (si)) > > + break; > > + gcc_unreachable (); > > + } > > + } > > + } > > + > > if (dump_enabled_p ()) > > { > > if (stmt_info) > > @@ -12499,7 +12605,7 @@ struct slp_scc_info > > static void > > vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance, > > hash_map<slp_tree, slp_scc_info> &scc_info, > > - int &maxdfs, vec<slp_tree> &stack) > > + int &maxdfs, vec<slp_tree> &stack, bool place_only) > > { > > bool existed_p; > > slp_scc_info *info = &scc_info.get_or_insert (node, &existed_p); > > @@ -12512,7 +12618,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree > > node, slp_instance instance, > > if (SLP_TREE_DEF_TYPE (node) != vect_internal_def) > > { > > info->on_stack = false; > > - vect_schedule_slp_node (vinfo, node, instance); > > + vect_schedule_slp_node (vinfo, node, instance, place_only); > > return; > > } > > > > @@ -12529,7 +12635,8 @@ vect_schedule_scc (vec_info *vinfo, slp_tree > > node, slp_instance instance, > > slp_scc_info *child_info = scc_info.get (child); > > if (!child_info) > > { > > - vect_schedule_scc (vinfo, child, instance, scc_info, maxdfs, stack); > > + vect_schedule_scc (vinfo, child, instance, scc_info, maxdfs, stack, > > + place_only); > > /* Recursion might have re-allocated the node. */ > > info = scc_info.get (node); > > child_info = scc_info.get (child); > > @@ -12548,7 +12655,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree > > node, slp_instance instance, > > { > > stack.pop (); > > info->on_stack = false; > > - vect_schedule_slp_node (vinfo, node, instance); > > + vect_schedule_slp_node (vinfo, node, instance, place_only); > > if (!SLP_TREE_PERMUTE_P (node) > > && is_a <gphi *> (SLP_TREE_REPRESENTATIVE (node)->stmt)) > > phis_to_fixup.quick_push (node); > > @@ -12600,7 +12707,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree > > node, slp_instance instance, > > } > > if (ready) > > { > > - vect_schedule_slp_node (vinfo, entry, instance); > > + vect_schedule_slp_node (vinfo, entry, instance, place_only); > > scc_info.get (entry)->on_stack = false; > > stack[idx] = NULL; > > todo--; > > @@ -12615,6 +12722,9 @@ vect_schedule_scc (vec_info *vinfo, slp_tree > > node, slp_instance instance, > > stack.truncate (last_idx); > > } > > > > + if (place_only) > > + return; > > + > > /* Now fixup the backedge def of the vectorized PHIs in this SCC. */ > > slp_tree phi_node; > > FOR_EACH_VEC_ELT (phis_to_fixup, i, phi_node) > > @@ -12665,10 +12775,12 @@ vect_schedule_scc (vec_info *vinfo, slp_tree > > node, slp_instance instance, > > } > > } > > > > -/* Generate vector code for SLP_INSTANCES in the loop/basic block. */ > > +/* Generate vector code for SLP_INSTANCES in the loop/basic block. Perform > > + vector stmt placement only when PLACE_ONLY is true. */ > > > > void > > -vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances) > > +vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances, > > + bool place_only) > > { > > slp_instance instance; > > unsigned int i; > > @@ -12678,7 +12790,7 @@ vect_schedule_slp (vec_info *vinfo, const > > vec<slp_instance> &slp_instances) > > FOR_EACH_VEC_ELT (slp_instances, i, instance) > > { > > slp_tree node = SLP_INSTANCE_TREE (instance); > > - if (dump_enabled_p ()) > > + if (!place_only && dump_enabled_p ()) > > { > > dump_printf_loc (MSG_NOTE, vect_location, > > "Vectorizing SLP tree:\n"); > > @@ -12693,16 +12805,20 @@ vect_schedule_slp (vec_info *vinfo, const > > vec<slp_instance> &slp_instances) > > have a PHI be the node breaking the cycle. */ > > auto_vec<slp_tree> stack; > > if (!scc_info.get (node)) > > - vect_schedule_scc (vinfo, node, instance, scc_info, maxdfs, stack); > > + vect_schedule_scc (vinfo, node, instance, scc_info, maxdfs, stack, > > + place_only); > > > > - if (!SLP_INSTANCE_ROOT_STMTS (instance).is_empty ()) > > + if (!place_only && !SLP_INSTANCE_ROOT_STMTS (instance).is_empty ()) > > vectorize_slp_instance_root_stmt (vinfo, node, instance); > > > > - if (dump_enabled_p ()) > > + if (!place_only && dump_enabled_p ()) > > dump_printf_loc (MSG_NOTE, vect_location, > > "vectorizing stmts using SLP.\n"); > > } > > > > + if (place_only) > > + return; > > + > > FOR_EACH_VEC_ELT (slp_instances, i, instance) > > { > > slp_tree root = SLP_INSTANCE_TREE (instance); > > diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h > > index 94ef81383f4..e79b9087b21 100644 > > --- a/gcc/tree-vectorizer.h > > +++ b/gcc/tree-vectorizer.h > > @@ -358,6 +358,8 @@ struct _slp_tree { > > tree vectype; > > /* Vectorized defs. */ > > vec<tree> vec_defs; > > + /* Insertion place. */ > > + gimple *si; > > > > /* Reference count in the SLP graph. */ > > unsigned int refcnt; > > @@ -2775,7 +2777,7 @@ extern bool vect_transform_slp_perm_load > > (vec_info *, slp_tree, const vec<tree> > > extern bool vectorizable_slp_permutation (vec_info *, gimple_stmt_iterator *, > > slp_tree, stmt_vector_for_cost *); > > extern bool vect_slp_analyze_operations (vec_info *); > > -extern void vect_schedule_slp (vec_info *, const vec<slp_instance> &); > > +extern void vect_schedule_slp (vec_info *, const vec<slp_instance> &, bool); > > extern opt_result vect_analyze_slp (vec_info *, unsigned, bool); > > extern bool vect_make_slp_decision (loop_vec_info); > > extern bool vect_detect_hybrid_slp (loop_vec_info); > > -- > > 2.51.0 > -- Richard Biener <[email protected]> SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg, Germany; GF: Jochen Jaser, Andrew McDonald, Abhinav Puri; (HRB 36809, AG Nuernberg)