[PATCH] tree-optimization/126028 - vector placement and live lane extracts
Richard Biener <[email protected]> Mon, 3 Aug 2026 14:47:20 +0200 (CEST)
| Newsgroups | gmane.comp.gcc.patches |
|---|---|
| Message-ID | <[email protected]> |
We currently conservatively assume vector stmt placement when
determining whether we can place a lane extraction for a live
scalar value. The following attempts to improve this by
pre-computing an insertion place for all nodes and using that
to improve live lane extract feasibility computation.
While we'd like to re-use such computation for the actual insertion
there are issues preventing that such as scalar stmts eventually
getting elided and thus iterators invalidated. So the following
re-computes the scheduling again but at least validates the earlier
schedule against it.
For PR126099 I have to turn this into schedule validation, but
not as part of this initial work.
The major pain-point of chosing a gimple * as scheduling anchor
are that for placement in empty BBs that is NULL (this hits us
with region boundaries), a gsi would be able to handle this
case. But then, as you can see in the blob that attempts to
verify the early schedule constraints hold at transform time
there's issues like the scalar stmts vanishing and statements
being inserted. As most viable and ugly option I briefly
considered inserting GIMPLE_NOPs as anchors. Instead of
vect_schedule_slp_node computing the insertion place this
could be also handled (or just overridden?) by vectorizable_*
which sometimes has its own idea where to place.
Bootstrapped and tested on x86_64-unknown-linux-gnu.
Any comments?
Thanks,
Richard.
PR tree-optimization/126028
* tree-vectorizer.h (_slp_tree::si): New member.
(vect_schedule_slp): Adjust prototype.
* tree-vect-loop.cc (vect_transform_loop): Adjust.
* tree-vect-slp.cc (_slp_tree::_slp_tree): Initialize si.
(vect_bb_slp_mark_live_stmts): Use vector stmt placement
constraint computed by pre-scheduling of SLP nodes.
(vect_slp_analyze_bb_1): Pre-schedule SLP nodes before
computing live lane extraction points.
(vect_slp_region): Adjust.
(vect_schedule_slp_node): Add parameter indicating to whether
we should perform pre-scehduling or not. Compute the
insertion place suitable for dominance checks when
pre-scheduling.
(vect_schedule_scc): Adjust.
(vect_schedule_slp): Likewise.
* gcc.dg/vect/bb-slp-46.c: Un-XFAIL.
* gcc.dg/vect/bb-slp-pr126053.c: New testcase.
* gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c: Likewise.
---
gcc/testsuite/gcc.dg/vect/bb-slp-46.c | 12 +-
gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c | 63 +++++++
.../costmodel/x86_64/costmodel-pr126028.c | 63 +++++++
gcc/tree-vect-loop.cc | 3 +-
gcc/tree-vect-slp.cc | 162 +++++++++++++++---
gcc/tree-vectorizer.h | 4 +-
6 files changed, 275 insertions(+), 32 deletions(-)
create mode 100644 gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c
create mode 100644 gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c
diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-46.c b/gcc/testsuite/gcc.dg/vect/bb-slp-46.c
index 4eceea44efc..ff4a64a011b 100644
--- a/gcc/testsuite/gcc.dg/vect/bb-slp-46.c
+++ b/gcc/testsuite/gcc.dg/vect/bb-slp-46.c
@@ -6,7 +6,7 @@ int a[4], b[4];
int foo ()
{
int tem0 = a[0] + b[0];
- int temx = tem0 * 17; /* this fails without a real need */
+ int temx = tem0 * 17;
int tem1 = a[1] + b[1];
int tem2 = a[2] + b[2];
int tem3 = a[3] + b[3];
@@ -18,11 +18,9 @@ int foo ()
return temx / temy;
}
-/* We should extract the live lane from the vectorized add rather than
- keeping the original scalar add.
- ??? Because of a too conservative check we fail for temx here. */
+/* We should extract the live lanes from the vectorized mul rather than
+ keeping the original scalar add. */
/* { dg-final { scan-tree-dump "optimized: basic block" "slp2" } } */
-/* { dg-final { scan-tree-dump "extracting lane for live stmt" "slp2" } } */
-/* { dg-final { scan-tree-dump-times "extracting lane for live stmt" 2 "slp2" { xfail *-*-* } } } */
+/* { dg-final { scan-tree-dump-times "extracting lane for live stmt" 2 "slp2" } } */
/* { dg-final { scan-tree-dump-not "tem3_\[0-9\]\+ = " "optimized" } } */
-/* { dg-final { scan-tree-dump-not "tem0_\[0-9\]\+ = " "optimized" { xfail *-*-* } } } */
+/* { dg-final { scan-tree-dump-not "tem0_\[0-9\]\+ = " "optimized" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c b/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c
new file mode 100644
index 00000000000..018810d37b8
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c
@@ -0,0 +1,63 @@
+/* { dg-do compile } */
+
+typedef double BigReal;
+
+struct Position { BigReal x, y, z; };
+
+struct CompAtom {
+ struct Position position;
+ float charge;
+ short vdwType;
+ unsigned char partition;
+ unsigned char nonbondedGroupSize;
+};
+
+void cull_goodgroups(const struct CompAtom * __restrict p_1,
+ const int * __restrict glist,
+ int gu,
+ BigReal p_i_x, BigReal p_i_y, BigReal p_i_z,
+ BigReal groupplcutoff2,
+ int * __restrict goodglist)
+{
+ int hu = 0;
+ int g = 0;
+ int jprev0 = glist[0];
+ int jprev1 = glist[1];
+ int j0, j1;
+
+ BigReal pj_x_0, pj_x_1, pj_y_0, pj_y_1, pj_z_0, pj_z_1;
+ BigReal t_0, t_1, r2_0, r2_1;
+
+ pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x;
+ pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y;
+ pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z;
+ g += 2;
+
+ for ( ; g < gu - 2; g += 2 ) {
+ j0 = jprev0;
+ j1 = jprev1;
+
+ t_0 = p_i_x - pj_x_0; t_1 = p_i_x - pj_x_1;
+ r2_0 = t_0 * t_0; r2_1 = t_1 * t_1;
+ t_0 = p_i_y - pj_y_0; t_1 = p_i_y - pj_y_1;
+ r2_0 += t_0 * t_0; r2_1 += t_1 * t_1;
+ t_0 = p_i_z - pj_z_0; t_1 = p_i_z - pj_z_1;
+ r2_0 += t_0 * t_0; r2_1 += t_1 * t_1;
+
+ jprev0 = glist[g];
+ jprev1 = glist[g + 1];
+ pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x;
+ pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y;
+ pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z;
+
+ bool test0 = (r2_0 < groupplcutoff2);
+ bool test1 = (r2_1 < groupplcutoff2);
+
+ goodglist[hu] = j0;
+ goodglist[hu + test0] = j1;
+ hu += test0 + test1;
+ }
+}
+
+/* { dg-final { scan-tree-dump "58:12: optimized: basic block part vectorized" "slp1" { target x86_64-*-* } } } */
+/* { dg-final { scan-tree-dump-not "Cannot determine insertion place" "slp1" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c b/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c
new file mode 100644
index 00000000000..15db1d390d9
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c
@@ -0,0 +1,63 @@
+/* { dg-do compile } */
+/* { dg-additional-options "-fdump-tree-slp1 -fopt-info-vec" } */
+
+typedef double BigReal;
+
+struct Position { BigReal x, y, z; };
+
+struct CompAtom {
+ struct Position position;
+ float charge;
+ short vdwType;
+ unsigned char partition;
+ unsigned char nonbondedGroupSize;
+};
+
+void cull_goodgroups(const struct CompAtom * __restrict p_1,
+ const int * __restrict glist,
+ int gu,
+ BigReal p_i_x, BigReal p_i_y, BigReal p_i_z,
+ BigReal groupplcutoff2,
+ int * __restrict goodglist)
+{
+ int hu = 0;
+ int g = 0;
+ int jprev0 = glist[0];
+ int jprev1 = glist[1];
+ int j0, j1;
+
+ BigReal pj_x_0, pj_x_1, pj_y_0, pj_y_1, pj_z_0, pj_z_1;
+ BigReal t_0, t_1, r2_0, r2_1;
+
+ pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x;
+ pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y;
+ pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z;
+ g += 2;
+
+ for ( ; g < gu - 2; g += 2 ) {
+ j0 = jprev0;
+ j1 = jprev1;
+
+ t_0 = p_i_x - pj_x_0; t_1 = p_i_x - pj_x_1;
+ r2_0 = t_0 * t_0; r2_1 = t_1 * t_1;
+ t_0 = p_i_y - pj_y_0; t_1 = p_i_y - pj_y_1;
+ r2_0 += t_0 * t_0; r2_1 += t_1 * t_1;
+ t_0 = p_i_z - pj_z_0; t_1 = p_i_z - pj_z_1;
+ r2_0 += t_0 * t_0; r2_1 += t_1 * t_1;
+
+ jprev0 = glist[g];
+ jprev1 = glist[g + 1];
+ pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x;
+ pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y;
+ pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z;
+
+ bool test0 = (r2_0 < groupplcutoff2);
+ bool test1 = (r2_1 < groupplcutoff2);
+
+ goodglist[hu] = j0;
+ goodglist[hu + test0] = j1;
+ hu += test0 + test1; /* { dg-optimized "basic block part vectorized using 8 byte vectors" } */
+ }
+}
+
+/* { dg-final { scan-tree-dump-times " = vect_t\[^ \]* \\\* vect_t" 3 "slp1" } } */
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 31de29c036b..a336df81bc7 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -11406,7 +11406,8 @@ vect_transform_loop (loop_vec_info loop_vinfo, gimple *loop_vectorized_call)
if (!loop_vinfo->slp_instances.is_empty ())
{
DUMP_VECT_SCOPE ("scheduling SLP instances");
- vect_schedule_slp (loop_vinfo, LOOP_VINFO_SLP_INSTANCES (loop_vinfo));
+ vect_schedule_slp (loop_vinfo, LOOP_VINFO_SLP_INSTANCES (loop_vinfo),
+ false);
}
/* Generate the loop invariant statements. */
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index 8ed7a0da377..dda9e046ade 100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -135,6 +135,7 @@ _slp_tree::_slp_tree ()
this->lanes = 0;
SLP_TREE_TYPE (this) = undef_vec_info_type;
this->data = NULL;
+ this->si = NULL;
}
/* Tear down a SLP node. */
@@ -9277,7 +9278,7 @@ vect_bb_slp_mark_live_stmts (bb_vec_info bb_vinfo, slp_tree node,
unsigned i;
stmt_vec_info stmt_info;
- stmt_vec_info last_stmt = vect_find_last_scalar_stmt_in_slp (node);
+ gimple *last_stmt = NULL;
FOR_EACH_VEC_ELT (SLP_TREE_SCALAR_STMTS (node), i, stmt_info)
{
if (!stmt_info || svisited.contains (stmt_info))
@@ -9329,7 +9330,11 @@ vect_bb_slp_mark_live_stmts (bb_vec_info bb_vinfo, slp_tree node,
|| !PURE_SLP_STMT (use_stmt_info)))
{
live_p = true;
- if (!vect_stmt_dominates_stmt_p (last_stmt->stmt, use_stmt))
+ if (!last_stmt)
+ last_stmt
+ = (node->si ? node->si
+ : vect_find_last_scalar_stmt_in_slp (node)->stmt);
+ if (!vect_stmt_dominates_stmt_p (last_stmt, use_stmt))
{
if (dump_enabled_p ())
dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
@@ -10448,6 +10453,9 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo, int n_stmts, bool &fatal,
/* Mark all the statements that we vectorize. */
vect_bb_slp_mark_stmts_vectorized (bb_vinfo);
+ /* Compute vector stmt placement. */
+ vect_schedule_slp (bb_vinfo, BB_VINFO_SLP_INSTANCES (bb_vinfo), true);
+
/* Compute vectorizable live stmts. */
vect_bb_slp_mark_live_stmts (bb_vinfo);
@@ -10602,12 +10610,11 @@ vect_slp_region (vec<basic_block> bbs, vec<data_reference_p> datarefs,
dump_user_location_t saved_vect_location = vect_location;
vect_location = instance->location ();
- vect_schedule_slp (bb_vinfo, instance->subgraph_entries);
+ vect_schedule_slp (bb_vinfo, instance->subgraph_entries, false);
vect_location = saved_vect_location;
}
-
/* Generate the invariant statements. */
if (!gimple_seq_empty_p (bb_vinfo->inv_pattern_def_seq))
{
@@ -12049,13 +12056,13 @@ vectorizable_slp_permutation (vec_info *vinfo, gimple_stmt_iterator *gsi,
return true;
}
-/* Vectorize SLP NODE. */
+/* Vectorize SLP NODE. Only compute the vector insertion places when
+ PLACE_ONLY is true. */
static void
vect_schedule_slp_node (vec_info *vinfo,
- slp_tree node, slp_instance instance)
+ slp_tree node, slp_instance instance, bool place_only)
{
- gimple_stmt_iterator si;
int i;
slp_tree child;
@@ -12063,6 +12070,9 @@ vect_schedule_slp_node (vec_info *vinfo,
if (SLP_TREE_DEF_TYPE (node) == vect_constant_def
|| SLP_TREE_DEF_TYPE (node) == vect_external_def)
{
+ if (place_only)
+ return;
+
/* ??? vectorizable_shift can end up using a scalar operand which is
currently denoted as !SLP_TREE_VECTYPE. No need to vectorize the
node in this case. */
@@ -12082,9 +12092,15 @@ vect_schedule_slp_node (vec_info *vinfo,
stmt_vec_info stmt_info = SLP_TREE_REPRESENTATIVE (node);
gcc_assert (SLP_TREE_VEC_DEFS (node).is_empty ());
- if (SLP_TREE_VECTYPE (node))
+ if (!place_only && SLP_TREE_VECTYPE (node))
SLP_TREE_VEC_DEFS (node).create (vect_get_num_copies (vinfo, node));
+ gimple *last_stmt;
+ gimple_stmt_iterator si;
+ /* ??? When !place_only we'd like to re-use place_only computed info,
+ but this is a bit awkward due to using gsi_insert_before and the
+ requirement to insert after vector defs. So we compute last_stmt
+ during pre-scheduling and si during scheduling. */
if (!SLP_TREE_PERMUTE_P (node) && STMT_VINFO_DATA_REF (stmt_info))
{
/* Vectorized loads go before the first scalar load to make it
@@ -12095,7 +12111,8 @@ vect_schedule_slp_node (vec_info *vinfo,
last_stmt_info = vect_find_first_scalar_stmt_in_slp (node);
else /* DR_IS_WRITE */
last_stmt_info = vect_find_last_scalar_stmt_in_slp (node);
- si = gsi_for_stmt (last_stmt_info->stmt);
+ last_stmt = last_stmt_info->stmt;
+ si = gsi_for_stmt (last_stmt);
}
else if (!SLP_TREE_PERMUTE_P (node)
&& (SLP_TREE_TYPE (node) == cycle_phi_info_type
@@ -12103,15 +12120,59 @@ vect_schedule_slp_node (vec_info *vinfo,
|| SLP_TREE_TYPE (node) == phi_info_type))
{
/* For PHI node vectorization we do not use the insertion iterator. */
+ last_stmt = SLP_TREE_SCALAR_STMTS (node)[0]->stmt;
si = gsi_none ();
}
else
{
/* Emit other stmts after the children vectorized defs which is
earliest possible. */
- gimple *last_stmt = NULL;
+ last_stmt = NULL;
FOR_EACH_VEC_ELT (SLP_TREE_CHILDREN (node), i, child)
- if (SLP_TREE_DEF_TYPE (child) == vect_internal_def)
+ if (place_only)
+ {
+ gimple *vstmt = child->si;
+ if (!vstmt)
+ {
+ if (SLP_TREE_DEF_TYPE (child) == vect_external_def)
+ {
+ vec<tree> &defs
+ = (!SLP_TREE_SCALAR_OPS (child).is_empty ()
+ ? SLP_TREE_SCALAR_OPS (child)
+ : SLP_TREE_VEC_DEFS (child));
+ for (tree def : defs)
+ /* If the stmt is not inside the region do not
+ use it as possible insertion point. */
+ if (auto stmt_info = vinfo->lookup_def (def))
+ {
+ gimple *stmt = stmt_info->stmt;
+ if (!last_stmt)
+ last_stmt = stmt;
+ else if (vect_stmt_dominates_stmt_p (last_stmt, stmt))
+ last_stmt = stmt;
+ else if (vect_stmt_dominates_stmt_p (stmt, last_stmt))
+ ;
+ else
+ gcc_unreachable ();
+ }
+ }
+ else
+ /* vect_constant_def and defs at region boundary do not
+ constrain placement. */
+ gcc_assert (SLP_TREE_DEF_TYPE (child) == vect_constant_def
+ /* ??? Region boundary is not representated. */
+ || true);
+ }
+ else if (!last_stmt)
+ last_stmt = vstmt;
+ else if (vect_stmt_dominates_stmt_p (last_stmt, vstmt))
+ last_stmt = vstmt;
+ else if (vect_stmt_dominates_stmt_p (vstmt, last_stmt))
+ ;
+ else
+ gcc_unreachable ();
+ }
+ else if (SLP_TREE_DEF_TYPE (child) == vect_internal_def)
{
/* For fold-left reductions we are retaining the scalar
reduction PHI but we still have SLP_TREE_NUM_VEC_STMTS
@@ -12225,11 +12286,16 @@ vect_schedule_slp_node (vec_info *vinfo,
gimple_bb (stmt_info->stmt),
gimple_bb (last_stmt)));
si = gsi_after_labels (gimple_bb (stmt_info->stmt));
+ last_stmt = gsi_stmt (si);
}
/* When there is no in-region child def to guide placement, insert
at region boundary. */
else if (!last_stmt)
- si = gsi_after_labels (vinfo->bbs[0]);
+ {
+ si = gsi_after_labels (vinfo->bbs[0]);
+ /* ??? last_stmt can be NULL if the block is empty. */
+ last_stmt = gsi_stmt (si);
+ }
else if (is_a <gphi *> (last_stmt))
si = gsi_after_labels (gimple_bb (last_stmt));
else
@@ -12241,6 +12307,8 @@ vect_schedule_slp_node (vec_info *vinfo,
if (auto loop_vinfo = dyn_cast <loop_vec_info> (vinfo))
{
+ /* We'll have to fix this up for loop vect. */
+ gcc_assert (!place_only);
/* Avoid scheduling stmts to random places in the CFG, any
stmt dominance check we performed is possibly wrong as UIDs
are not initialized for all of the function for loop
@@ -12274,6 +12342,44 @@ vect_schedule_slp_node (vec_info *vinfo,
}
}
+ if (place_only)
+ {
+ if (dump_enabled_p () && last_stmt)
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "placing node %p at %G:", (void *)node, last_stmt);
+ gcc_assert ((last_stmt && gimple_bb (last_stmt)) || gsi_bb (si));
+ node->si = last_stmt;
+ return;
+ }
+
+ /* ??? Asserting vect_stmt_dominates_stmt_p (gsi_stmt (si), node->si)
+ does not work because in some cases we advance si from last_stmt (as
+ we want to insert after vector stmts) and because vector stmts of
+ children have been inserted possibly at the same location constraint,
+ moving si even further. */
+ if (flag_checking && node->si && gimple_bb (node->si) && !gsi_end_p (si))
+ {
+ auto gsi2 = si;
+ while (1)
+ {
+ if (vect_stmt_dominates_stmt_p (gsi_stmt (gsi2), node->si))
+ break;
+ /* As we have possibly advanced si it might now point to the
+ scalar stmt immediately following node->si. That's OK. */
+ if (gsi_stmt (gsi2) != gsi_stmt (si)
+ && gimple_uid (gsi_stmt (gsi2)) != 0)
+ gcc_unreachable ();
+ gsi_prev (&gsi2);
+ if (gsi_end_p (gsi2))
+ {
+ if (is_a <gphi *> (node->si)
+ && gimple_bb (node->si) == gsi_bb (si))
+ break;
+ gcc_unreachable ();
+ }
+ }
+ }
+
if (dump_enabled_p ())
{
if (stmt_info)
@@ -12499,7 +12605,7 @@ struct slp_scc_info
static void
vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance,
hash_map<slp_tree, slp_scc_info> &scc_info,
- int &maxdfs, vec<slp_tree> &stack)
+ int &maxdfs, vec<slp_tree> &stack, bool place_only)
{
bool existed_p;
slp_scc_info *info = &scc_info.get_or_insert (node, &existed_p);
@@ -12512,7 +12618,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance,
if (SLP_TREE_DEF_TYPE (node) != vect_internal_def)
{
info->on_stack = false;
- vect_schedule_slp_node (vinfo, node, instance);
+ vect_schedule_slp_node (vinfo, node, instance, place_only);
return;
}
@@ -12529,7 +12635,8 @@ vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance,
slp_scc_info *child_info = scc_info.get (child);
if (!child_info)
{
- vect_schedule_scc (vinfo, child, instance, scc_info, maxdfs, stack);
+ vect_schedule_scc (vinfo, child, instance, scc_info, maxdfs, stack,
+ place_only);
/* Recursion might have re-allocated the node. */
info = scc_info.get (node);
child_info = scc_info.get (child);
@@ -12548,7 +12655,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance,
{
stack.pop ();
info->on_stack = false;
- vect_schedule_slp_node (vinfo, node, instance);
+ vect_schedule_slp_node (vinfo, node, instance, place_only);
if (!SLP_TREE_PERMUTE_P (node)
&& is_a <gphi *> (SLP_TREE_REPRESENTATIVE (node)->stmt))
phis_to_fixup.quick_push (node);
@@ -12600,7 +12707,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance,
}
if (ready)
{
- vect_schedule_slp_node (vinfo, entry, instance);
+ vect_schedule_slp_node (vinfo, entry, instance, place_only);
scc_info.get (entry)->on_stack = false;
stack[idx] = NULL;
todo--;
@@ -12615,6 +12722,9 @@ vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance,
stack.truncate (last_idx);
}
+ if (place_only)
+ return;
+
/* Now fixup the backedge def of the vectorized PHIs in this SCC. */
slp_tree phi_node;
FOR_EACH_VEC_ELT (phis_to_fixup, i, phi_node)
@@ -12665,10 +12775,12 @@ vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance,
}
}
-/* Generate vector code for SLP_INSTANCES in the loop/basic block. */
+/* Generate vector code for SLP_INSTANCES in the loop/basic block. Perform
+ vector stmt placement only when PLACE_ONLY is true. */
void
-vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances)
+vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances,
+ bool place_only)
{
slp_instance instance;
unsigned int i;
@@ -12678,7 +12790,7 @@ vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances)
FOR_EACH_VEC_ELT (slp_instances, i, instance)
{
slp_tree node = SLP_INSTANCE_TREE (instance);
- if (dump_enabled_p ())
+ if (!place_only && dump_enabled_p ())
{
dump_printf_loc (MSG_NOTE, vect_location,
"Vectorizing SLP tree:\n");
@@ -12693,16 +12805,20 @@ vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances)
have a PHI be the node breaking the cycle. */
auto_vec<slp_tree> stack;
if (!scc_info.get (node))
- vect_schedule_scc (vinfo, node, instance, scc_info, maxdfs, stack);
+ vect_schedule_scc (vinfo, node, instance, scc_info, maxdfs, stack,
+ place_only);
- if (!SLP_INSTANCE_ROOT_STMTS (instance).is_empty ())
+ if (!place_only && !SLP_INSTANCE_ROOT_STMTS (instance).is_empty ())
vectorize_slp_instance_root_stmt (vinfo, node, instance);
- if (dump_enabled_p ())
+ if (!place_only && dump_enabled_p ())
dump_printf_loc (MSG_NOTE, vect_location,
"vectorizing stmts using SLP.\n");
}
+ if (place_only)
+ return;
+
FOR_EACH_VEC_ELT (slp_instances, i, instance)
{
slp_tree root = SLP_INSTANCE_TREE (instance);
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 94ef81383f4..e79b9087b21 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -358,6 +358,8 @@ struct _slp_tree {
tree vectype;
/* Vectorized defs. */
vec<tree> vec_defs;
+ /* Insertion place. */
+ gimple *si;
/* Reference count in the SLP graph. */
unsigned int refcnt;
@@ -2775,7 +2777,7 @@ extern bool vect_transform_slp_perm_load (vec_info *, slp_tree, const vec<tree>
extern bool vectorizable_slp_permutation (vec_info *, gimple_stmt_iterator *,
slp_tree, stmt_vector_for_cost *);
extern bool vect_slp_analyze_operations (vec_info *);
-extern void vect_schedule_slp (vec_info *, const vec<slp_instance> &);
+extern void vect_schedule_slp (vec_info *, const vec<slp_instance> &, bool);
extern opt_result vect_analyze_slp (vec_info *, unsigned, bool);
extern bool vect_make_slp_decision (loop_vec_info);
extern bool vect_detect_hybrid_slp (loop_vec_info);
--
2.51.0