RE: [PATCH] tree-optimization/126028 - vector placement and live lane extracts

> -----Original Message-----
> From: Richard Biener <[email protected]>
> Sent: 03 August 2026 13:47
> To: [email protected]
> Cc: Tamar Christina <[email protected]>;
> [email protected]
> Subject: [PATCH] tree-optimization/126028 - vector placement and live lane
> extracts
> 
> We currently conservatively assume vector stmt placement when
> determining whether we can place a lane extraction for a live
> scalar value.  The following attempts to improve this by
> pre-computing an insertion place for all nodes and using that
> to improve live lane extract feasibility computation.
> 
> While we'd like to re-use such computation for the actual insertion
> there are issues preventing that such as scalar stmts eventually
> getting elided and thus iterators invalidated.  So the following
> re-computes the scheduling again but at least validates the earlier
> schedule against it.
> 
> For PR126099 I have to turn this into schedule validation, but
> not as part of this initial work.
> 

What I've been trying to figure out here is why  the insertion point
isn't just at the end of the SLP region of the tree with the lane extract.
Since after scheduling, we should know the dependency order, so for a live
lane it feels like placing the extract immediately after the vector stmt for that node
would be the natural/latest-needed point. At least for bb-slp-46.c I don't immediately
see why a more global/conservative placement is needed.

So I'm pretty sure I'm missing something. Other than that the patch looks
sensible to me.

> The major pain-point of chosing a gimple * as scheduling anchor
> are that for placement in empty BBs that is NULL (this hits us
> with region boundaries), a gsi would be able to handle this

You could store a tuple of bb and gimple* maybe? The bb would
Be the fallback if insertion point is empty at the time and so you
can't get the gimple *.

Thanks,
Tamar

> case.  But then, as you can see in the blob that attempts to
> verify the early schedule constraints hold at transform time
> there's issues like the scalar stmts vanishing and statements
> being inserted.  As most viable and ugly option I briefly
> considered inserting GIMPLE_NOPs as anchors.  Instead of
> vect_schedule_slp_node computing the insertion place this
> could be also handled (or just overridden?) by vectorizable_*
> which sometimes has its own idea where to place.
> 
> Bootstrapped and tested on x86_64-unknown-linux-gnu.
> 
> Any comments?
> 
> Thanks,
> Richard.
> 
> 	PR tree-optimization/126028
> 	* tree-vectorizer.h (_slp_tree::si): New member.
> 	(vect_schedule_slp): Adjust prototype.
> 	* tree-vect-loop.cc (vect_transform_loop): Adjust.
> 	* tree-vect-slp.cc (_slp_tree::_slp_tree): Initialize si.
> 	(vect_bb_slp_mark_live_stmts): Use vector stmt placement
> 	constraint computed by pre-scheduling of SLP nodes.
> 	(vect_slp_analyze_bb_1): Pre-schedule SLP nodes before
> 	computing live lane extraction points.
> 	(vect_slp_region): Adjust.
> 	(vect_schedule_slp_node): Add parameter indicating to whether
> 	we should perform pre-scehduling or not.  Compute the
> 	insertion place suitable for dominance checks when
> 	pre-scheduling.
> 	(vect_schedule_scc): Adjust.
> 	(vect_schedule_slp): Likewise.
> 
> 	* gcc.dg/vect/bb-slp-46.c: Un-XFAIL.
> 	* gcc.dg/vect/bb-slp-pr126053.c: New testcase.
> 	* gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c: Likewise.
> ---
>  gcc/testsuite/gcc.dg/vect/bb-slp-46.c         |  12 +-
>  gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c   |  63 +++++++
>  .../costmodel/x86_64/costmodel-pr126028.c     |  63 +++++++
>  gcc/tree-vect-loop.cc                         |   3 +-
>  gcc/tree-vect-slp.cc                          | 162 +++++++++++++++---
>  gcc/tree-vectorizer.h                         |   4 +-
>  6 files changed, 275 insertions(+), 32 deletions(-)
>  create mode 100644 gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c
>  create mode 100644
> gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c
> 
> diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-46.c
> b/gcc/testsuite/gcc.dg/vect/bb-slp-46.c
> index 4eceea44efc..ff4a64a011b 100644
> --- a/gcc/testsuite/gcc.dg/vect/bb-slp-46.c
> +++ b/gcc/testsuite/gcc.dg/vect/bb-slp-46.c
> @@ -6,7 +6,7 @@ int a[4], b[4];
>  int foo ()
>  {
>    int tem0 = a[0] + b[0];
> -  int temx = tem0 * 17;  /* this fails without a real need */
> +  int temx = tem0 * 17;
>    int tem1 = a[1] + b[1];
>    int tem2 = a[2] + b[2];
>    int tem3 = a[3] + b[3];
> @@ -18,11 +18,9 @@ int foo ()
>    return temx / temy;
>  }
> 
> -/* We should extract the live lane from the vectorized add rather than
> -   keeping the original scalar add.
> -   ???  Because of a too conservative check we fail for temx here.  */
> +/* We should extract the live lanes from the vectorized mul rather than
> +   keeping the original scalar add.  */
>  /* { dg-final { scan-tree-dump "optimized: basic block" "slp2" } } */
> -/* { dg-final { scan-tree-dump "extracting lane for live stmt" "slp2" } } */
> -/* { dg-final { scan-tree-dump-times "extracting lane for live stmt" 2 "slp2" {
> xfail *-*-* } } } */
> +/* { dg-final { scan-tree-dump-times "extracting lane for live stmt" 2 "slp2" } }
> */
>  /* { dg-final { scan-tree-dump-not "tem3_\[0-9\]\+ = " "optimized" } } */
> -/* { dg-final { scan-tree-dump-not "tem0_\[0-9\]\+ = " "optimized" { xfail *-*-
> * } } } */
> +/* { dg-final { scan-tree-dump-not "tem0_\[0-9\]\+ = " "optimized" } } */
> diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c
> b/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c
> new file mode 100644
> index 00000000000..018810d37b8
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c
> @@ -0,0 +1,63 @@
> +/* { dg-do compile } */
> +
> +typedef double BigReal;
> +
> +struct Position { BigReal x, y, z; };
> +
> +struct CompAtom {
> +    struct Position position;
> +    float    charge;
> +    short    vdwType;
> +    unsigned char partition;
> +    unsigned char nonbondedGroupSize;
> +};
> +
> +void cull_goodgroups(const struct CompAtom * __restrict p_1,
> +                     const int      * __restrict glist,
> +                     int gu,
> +                     BigReal p_i_x, BigReal p_i_y, BigReal p_i_z,
> +                     BigReal groupplcutoff2,
> +                     int * __restrict goodglist)
> +{
> +    int hu = 0;
> +    int g = 0;
> +    int jprev0 = glist[0];
> +    int jprev1 = glist[1];
> +    int j0, j1;
> +
> +    BigReal pj_x_0, pj_x_1, pj_y_0, pj_y_1, pj_z_0, pj_z_1;
> +    BigReal t_0, t_1, r2_0, r2_1;
> +
> +    pj_x_0 = p_1[jprev0].position.x;  pj_x_1 = p_1[jprev1].position.x;
> +    pj_y_0 = p_1[jprev0].position.y;  pj_y_1 = p_1[jprev1].position.y;
> +    pj_z_0 = p_1[jprev0].position.z;  pj_z_1 = p_1[jprev1].position.z;
> +    g += 2;
> +
> +    for ( ; g < gu - 2; g += 2 ) {
> +        j0 = jprev0;
> +        j1 = jprev1;
> +
> +        t_0 = p_i_x - pj_x_0;  t_1 = p_i_x - pj_x_1;
> +        r2_0 = t_0 * t_0;      r2_1 = t_1 * t_1;
> +        t_0 = p_i_y - pj_y_0;  t_1 = p_i_y - pj_y_1;
> +        r2_0 += t_0 * t_0;     r2_1 += t_1 * t_1;
> +        t_0 = p_i_z - pj_z_0;  t_1 = p_i_z - pj_z_1;
> +        r2_0 += t_0 * t_0;     r2_1 += t_1 * t_1;
> +
> +        jprev0 = glist[g];
> +        jprev1 = glist[g + 1];
> +        pj_x_0 = p_1[jprev0].position.x;  pj_x_1 = p_1[jprev1].position.x;
> +        pj_y_0 = p_1[jprev0].position.y;  pj_y_1 = p_1[jprev1].position.y;
> +        pj_z_0 = p_1[jprev0].position.z;  pj_z_1 = p_1[jprev1].position.z;
> +
> +        bool test0 = (r2_0 < groupplcutoff2);
> +        bool test1 = (r2_1 < groupplcutoff2);
> +
> +	goodglist[hu]         = j0;
> +        goodglist[hu + test0] = j1;
> +        hu += test0 + test1;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump "58:12: optimized: basic block part vectorized"
> "slp1" { target x86_64-*-* } } } */
> +/* { dg-final { scan-tree-dump-not "Cannot determine insertion place" "slp1"
> } } */
> diff --git a/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-
> pr126028.c b/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-
> pr126028.c
> new file mode 100644
> index 00000000000..15db1d390d9
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c
> @@ -0,0 +1,63 @@
> +/* { dg-do compile } */
> +/* { dg-additional-options "-fdump-tree-slp1 -fopt-info-vec" } */
> +
> +typedef double BigReal;
> +
> +struct Position { BigReal x, y, z; };
> +
> +struct CompAtom {
> +    struct Position position;
> +    float    charge;
> +    short    vdwType;
> +    unsigned char partition;
> +    unsigned char nonbondedGroupSize;
> +};
> +
> +void cull_goodgroups(const struct CompAtom * __restrict p_1,
> +		     const int      * __restrict glist,
> +		     int gu,
> +		     BigReal p_i_x, BigReal p_i_y, BigReal p_i_z,
> +		     BigReal groupplcutoff2,
> +		     int * __restrict goodglist)
> +{
> +  int hu = 0;
> +  int g = 0;
> +  int jprev0 = glist[0];
> +  int jprev1 = glist[1];
> +  int j0, j1;
> +
> +  BigReal pj_x_0, pj_x_1, pj_y_0, pj_y_1, pj_z_0, pj_z_1;
> +  BigReal t_0, t_1, r2_0, r2_1;
> +
> +  pj_x_0 = p_1[jprev0].position.x;  pj_x_1 = p_1[jprev1].position.x;
> +  pj_y_0 = p_1[jprev0].position.y;  pj_y_1 = p_1[jprev1].position.y;
> +  pj_z_0 = p_1[jprev0].position.z;  pj_z_1 = p_1[jprev1].position.z;
> +  g += 2;
> +
> +  for ( ; g < gu - 2; g += 2 ) {
> +      j0 = jprev0;
> +      j1 = jprev1;
> +
> +      t_0 = p_i_x - pj_x_0;  t_1 = p_i_x - pj_x_1;
> +      r2_0 = t_0 * t_0;      r2_1 = t_1 * t_1;
> +      t_0 = p_i_y - pj_y_0;  t_1 = p_i_y - pj_y_1;
> +      r2_0 += t_0 * t_0;     r2_1 += t_1 * t_1;
> +      t_0 = p_i_z - pj_z_0;  t_1 = p_i_z - pj_z_1;
> +      r2_0 += t_0 * t_0;     r2_1 += t_1 * t_1;
> +
> +      jprev0 = glist[g];
> +      jprev1 = glist[g + 1];
> +      pj_x_0 = p_1[jprev0].position.x;  pj_x_1 = p_1[jprev1].position.x;
> +      pj_y_0 = p_1[jprev0].position.y;  pj_y_1 = p_1[jprev1].position.y;
> +      pj_z_0 = p_1[jprev0].position.z;  pj_z_1 = p_1[jprev1].position.z;
> +
> +      bool test0 = (r2_0 < groupplcutoff2);
> +      bool test1 = (r2_1 < groupplcutoff2);
> +
> +      goodglist[hu]         = j0;
> +      goodglist[hu + test0] = j1;
> +      hu += test0 + test1; /* { dg-optimized "basic block part vectorized using 8
> byte vectors" } */
> +  }
> +}
> +
> +/* { dg-final { scan-tree-dump-times " = vect_t\[^ \]* \\\* vect_t" 3 "slp1" } }
> */
> diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
> index 31de29c036b..a336df81bc7 100644
> --- a/gcc/tree-vect-loop.cc
> +++ b/gcc/tree-vect-loop.cc
> @@ -11406,7 +11406,8 @@ vect_transform_loop (loop_vec_info loop_vinfo,
> gimple *loop_vectorized_call)
>    if (!loop_vinfo->slp_instances.is_empty ())
>      {
>        DUMP_VECT_SCOPE ("scheduling SLP instances");
> -      vect_schedule_slp (loop_vinfo, LOOP_VINFO_SLP_INSTANCES
> (loop_vinfo));
> +      vect_schedule_slp (loop_vinfo, LOOP_VINFO_SLP_INSTANCES
> (loop_vinfo),
> +			 false);
>      }
> 
>    /* Generate the loop invariant statements.  */
> diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
> index 8ed7a0da377..dda9e046ade 100644
> --- a/gcc/tree-vect-slp.cc
> +++ b/gcc/tree-vect-slp.cc
> @@ -135,6 +135,7 @@ _slp_tree::_slp_tree ()
>    this->lanes = 0;
>    SLP_TREE_TYPE (this) = undef_vec_info_type;
>    this->data = NULL;
> +  this->si = NULL;
>  }
> 
>  /* Tear down a SLP node.  */
> @@ -9277,7 +9278,7 @@ vect_bb_slp_mark_live_stmts (bb_vec_info
> bb_vinfo, slp_tree node,
> 
>    unsigned i;
>    stmt_vec_info stmt_info;
> -  stmt_vec_info last_stmt = vect_find_last_scalar_stmt_in_slp (node);
> +  gimple *last_stmt = NULL;
>    FOR_EACH_VEC_ELT (SLP_TREE_SCALAR_STMTS (node), i, stmt_info)
>      {
>        if (!stmt_info || svisited.contains (stmt_info))
> @@ -9329,7 +9330,11 @@ vect_bb_slp_mark_live_stmts (bb_vec_info
> bb_vinfo, slp_tree node,
>  		    || !PURE_SLP_STMT (use_stmt_info)))
>  	      {
>  		live_p = true;
> -		if (!vect_stmt_dominates_stmt_p (last_stmt->stmt,
> use_stmt))
> +		if (!last_stmt)
> +		  last_stmt
> +		    = (node->si ? node->si
> +		       : vect_find_last_scalar_stmt_in_slp (node)->stmt);
> +		if (!vect_stmt_dominates_stmt_p (last_stmt, use_stmt))
>  		  {
>  		    if (dump_enabled_p ())
>  		      dump_printf_loc (MSG_MISSED_OPTIMIZATION,
> vect_location,
> @@ -10448,6 +10453,9 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo,
> int n_stmts, bool &fatal,
>    /* Mark all the statements that we vectorize.  */
>    vect_bb_slp_mark_stmts_vectorized (bb_vinfo);
> 
> +  /* Compute vector stmt placement.  */
> +  vect_schedule_slp (bb_vinfo, BB_VINFO_SLP_INSTANCES (bb_vinfo), true);
> +
>    /* Compute vectorizable live stmts.  */
>    vect_bb_slp_mark_live_stmts (bb_vinfo);
> 
> @@ -10602,12 +10610,11 @@ vect_slp_region (vec<basic_block> bbs,
> vec<data_reference_p> datarefs,
>  	      dump_user_location_t saved_vect_location = vect_location;
>  	      vect_location = instance->location ();
> 
> -	      vect_schedule_slp (bb_vinfo, instance->subgraph_entries);
> +	      vect_schedule_slp (bb_vinfo, instance->subgraph_entries, false);
> 
>  	      vect_location = saved_vect_location;
>  	    }
> 
> -
>  	  /* Generate the invariant statements.  */
>  	  if (!gimple_seq_empty_p (bb_vinfo->inv_pattern_def_seq))
>  	    {
> @@ -12049,13 +12056,13 @@ vectorizable_slp_permutation (vec_info
> *vinfo, gimple_stmt_iterator *gsi,
>    return true;
>  }
> 
> -/* Vectorize SLP NODE.  */
> +/* Vectorize SLP NODE.  Only compute the vector insertion places when
> +   PLACE_ONLY is true.  */
> 
>  static void
>  vect_schedule_slp_node (vec_info *vinfo,
> -			slp_tree node, slp_instance instance)
> +			slp_tree node, slp_instance instance, bool place_only)
>  {
> -  gimple_stmt_iterator si;
>    int i;
>    slp_tree child;
> 
> @@ -12063,6 +12070,9 @@ vect_schedule_slp_node (vec_info *vinfo,
>    if (SLP_TREE_DEF_TYPE (node) == vect_constant_def
>        || SLP_TREE_DEF_TYPE (node) == vect_external_def)
>      {
> +      if (place_only)
> +	return;
> +
>        /* ???  vectorizable_shift can end up using a scalar operand which is
>  	 currently denoted as !SLP_TREE_VECTYPE.  No need to vectorize the
>  	 node in this case.  */
> @@ -12082,9 +12092,15 @@ vect_schedule_slp_node (vec_info *vinfo,
>    stmt_vec_info stmt_info = SLP_TREE_REPRESENTATIVE (node);
> 
>    gcc_assert (SLP_TREE_VEC_DEFS (node).is_empty ());
> -  if (SLP_TREE_VECTYPE (node))
> +  if (!place_only && SLP_TREE_VECTYPE (node))
>      SLP_TREE_VEC_DEFS (node).create (vect_get_num_copies (vinfo, node));
> 
> +  gimple *last_stmt;
> +  gimple_stmt_iterator si;
> +  /* ???  When !place_only we'd like to re-use place_only computed info,
> +     but this is a bit awkward due to using gsi_insert_before and the
> +     requirement to insert after vector defs.  So we compute last_stmt
> +     during pre-scheduling and si during scheduling.  */
>    if (!SLP_TREE_PERMUTE_P (node) && STMT_VINFO_DATA_REF (stmt_info))
>      {
>        /* Vectorized loads go before the first scalar load to make it
> @@ -12095,7 +12111,8 @@ vect_schedule_slp_node (vec_info *vinfo,
>  	last_stmt_info = vect_find_first_scalar_stmt_in_slp (node);
>        else /* DR_IS_WRITE */
>  	last_stmt_info = vect_find_last_scalar_stmt_in_slp (node);
> -      si = gsi_for_stmt (last_stmt_info->stmt);
> +      last_stmt = last_stmt_info->stmt;
> +      si = gsi_for_stmt (last_stmt);
>      }
>    else if (!SLP_TREE_PERMUTE_P (node)
>  	   && (SLP_TREE_TYPE (node) == cycle_phi_info_type
> @@ -12103,15 +12120,59 @@ vect_schedule_slp_node (vec_info *vinfo,
>  	       || SLP_TREE_TYPE (node) == phi_info_type))
>      {
>        /* For PHI node vectorization we do not use the insertion iterator.  */
> +      last_stmt = SLP_TREE_SCALAR_STMTS (node)[0]->stmt;
>        si = gsi_none ();
>      }
>    else
>      {
>        /* Emit other stmts after the children vectorized defs which is
>  	 earliest possible.  */
> -      gimple *last_stmt = NULL;
> +      last_stmt = NULL;
>        FOR_EACH_VEC_ELT (SLP_TREE_CHILDREN (node), i, child)
> -	if (SLP_TREE_DEF_TYPE (child) == vect_internal_def)
> +	if (place_only)
> +	  {
> +	    gimple *vstmt = child->si;
> +	    if (!vstmt)
> +	      {
> +		if (SLP_TREE_DEF_TYPE (child) == vect_external_def)
> +		  {
> +		    vec<tree> &defs
> +		      = (!SLP_TREE_SCALAR_OPS (child).is_empty ()
> +			 ? SLP_TREE_SCALAR_OPS (child)
> +			 : SLP_TREE_VEC_DEFS (child));
> +		    for (tree def : defs)
> +		      /* If the stmt is not inside the region do not
> +			 use it as possible insertion point.  */
> +		      if (auto stmt_info = vinfo->lookup_def (def))
> +			{
> +			  gimple *stmt = stmt_info->stmt;
> +			  if (!last_stmt)
> +			    last_stmt = stmt;
> +			  else if (vect_stmt_dominates_stmt_p (last_stmt,
> stmt))
> +			    last_stmt = stmt;
> +			  else if (vect_stmt_dominates_stmt_p (stmt,
> last_stmt))
> +			    ;
> +			  else
> +			    gcc_unreachable ();
> +			}
> +		  }
> +		else
> +		  /* vect_constant_def and defs at region boundary do not
> +		     constrain placement.  */
> +		  gcc_assert (SLP_TREE_DEF_TYPE (child) == vect_constant_def
> +			      /* ???  Region boundary is not representated.  */
> +			      || true);
> +	      }
> +	    else if (!last_stmt)
> +	      last_stmt = vstmt;
> +	    else if (vect_stmt_dominates_stmt_p (last_stmt, vstmt))
> +	      last_stmt = vstmt;
> +	    else if (vect_stmt_dominates_stmt_p (vstmt, last_stmt))
> +	      ;
> +	    else
> +	      gcc_unreachable ();
> +	  }
> +	else if (SLP_TREE_DEF_TYPE (child) == vect_internal_def)
>  	  {
>  	    /* For fold-left reductions we are retaining the scalar
>  	       reduction PHI but we still have SLP_TREE_NUM_VEC_STMTS
> @@ -12225,11 +12286,16 @@ vect_schedule_slp_node (vec_info *vinfo,
>  					 gimple_bb (stmt_info->stmt),
>  					 gimple_bb (last_stmt)));
>  	  si = gsi_after_labels (gimple_bb (stmt_info->stmt));
> +	  last_stmt = gsi_stmt (si);
>  	}
>        /* When there is no in-region child def to guide placement, insert
>  	 at region boundary.  */
>        else if (!last_stmt)
> -	si = gsi_after_labels (vinfo->bbs[0]);
> +	{
> +	  si = gsi_after_labels (vinfo->bbs[0]);
> +	  /* ???  last_stmt can be NULL if the block is empty.  */
> +	  last_stmt = gsi_stmt (si);
> +	}
>        else if (is_a <gphi *> (last_stmt))
>  	si = gsi_after_labels (gimple_bb (last_stmt));
>        else
> @@ -12241,6 +12307,8 @@ vect_schedule_slp_node (vec_info *vinfo,
> 
>  	  if (auto loop_vinfo = dyn_cast <loop_vec_info> (vinfo))
>  	    {
> +	      /* We'll have to fix this up for loop vect.  */
> +	      gcc_assert (!place_only);
>  	      /* Avoid scheduling stmts to random places in the CFG, any
>  		 stmt dominance check we performed is possibly wrong as
> UIDs
>  		 are not initialized for all of the function for loop
> @@ -12274,6 +12342,44 @@ vect_schedule_slp_node (vec_info *vinfo,
>  	}
>      }
> 
> +  if (place_only)
> +    {
> +      if (dump_enabled_p () && last_stmt)
> +	dump_printf_loc (MSG_NOTE, vect_location,
> +			 "placing node %p at %G:", (void *)node, last_stmt);
> +      gcc_assert ((last_stmt && gimple_bb (last_stmt)) || gsi_bb (si));
> +      node->si = last_stmt;
> +      return;
> +    }
> +
> +  /* ???  Asserting vect_stmt_dominates_stmt_p (gsi_stmt (si), node->si)
> +     does not work because in some cases we advance si from last_stmt (as
> +     we want to insert after vector stmts) and because vector stmts of
> +     children have been inserted possibly at the same location constraint,
> +     moving si even further.  */
> +  if (flag_checking && node->si && gimple_bb (node->si) && !gsi_end_p (si))
> +    {
> +      auto gsi2 = si;
> +      while (1)
> +	{
> +	  if (vect_stmt_dominates_stmt_p (gsi_stmt (gsi2), node->si))
> +	    break;
> +	  /* As we have possibly advanced si it might now point to the
> +	     scalar stmt immediately following node->si.  That's OK.  */
> +	  if (gsi_stmt (gsi2) != gsi_stmt (si)
> +	      && gimple_uid (gsi_stmt (gsi2)) != 0)
> +	    gcc_unreachable ();
> +	  gsi_prev (&gsi2);
> +	  if (gsi_end_p (gsi2))
> +	    {
> +	      if (is_a <gphi *> (node->si)
> +		  && gimple_bb (node->si) == gsi_bb (si))
> +		break;
> +	      gcc_unreachable ();
> +	    }
> +	}
> +    }
> +
>    if (dump_enabled_p ())
>      {
>        if (stmt_info)
> @@ -12499,7 +12605,7 @@ struct slp_scc_info
>  static void
>  vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance,
>  		   hash_map<slp_tree, slp_scc_info> &scc_info,
> -		   int &maxdfs, vec<slp_tree> &stack)
> +		   int &maxdfs, vec<slp_tree> &stack, bool place_only)
>  {
>    bool existed_p;
>    slp_scc_info *info = &scc_info.get_or_insert (node, &existed_p);
> @@ -12512,7 +12618,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree
> node, slp_instance instance,
>    if (SLP_TREE_DEF_TYPE (node) != vect_internal_def)
>      {
>        info->on_stack = false;
> -      vect_schedule_slp_node (vinfo, node, instance);
> +      vect_schedule_slp_node (vinfo, node, instance, place_only);
>        return;
>      }
> 
> @@ -12529,7 +12635,8 @@ vect_schedule_scc (vec_info *vinfo, slp_tree
> node, slp_instance instance,
>        slp_scc_info *child_info = scc_info.get (child);
>        if (!child_info)
>  	{
> -	  vect_schedule_scc (vinfo, child, instance, scc_info, maxdfs, stack);
> +	  vect_schedule_scc (vinfo, child, instance, scc_info, maxdfs, stack,
> +			     place_only);
>  	  /* Recursion might have re-allocated the node.  */
>  	  info = scc_info.get (node);
>  	  child_info = scc_info.get (child);
> @@ -12548,7 +12655,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree
> node, slp_instance instance,
>      {
>        stack.pop ();
>        info->on_stack = false;
> -      vect_schedule_slp_node (vinfo, node, instance);
> +      vect_schedule_slp_node (vinfo, node, instance, place_only);
>        if (!SLP_TREE_PERMUTE_P (node)
>  	  && is_a <gphi *> (SLP_TREE_REPRESENTATIVE (node)->stmt))
>  	phis_to_fixup.quick_push (node);
> @@ -12600,7 +12707,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree
> node, slp_instance instance,
>  		    }
>  	      if (ready)
>  		{
> -		  vect_schedule_slp_node (vinfo, entry, instance);
> +		  vect_schedule_slp_node (vinfo, entry, instance, place_only);
>  		  scc_info.get (entry)->on_stack = false;
>  		  stack[idx] = NULL;
>  		  todo--;
> @@ -12615,6 +12722,9 @@ vect_schedule_scc (vec_info *vinfo, slp_tree
> node, slp_instance instance,
>        stack.truncate (last_idx);
>      }
> 
> +  if (place_only)
> +    return;
> +
>    /* Now fixup the backedge def of the vectorized PHIs in this SCC.  */
>    slp_tree phi_node;
>    FOR_EACH_VEC_ELT (phis_to_fixup, i, phi_node)
> @@ -12665,10 +12775,12 @@ vect_schedule_scc (vec_info *vinfo, slp_tree
> node, slp_instance instance,
>      }
>  }
> 
> -/* Generate vector code for SLP_INSTANCES in the loop/basic block.  */
> +/* Generate vector code for SLP_INSTANCES in the loop/basic block.  Perform
> +   vector stmt placement only when PLACE_ONLY is true.  */
> 
>  void
> -vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances)
> +vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances,
> +		   bool place_only)
>  {
>    slp_instance instance;
>    unsigned int i;
> @@ -12678,7 +12790,7 @@ vect_schedule_slp (vec_info *vinfo, const
> vec<slp_instance> &slp_instances)
>    FOR_EACH_VEC_ELT (slp_instances, i, instance)
>      {
>        slp_tree node = SLP_INSTANCE_TREE (instance);
> -      if (dump_enabled_p ())
> +      if (!place_only && dump_enabled_p ())
>  	{
>  	  dump_printf_loc (MSG_NOTE, vect_location,
>  			   "Vectorizing SLP tree:\n");
> @@ -12693,16 +12805,20 @@ vect_schedule_slp (vec_info *vinfo, const
> vec<slp_instance> &slp_instances)
>  	 have a PHI be the node breaking the cycle.  */
>        auto_vec<slp_tree> stack;
>        if (!scc_info.get (node))
> -	vect_schedule_scc (vinfo, node, instance, scc_info, maxdfs, stack);
> +	vect_schedule_scc (vinfo, node, instance, scc_info, maxdfs, stack,
> +			   place_only);
> 
> -      if (!SLP_INSTANCE_ROOT_STMTS (instance).is_empty ())
> +      if (!place_only && !SLP_INSTANCE_ROOT_STMTS (instance).is_empty ())
>  	vectorize_slp_instance_root_stmt (vinfo, node, instance);
> 
> -      if (dump_enabled_p ())
> +      if (!place_only && dump_enabled_p ())
>  	dump_printf_loc (MSG_NOTE, vect_location,
>                           "vectorizing stmts using SLP.\n");
>      }
> 
> +  if (place_only)
> +    return;
> +
>    FOR_EACH_VEC_ELT (slp_instances, i, instance)
>      {
>        slp_tree root = SLP_INSTANCE_TREE (instance);
> diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
> index 94ef81383f4..e79b9087b21 100644
> --- a/gcc/tree-vectorizer.h
> +++ b/gcc/tree-vectorizer.h
> @@ -358,6 +358,8 @@ struct _slp_tree {
>    tree vectype;
>    /* Vectorized defs.  */
>    vec<tree> vec_defs;
> +  /* Insertion place.  */
> +  gimple *si;
> 
>    /* Reference count in the SLP graph.  */
>    unsigned int refcnt;
> @@ -2775,7 +2777,7 @@ extern bool vect_transform_slp_perm_load
> (vec_info *, slp_tree, const vec<tree>
>  extern bool vectorizable_slp_permutation (vec_info *, gimple_stmt_iterator *,
>  					  slp_tree, stmt_vector_for_cost *);
>  extern bool vect_slp_analyze_operations (vec_info *);
> -extern void vect_schedule_slp (vec_info *, const vec<slp_instance> &);
> +extern void vect_schedule_slp (vec_info *, const vec<slp_instance> &, bool);
>  extern opt_result vect_analyze_slp (vec_info *, unsigned, bool);
>  extern bool vect_make_slp_decision (loop_vec_info);
>  extern bool vect_detect_hybrid_slp (loop_vec_info);
> --
> 2.51.0