Re: [PATCH v14 09/19] vect: Record/cost/set BB SLP partial vector controls for WHILE_ULT
Pengfei Li <[email protected]>
| Newsgroups | gmane.comp.gcc.patches |
|---|---|
| Message-ID | <[email protected]> |
On 30/07/2026 15:56, Christopher Bazley wrote:
> This is part of enablement of support for partial vectors in basic
> block SLP vectorization.
>
> Adapt existing functions used by the loop vectorizer to record partial
> vector masks during the analysis phase, verify that the recorded
> requirements can be satisfied, and get SSA names that refer to those
> masks for use during transformation.
>
> A different scheme is used for recording masks for BB SLP vectorization
> but it reuses as much loop vectorizer code as possible: instead of
> recording the number of vectors to be masked in vect_record_mask and
> later using that as the index to find the relevant rgroup controls in
> vect_get_mask, rgroup controls are instead indexed by the number of
> lanes in an SLP tree node, scaled by the ratio of the maximum number
> of subparts in any vector type to the number of subparts in the
> vector type to be masked. This allows a mask of 12 lanes for VNx16QI
> to be reinterpreted as a mask of 3 lanes for VNx4SI, for example.
> The main purpose of this reinterpretation is to avoid having to
> introduce an extra dimension to storage of rgroup controls. Sharing of
> mask constants typically has no effect on compiler output because the
> 'fre' pass eliminates the resultant VIEW_CONVERT_EXPR expressions.
>
> Since the number of vectors is not recorded for BB SLP, it must always
> be one. Consequently, the number of controls per rgroup_controls
> instance is also one for BB SLP. (One mask for each unique number of
> lanes.)
>
> As part of analyzing whether an SLP region can be vectorized, after
> analyzing all statements in the region, call vect_verify_full_masking
> to check whether we can generate the necessary partial vector masks.
> If not, return false from vect_slp_analyze_bb_1.
>
> If vect_slp_analyze_bb_1 returns true (because vectorization of the
> SLP region is possible) then vect_bb_vectorization_profitable_p is
> called to estimate whether it would be profitable. New code in
> that function estimates partial vector setup costs for an SLP region
> based on its recorded requirements. This is imprecise because
> masks can be shared between SLP nodes and we don't yet know which
> subgraphs will be scheduled.
>
> After scheduling any profitable subgraphs and generating any
> invariant statements, a new function, vect_set_bb_slp_controls, is
> called to insert any new stmts required to set up partial vector
> masks on entry to the SLP vectorization region.
>
> For now, only the WHILE_ULT style of partial vectors is supported
> but there is scope to extend that to support other styles in future
> commits.
>
> A drawback of using inheritance from class vec_info to record partial
> vector requirements for both BB SLP and loop vectorisation is that
> stmts might be inserted to set up masks that are used by fewer SLP
> subgraphs than expected, or even by none. Dead stmts are expected
> to be eliminated by later passes.
>
> gcc/ChangeLog:
>
> * tree-vect-loop.cc (can_produce_all_loop_masks_p): Renamed as
> can_produce_all_masks_p.
> (can_produce_all_masks_p): Change parameter of type
> loop_vec_info to vec_info *. Access the recorded masks
> directly instead of via LOOP_VINFO_MASKS.
> (vect_get_max_nscalars_per_iter): As above.
> (vect_min_prec_for_max_group_size): New helper to find the
> maximum group size in an SLP vectorization region by a linear
> search through the vector of rgroup_controls.
> (vect_verify_full_masking): Change parameter of type
> loop_vec_info to vec_info *. Access the recorded masks directly
> instead of via LOOP_VINFO_MASKS. When this function is used
> for loop vectorization, the index into the vector of rgroup
> controls is derived from the number of vectors, which is the
> second value of each pair in mask_set; when used for BB SLP
> vectorization, the rgroup index is instead derived
> from the SLP group size, which is still the second value,
> scaled by max_nunits / nunits.
> For BB SLP, use the SLP group size as nscalars_per_iter, such
> that the truth type of an rgroup is only replaced by the truth
> type of a vectype for which a wider mask is needed. (If
> previous and new mask requirements have the same rgroup index
> and group size then nunits must also be equal.)
> Call vect_min_prec_for_max_group_size to get an alternative
> value of min_ni_width for BB SLP. Avoid calling
> vect_iv_limit_for_partial_vectors for BB SLP because the concept
> of an induction variable is not applicable. Instead, set
> iv_precision to the calculated min_ni_width.
> Set rgroup_compare_type, rgroup_iv_type and
> partial_vector_style directly instead of via LOOP_VINFO_
> macros.
> (vect_analyze_loop_2): Reset max_nunits to zero at the same time
> as emptying the mask_set.
> (vect_record_mask): If used for BB SLP, get the group size
> from the SLP tree node and use that as the second value of the
> pair added to mask_set (instead of the nvectors argument).
> Assert that nvectors is 1, if used for BB SLP. Add code
> to increase the max_nunits of vec_masks if the passed-in
> vectype is known to have a greater number of subparts.
> (vect_get_mask): If used for BB SLP, get the group size
> from the SLP tree node, scale that by max_nunits / nunits,
> and use the product as the index of the rgroup control to
> be queried (instead of using the nvectors argument). Don't
> call make_temp_ssa_name with "loop_mask" as the name, if
> used for BB SLP. Assert that nvectors is 1 and only the first
> vector is queried, if used for BB SLP.
> * tree-vect-slp.cc (vect_bb_vectorization_profitable_p):
> Calculate the number of rgroup controls that need to be
> produced for the whole SLP region. Divide that by the
> number of SLP instances to get the average number of
> rgroup controls per instance. Call record_stmt_cost to
> assign a fair proportion of the vector_stmt costs in
> vect_prologue to each instance. Spread remainder of the
> division across the SLP instances until none is left.
> (vect_slp_analyze_bb_1): If masks have been recorded for
> an SLP region then call vect_verify_full_masking to
> check whether we can generate the necessary controls.
> If not, return false.
> (vect_set_bb_slp_controls_directly): New helper.
> Assert that there is only one rgroup control (because
> vect_get_mask creates only one control per recorded mask
> for BB SLP, instead of one per vector for loop vectorization).
> Call build_int_cst to build constant bounds for a WHILE_ULT
> statement. Call vect_gen_while to generate and return a mask
> of the truth type and number of lanes chosen by
> vect_verify_full_masking. Call gimple_build_assign to build
> an assignment of the vector mask to the temporary SSA name
> made by vect_get_mask. Call gimple_seq_add_stmt to add the
> assignment to the gimple_seq passed by the caller.
> (vect_set_bb_slp_controls_partial_vectors): Assert that the
> partial vectors style is WHILE_ULT and that some masks were
> recorded. Iterate over all the rgroups and call
> vect_set_bb_slp_controls_directly to set up each control,
> passing the address of a gimple_seq to populate. Lastly,
> use the insert_seq_on_entry method of the bb_vec_info to
> insert the stmts contained in that gimple_seq at entry to
> the SLP vectorization region.
> (vect_set_bb_slp_controls): Assert that the partial vectors
> style is WHILE_ULT and call
> vect_set_bb_slp_controls_partial_vectors.
> (vect_slp_region): If a partial vectors style has been set
> by vect_verify_full_masking, call vect_set_bb_slp_controls.
> * tree-vectorizer.h (rgroup_controls): Update the description
> of max_nscalars_per_iter.
> (vec_masks): Add max_nunits as a member of vec_masks, with a
> default initial value of 0. This is used to track the
> maximum number of subparts of all the vector types recorded
> in mask_set.
> (vect_verify_full_masking): Declare function for use by
> vect_slp_analyze_bb_1. This function no longer has static
> linkage.
> ---
> gcc/tree-vect-loop.cc | 190 ++++++++++++++++++++++++++++++++----------
> gcc/tree-vect-slp.cc | 110 ++++++++++++++++++++++++
> gcc/tree-vectorizer.h | 11 ++-
> 3 files changed, 267 insertions(+), 44 deletions(-)
>
> diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
> index 450d6a23ec9..fe3ddb2fdc8 100644
> --- a/gcc/tree-vect-loop.cc
> +++ b/gcc/tree-vect-loop.cc
> @@ -891,14 +891,14 @@ cse_and_gimplify_to_preheader (loop_vec_info loop_vinfo, tree expr)
> }
>
> /* Return true if we can use CMP_TYPE as the comparison type to produce
> - all masks required to mask LOOP_VINFO. */
> + all masks required to mask VINFO. */
>
> static bool
> -can_produce_all_loop_masks_p (loop_vec_info loop_vinfo, tree cmp_type)
> +can_produce_all_masks_p (vec_info *vinfo, tree cmp_type)
> {
> rgroup_controls *rgm;
> unsigned int i;
> - FOR_EACH_VEC_ELT (LOOP_VINFO_MASKS (loop_vinfo).rgc_vec, i, rgm)
> + FOR_EACH_VEC_ELT (vinfo->masks.rgc_vec, i, rgm)
> if (rgm->type != NULL_TREE
> && !direct_internal_fn_supported_p (IFN_WHILE_ULT,
> cmp_type, rgm->type,
> @@ -908,15 +908,15 @@ can_produce_all_loop_masks_p (loop_vec_info loop_vinfo, tree cmp_type)
> }
>
> /* Calculate the maximum number of scalars per iteration for every
> - rgroup in LOOP_VINFO. */
> + rgroup in VINFO. */
>
> static unsigned int
> -vect_get_max_nscalars_per_iter (loop_vec_info loop_vinfo)
> +vect_get_max_nscalars_per_iter (vec_info *vinfo)
> {
> unsigned int res = 1;
> unsigned int i;
> rgroup_controls *rgm;
> - FOR_EACH_VEC_ELT (LOOP_VINFO_MASKS (loop_vinfo).rgc_vec, i, rgm)
> + FOR_EACH_VEC_ELT (vinfo->masks.rgc_vec, i, rgm)
> res = MAX (res, rgm->max_nscalars_per_iter);
> return res;
> }
> @@ -989,37 +989,91 @@ vect_need_peeling_or_partial_vectors_p (loop_vec_info loop_vinfo)
> return true;
> }
>
> -/* Each statement in LOOP_VINFO can be masked where necessary. Check
> - whether we can actually generate the masks required. Return true if so,
> - storing the type of the scalar IV in LOOP_VINFO_RGROUP_COMPARE_TYPE. */
> +/* Given some rgroup CONTROLS used for basic block SLP vectorization,
> + return the maximum SLP group size from the recorded partial vector
> + requirements. */
> +static unsigned int
> +vect_min_prec_for_max_group_size (vec<rgroup_controls> *controls)
> +{
> + unsigned int i, max_group_size = 1;
> + rgroup_controls *rgc;
>
> -static bool
> -vect_verify_full_masking (loop_vec_info loop_vinfo)
> + FOR_EACH_VEC_ELT (*controls, i, rgc)
> + {
> + if (!rgc->type)
> + continue;
> + gcc_assert (rgc->factor == 1);
> + unsigned int group_size = i + 1;
> + max_group_size = MAX (max_group_size, group_size);
> + }
Since i increases monotonically during this FOR_EACH_VEC_ELT, isn't each
valid group_size necessarily larger than the previous one? Could this
just be max_group_size = i + 1 instead of using MAX?
IIUC, the group_size here is normalized. So perhaps
max_normalized_group_size could be a better name?
Thanks,
Pengfei
> +
> + return wi::min_precision (max_group_size, UNSIGNED);
> +}
> +
> +/* Each statement in a loop or basic block SLP vectorization region represented
> + by VINFO can be masked where necessary. Check whether we can actually
> + generate the masks required. Return true if so, storing the type of the
> + scalar IV and the type that IV should be converted to for comparison purposes
> + in VINFO. For BB SLP vectorization, both types are the same and are used to
> + build constant bounds for WHILE_ULT stmts. */
> +bool
> +vect_verify_full_masking (vec_info *vinfo)
> {
> + loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
> + bb_vec_info bb_vinfo = dyn_cast<bb_vec_info> (vinfo);
> unsigned int min_ni_width;
>
> /* Use a normal loop if there are no statements that need masking.
> This only happens in rare degenerate cases: it means that the loop
> has no loads, no stores, and no live-out values. */
> - if (LOOP_VINFO_MASKS (loop_vinfo).is_empty ())
> + if (vinfo->masks.is_empty ())
> return false;
>
> /* Produce the rgroup controls. */
> - for (auto mask : LOOP_VINFO_MASKS (loop_vinfo).mask_set)
> + for (auto mask : vinfo->masks.mask_set)
> {
> - vec_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
> + vec_masks *masks = &vinfo->masks;
> tree vectype = mask.first;
> - unsigned nvectors = mask.second;
> + unsigned int nvectors_or_group_size = mask.second;
> + poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
>
> - if (masks->rgc_vec.length () < nvectors)
> - masks->rgc_vec.safe_grow_cleared (nvectors, true);
> - rgroup_controls *rgm = &(*masks).rgc_vec[nvectors - 1];
> /* The number of scalars per iteration and the number of vectors are
> both compile-time constants. */
> - unsigned int nscalars_per_iter
> - = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
> - LOOP_VINFO_VECT_FACTOR (loop_vinfo)).to_constant ();
> + unsigned int nscalars_per_iter, key;
> + if (loop_vinfo)
> + {
> + unsigned int nvectors = nvectors_or_group_size;
> +
> + /* With VNx16QI, the rgroup operates on nV * 16 / VF values from
> + each scalar iteration; with VNx4SI, it instead operates on
> + nV * 4 / VF values. Calculate a value of nS suitable for both
> + cases. */
> + nscalars_per_iter
> + = exact_div (nvectors * nunits, vinfo->vectorization_factor)
> + .to_constant ();
> + key = nvectors;
> + }
> + else
> + {
> + unsigned int group_size = nvectors_or_group_size;
> +
> + /* With VNx16QI, the rgroup operates on GS1 * max(4,16) / 16 values;
> + with VNx4SI, it instead operates on GS2 * max(4,16) / 4 values.
> + Use the same mask for both cases (e.g., key is 12 for VNx16QI with
> + group_size 12, but also for VNx4SI with group_size 3). */
> + key = exact_div (group_size * vinfo->masks.max_nunits, nunits)
> + .to_constant ();
> + nscalars_per_iter = group_size;
> + }
> +
> + if (masks->rgc_vec.length () < key)
> + masks->rgc_vec.safe_grow_cleared (key, true);
> + rgroup_controls *rgm = &(*masks).rgc_vec[key - 1];
>
> + /* A mask produced for nS values with a vector type that has nL lanes can
> + be reinterpreted as a mask for nS/2 values with a vector type that has
> + nL/2 lanes, so plan to build a mask of a type suitable for the vector
> + type that is used to operate on the highest nS. */
> if (rgm->max_nscalars_per_iter < nscalars_per_iter)
> {
> rgm->max_nscalars_per_iter = nscalars_per_iter;
> @@ -1028,23 +1082,31 @@ vect_verify_full_masking (loop_vec_info loop_vinfo)
> }
> }
>
> - unsigned int max_nscalars_per_iter
> - = vect_get_max_nscalars_per_iter (loop_vinfo);
> + unsigned int max_nscalars_per_iter = vect_get_max_nscalars_per_iter (vinfo);
>
> /* Work out how many bits we need to represent the limit. */
> - min_ni_width
> - = vect_min_prec_for_max_niters (loop_vinfo, max_nscalars_per_iter);
> + if (loop_vinfo)
> + min_ni_width
> + = vect_min_prec_for_max_niters (loop_vinfo, max_nscalars_per_iter);
> + else
> + min_ni_width = vect_min_prec_for_max_group_size (&bb_vinfo->masks.rgc_vec);
>
> /* Find a scalar mode for which WHILE_ULT is supported. */
> opt_scalar_int_mode cmp_mode_iter;
> tree cmp_type = NULL_TREE;
> tree iv_type = NULL_TREE;
> - widest_int iv_limit = vect_iv_limit_for_partial_vectors (loop_vinfo);
> - unsigned int iv_precision = UINT_MAX;
> + unsigned int iv_precision;
> + if (loop_vinfo)
> + {
> + widest_int iv_limit = vect_iv_limit_for_partial_vectors (loop_vinfo);
> + iv_precision = UINT_MAX;
>
> - if (iv_limit != -1)
> - iv_precision = wi::min_precision (iv_limit * max_nscalars_per_iter,
> - UNSIGNED);
> + if (iv_limit != -1)
> + iv_precision
> + = wi::min_precision (iv_limit * max_nscalars_per_iter, UNSIGNED);
> + }
> + else
> + iv_precision = min_ni_width;
>
> FOR_EACH_MODE_IN_CLASS (cmp_mode_iter, MODE_INT)
> {
> @@ -1054,7 +1116,7 @@ vect_verify_full_masking (loop_vec_info loop_vinfo)
> {
> tree this_type = build_nonstandard_integer_type (cmp_bits, true);
> if (this_type
> - && can_produce_all_loop_masks_p (loop_vinfo, this_type))
> + && can_produce_all_masks_p (vinfo, this_type))
> {
> /* Although we could stop as soon as we find a valid mode,
> there are at least two reasons why that's not always the
> @@ -1091,13 +1153,13 @@ vect_verify_full_masking (loop_vec_info loop_vinfo)
>
> if (!cmp_type)
> {
> - LOOP_VINFO_MASKS (loop_vinfo).rgc_vec.release ();
> + vinfo->masks.rgc_vec.release ();
> return false;
> }
>
> - LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo) = cmp_type;
> - LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo) = iv_type;
> - LOOP_VINFO_PARTIAL_VECTORS_STYLE (loop_vinfo) = vect_partial_vectors_while_ult;
> + vinfo->rgroup_compare_type = cmp_type;
> + vinfo->rgroup_iv_type = iv_type;
> + vinfo->partial_vector_style = vect_partial_vectors_while_ult;
> return true;
> }
>
> @@ -2723,6 +2785,7 @@ again:
> loop_vinfo->vector_costs = nullptr;
> /* Reset accumulated rgroup information. */
> LOOP_VINFO_MASKS (loop_vinfo).mask_set.empty ();
> + LOOP_VINFO_MASKS (loop_vinfo).max_nunits = 0;
> release_vec_loop_controls (&LOOP_VINFO_MASKS (loop_vinfo).rgc_vec);
> release_vec_loop_controls (&LOOP_VINFO_LENS (loop_vinfo));
> /* Reset assorted flags. */
> @@ -10593,14 +10656,31 @@ vect_record_mask (vec_info *vinfo, unsigned int nvectors,
> gcc_assert (nvectors != 0);
> vec_masks *masks = &vinfo->masks;
> loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
> + unsigned int nvectors_or_group_size;
>
> - if (loop_vinfo && scalar_mask)
> + if (loop_vinfo)
> {
> - scalar_cond_masked_key cond (scalar_mask, nvectors);
> - loop_vinfo->scalar_cond_masked_set.add (cond);
> + nvectors_or_group_size = nvectors;
> + if (scalar_mask)
> + {
> + scalar_cond_masked_key cond (scalar_mask, nvectors);
> + loop_vinfo->scalar_cond_masked_set.add (cond);
> + }
> + }
> + else
> + {
> + gcc_assert (nvectors == 1);
> + nvectors_or_group_size = SLP_TREE_LANES (slp_node);
> +
> + /* Keep track of the highest number of subparts, so we can scale the SLP
> + group size by that value to determine when mask requirements for
> + different group sizes can be satisfied by a common mask. */
> + poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
> + if (known_gt (nunits, masks->max_nunits))
> + masks->max_nunits = nunits;
> }
>
> - masks->mask_set.add (std::make_pair (vectype, nvectors));
> + masks->mask_set.add (std::make_pair (vectype, nvectors_or_group_size));
> }
>
> /* Given a complete set of masks for the loop or SLP region represented by
> @@ -10618,11 +10698,31 @@ vect_get_mask (vec_info *vinfo, gimple_stmt_iterator *gsi,
> slp_tree slp_node)
> {
> gcc_assert (!is_a<bb_vec_info> (vinfo) || slp_node != nullptr);
> - vec_masks *masks = &LOOP_VINFO_MASKS (vinfo);
> + gcc_assert (nvectors != 0);
> + gcc_assert (index < nvectors);
> +
> + vec_masks *masks = &vinfo->masks;
> + loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
>
> if (vinfo->partial_vector_style == vect_partial_vectors_while_ult)
> {
> - rgroup_controls *rgm = &(masks->rgc_vec)[nvectors - 1];
> + unsigned int key;
> +
> + if (loop_vinfo)
> + key = nvectors;
> + else
> + {
> + gcc_assert (nvectors == 1);
> + unsigned int group_size = SLP_TREE_LANES (slp_node);
> + poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
> + unsigned int nscalars_per_iter
> + = exact_div (group_size * vinfo->masks.max_nunits, nunits)
> + .to_constant ();
> + key = nscalars_per_iter;
> + }
> +
> + gcc_assert (key != 0);
> + rgroup_controls *rgm = &(masks->rgc_vec)[key - 1];
> tree mask_type = rgm->type;
>
> /* Populate the rgroup's mask array, if this is the first time we've
> @@ -10632,7 +10732,9 @@ vect_get_mask (vec_info *vinfo, gimple_stmt_iterator *gsi,
> rgm->controls.safe_grow_cleared (nvectors, true);
> for (unsigned int i = 0; i < nvectors; ++i)
> {
> - tree mask = make_temp_ssa_name (mask_type, NULL, "loop_mask");
> + tree mask
> + = make_temp_ssa_name (mask_type, NULL,
> + loop_vinfo ? "loop_mask" : "slp_mask");
> /* Provide a dummy definition until the real one is available. */
> SSA_NAME_DEF_STMT (mask) = gimple_build_nop ();
> rgm->controls[i] = mask;
> @@ -10683,7 +10785,9 @@ vect_get_mask (vec_info *vinfo, gimple_stmt_iterator *gsi,
> rgm->controls.safe_grow_cleared (nvectors, true);
> for (unsigned int i = 0; i < nvectors; ++i)
> {
> - tree mask = make_temp_ssa_name (rgm->type, NULL, "loop_mask");
> + tree mask
> + = make_temp_ssa_name (rgm->type, NULL,
> + loop_vinfo ? "loop_mask" : "slp_mask");
> /* Provide a dummy definition until the real one is available. */
> SSA_NAME_DEF_STMT (mask) = gimple_build_nop ();
> rgm->controls[i] = mask;
> diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
> index b9b2224dbe4..2998f527213 100644
> --- a/gcc/tree-vect-slp.cc
> +++ b/gcc/tree-vect-slp.cc
> @@ -9659,6 +9659,29 @@ vect_bb_vectorization_profitable_p (bb_vec_info bb_vinfo,
> SLP_INSTANCE_TREE (instance), visited);
> }
>
> + /* Costing each partial vector would be excessive for many SLP instances,
> + because it is common to materialise identical masks/lengths for related
> + operations. This alternative allows costs to be shared between SLP
> + subgraphs, but does not take into account any further savings that might be
> + discovered during lowering. */
> + int control_count = 0;
> + enum vect_cost_for_stmt control_kind = vector_stmt;
> +
> + if (!bb_vinfo->masks.is_empty ())
> + {
> + gcc_assert (bb_vinfo->partial_vector_style
> + == vect_partial_vectors_while_ult);
> + for (auto rgc : bb_vinfo->masks.rgc_vec)
> + if (rgc.type)
> + control_count++;
> + }
> +
> + gcc_assert (bb_vinfo->lens.is_empty ());
> +
> + const int slp_instance_count = slp_instances.length (),
> + control_cost_per_inst_q = control_count / slp_instance_count;
> + int control_cost_per_inst_r = control_count % slp_instance_count;
> +
> /* Then DFS walk scalar stmts, performing costing and handling
> still live scalar stmts via the previously computed vector coverage. */
> stmt_vector_for_cost scalar_costs = vNULL;
> @@ -9680,6 +9703,20 @@ vect_bb_vectorization_profitable_p (bb_vec_info bb_vinfo,
> worklist.safe_push (stmt);
> }
> vect_bb_slp_scalar_cost (bb_vinfo, worklist, &scalar_costs, svisited);
> +
> + /* We cannot ascribe shared costs of partial setup to any one instance,
> + but we can try to split them fairly between instances. */
> + int stmt_cost = control_cost_per_inst_q;
> + if (control_cost_per_inst_r)
> + {
> + control_cost_per_inst_r--;
> + stmt_cost++;
> + }
> +
> + if (stmt_cost)
> + record_stmt_cost (&instance->cost_vec, stmt_cost, control_kind, nullptr,
> + nullptr, NULL_TREE, 0, vect_prologue);
> +
> vector_costs.safe_splice (instance->cost_vec);
> instance->cost_vec.release ();
> }
> @@ -10237,6 +10274,14 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo, int n_stmts, bool &fatal,
> return false;
> }
>
> + /* Check whether we can generate the necessary controls. */
> + gcc_assert (bb_vinfo->lens.is_empty ());
> + if (!bb_vinfo->masks.is_empty ())
> + {
> + if (!vect_verify_full_masking (bb_vinfo))
> + return false;
> + }
> +
> /* Mark all the statements that we vectorize. */
> vect_bb_slp_mark_stmts_vectorized (bb_vinfo);
>
> @@ -10248,6 +10293,68 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo, int n_stmts, bool &fatal,
> return true;
> }
>
> +/* Helper for vect_set_bb_slp_controls_partial_vectors. Generate definitions
> + for all the rgroup controls in RGC. */
> +
> +static void
> +vect_set_bb_slp_controls_directly (bb_vec_info bb_vinfo, gimple_seq *seq,
> + rgroup_controls *rgc)
> +{
> + tree compare_type = bb_vinfo->rgroup_compare_type;
> +
> + tree ctrl_type = rgc->type;
> + gcc_assert (rgc->factor == 1);
> + unsigned int nitems = rgc->max_nscalars_per_iter;
> + tree zero_index = build_int_cst (compare_type, 0);
> + tree nitems_cst = build_int_cst (compare_type, nitems);
> +
> + gcc_assert (rgc->controls.length () == 1);
> + tree ctrl = rgc->controls[0];
> + tree init_ctrl;
> +
> + gcc_assert (bb_vinfo->partial_vector_style == vect_partial_vectors_while_ult);
> + init_ctrl
> + = vect_gen_while (seq, ctrl_type, zero_index, nitems_cst, "max_mask");
> +
> + gimple *stmt = gimple_build_assign (ctrl, init_ctrl);
> + gimple_seq_add_stmt (seq, stmt);
> +}
> +
> +/* Set up the rgroup controls for the region described by BB_VINFO, given that
> + the region uses partial vectors. */
> +
> +static void
> +vect_set_bb_slp_controls_partial_vectors (bb_vec_info bb_vinfo)
> +{
> + gcc_assert (bb_vinfo->partial_vector_style == vect_partial_vectors_while_ult);
> + gcc_assert (!bb_vinfo->masks.is_empty ());
> +
> + gimple_seq seq = NULL;
> +
> + /* Iterate over all the rgroups and fill in their controls. */
> + rgroup_controls *rgc;
> + unsigned int i;
> + auto_vec<rgroup_controls> *controls = &bb_vinfo->masks.rgc_vec;
> + FOR_EACH_VEC_ELT (*controls, i, rgc)
> + if (!rgc->controls.is_empty ())
> + {
> + /* Set up the control for this group. */
> + vect_set_bb_slp_controls_directly (bb_vinfo, &seq, rgc);
> + }
> +
> + bb_vinfo->insert_seq_on_entry (NULL, seq);
> +}
> +
> +/* Set up the rgroup controls for the region described by BB_VINFO, if
> + the region uses partial vectors. */
> +
> +static void
> +vect_set_bb_slp_controls (bb_vec_info bb_vinfo)
> +{
> + gcc_assert (bb_vinfo->partial_vector_style == vect_partial_vectors_while_ult);
> + vect_set_bb_slp_controls_partial_vectors (bb_vinfo);
> +}
> +
> /* Subroutine of vect_slp_bb. Try to vectorize the statements for all
> basic blocks in BBS, returning true on success.
> The region has N_STMTS statements and has the datarefs given by DATAREFS. */
> @@ -10410,6 +10517,9 @@ vect_slp_region (vec<basic_block> bbs, vec<data_reference_p> datarefs,
> bb_vinfo->insert_seq_on_entry (NULL,
> bb_vinfo->inv_pattern_def_seq);
> }
> +
> + if (bb_vinfo->partial_vector_style != vect_partial_vectors_none)
> + vect_set_bb_slp_controls (bb_vinfo);
> }
> else
> {
> diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
> index 46a54dfed64..a6e2b229bf2 100644
> --- a/gcc/tree-vectorizer.h
> +++ b/gcc/tree-vectorizer.h
> @@ -706,7 +706,10 @@ public:
> struct rgroup_controls {
> /* The largest nS for all rgroups that use these controls.
> For vect_partial_vectors_avx512 this is the constant nscalars_per_iter
> - for all members of the group. */
> + for all members of the group.
> + For BB SLP vectorization with vect_partial_vectors_while_ult, this is the
> + highest minimum number of subparts of all the vector types that use this
> + rgroup (e.g., 16 from VNx4HI, VNx4SI, VNx4HF and VNx16QI). */
> unsigned int max_nscalars_per_iter;
>
> /* For the largest nS recorded above, the loop controls divide each scalar
> @@ -743,6 +746,11 @@ struct vec_masks
>
> /* rgroup_controls used for the partial vector scheme. */
> auto_vec<rgroup_controls> rgc_vec;
> +
> + /* For BB SLP vectorization, the maximum number of subparts of all of the
> + vector types recorded in mask_set. Used to scale SLP group sizes to
> + allow a compatible mask to be located in rgc_vec. */
> + poly_uint64 max_nunits = 0;
> };
>
> typedef auto_vec<rgroup_controls> vec_lens;
> @@ -2770,6 +2778,7 @@ extern bool vect_can_vectorize_without_simd_p (tree_code);
> extern bool vect_can_vectorize_without_simd_p (code_helper);
> extern int vect_get_known_peeling_cost (loop_vec_info, int);
> extern tree cse_and_gimplify_to_preheader (loop_vec_info, tree);
> +extern bool vect_verify_full_masking (vec_info *);
>
> /* Nonlinear induction. */
> extern tree vect_peel_nonlinear_iv_init (gimple_seq*, tree, tree,