Re: [PATCH v14 01/19] vect: Preparation to support predicated tails for BB SLP
Pengfei Li <[email protected]>
| Newsgroups | gmane.comp.gcc.patches |
|---|---|
| Message-ID | <[email protected]> |
On 30/07/2026 15:56, Christopher Bazley wrote:
> Modify the interface used by the vectorizer to record and get
> masks, in preparation for extending use of that interface to
> basic block SLP vectorization (as well as keeping its existing
> use as part of loop vectorization). Add an optional parameter
> of type slp_tree. At the moment, this is unused; in
> future, it will be used to get the number of active lanes from
> an SLP node in a more type-safe way than passing an int.
>
> Rename the types that store sets of rgroup controls used in the
> partial vector scheme, to the less loop-specific names vec_masks
> and vec_lens.
>
> Move certain members of _loop_vec_info to its superclass,
> vec_info, in anticipation of those members being reused for
> BB SLP vectorization too.
>
> Notably, using_partial_vectors_p, must_use_partial_vectors_p
> and can_use_partial_vectors_p are *not* expected to be reused
> for BB SLP because it does not make sense to maintain that
> state for a whole SLP region. Decisions about use of partial
> vectors will be made immediately on a per-statement basis during
> analysis of the SLP tree instead of being deferred so that
> vectorization of a loop can be determined holistically. It
> follows that any existing code using the function-like macros
> LOOP_VINFO_USING_PARTIAL_VECTORS_P,
> LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P,
> LOOP_VINFO_MUST_USE_PARTIAL_VECTORS_P, LOOP_VINFO_FULLY_MASKED_P,
> and LOOP_VINFO_FULLY_WITH_LENGTH_P will need to be rewritten for
> BB SLP.
>
> This is intended to be a pure refactoring and to remain
> minimal (e.g., vec_info and slp_tree pointers are not passed
> into vect_get_len whenever such arguments are available).
>
> gcc/ChangeLog:
>
> * tree-vect-loop.cc (_loop_vec_info::_loop_vec_info):
> Delete initializers for vectorization_factor,
> rgroup_compare_type, partial_vector_style and
> partial_load_store_bias.
> (vect_verify_full_masking_avx512): Update comment.
> (vect_estimate_min_profitable_iters): As above.
> (vectorize_fold_left_reduction): Delete the parameters
> that point to vec_loop_masks and vec_loop_lens and
> don't pass those arguments to vect_get_mask or vect_get_len.
> (vect_reduction_update_partial_vector_usage): Don't pass a
> pointer to vec_loop_masks or vec_loop_lens when calling
> vect_record_mask or vect_record_len.
> (vect_transform_reduction): Don't pass a
> pointer to vec_loop_masks or vec_loop_lens when calling
> vect_get_mask, vect_get_len or vectorize_fold_left_reduction.
> (vectorizable_live_operation_1): Don't pass a
> pointer to vec_loop_masks or vec_loop_lens when calling
> vect_get_mask or vect_get_len.
> (vectorizable_live_operation): Don't pass a
> pointer to vec_loop_masks or vec_loop_lens when calling
> vect_record_mask or vect_record_len.
> (vect_record_loop_mask): Rename as vect_record_mask.
> (vect_record_mask): Change the type of the first parameter from
> loop_vec_info to a pointer to vec_info.
> Get vec_masks from vec_info instead of requiring an extra
> argument for vec_loop_masks (old type name).
> Only add a mask to the scalar_cond_masked_set if called with a
> loop_vec_info.
> Add a (currently unused) parameter of type slp_tree.
> (vect_get_loop_mask): Rename as vect_get_mask.
> (vect_get_mask): Change the type of the first parameter from
> loop_vec_info to a pointer to vec_info.
> Get vec_masks from vec_info instead of requiring an extra
> argument for vec_loop_masks (old type name).
> Access members of the vec_info directly instead of via
> LOOP_VINFO_ accessor macros.
> Add a (currently unused) parameter of type slp_tree.
> (vect_record_loop_len): Rename as vect_record_len.
> (vect_record_len): Change the type of the first parameter from
> loop_vec_info to a pointer to vec_info.
> Get vec_lens from vec_info instead of requiring an extra
> argument for vec_loop_lens (old name).
> Access members of the vec_info directly instead of via
> LOOP_VINFO_ accessor macros.
> Add a (currently unused) parameter of type slp_tree.
> (vect_get_loop_len): Rename as vect_get_len.
> (vect_get_len): Change the type of the first parameter from
> loop_vec_info to a pointer to vec_info.
> Get vec_lens from vec_info instead of requiring an extra
> argument for vec_loop_lens (old name).
> Access members of the vec_info directly instead of via
> LOOP_VINFO_ accessor macros.
> Add a (currently unused) parameter of type slp_tree.
> (vect_gen_loop_len_mask): Delete the parameter that points to
> vec_loop_lens (old name) and don't pass vec_lens to
> vect_get_len.
> * tree-vect-stmts.cc (check_load_store_for_partial_vectors):
> Don't pass a pointer to vec_loop_masks or vec_loop_lens when
> calling vect_record_mask or vect_record_len.
> (prepare_vec_mask): Change the type of the first parameter from
> loop_vec_info to a pointer to vec_info and use it when querying
> vec_cond_masked_set.
> Rename the loop_mask parameter as loop_or_tail_mask.
> (vectorizable_call): Don't pass a pointer to vec_loop_masks or
> vec_loop_lens when calling vect_record_mask, vect_record_len,
> vect_get_mask or vect_get_len.
> (vectorizable_simd_clone_call): As above.
> (vectorizable_operation): As above.
> Replace uses of local vec_loop_masks and vec_loop_lens pointers
> for flow control with Boolean local variables that can
> be true for BB SLP if using partial vectors.
> (vectorizable_store): Replace uses of local vec_loop_masks and
> vec_loop_lens pointers for flow control with Boolean local
> variables that can be true for BB SLP if using partial vectors.
> Stop passing a vec_loop_masks or vec_loop_lens pointer as an
> explicit argument to vect_get_loop_mask, vect_get_loop_len
> (old function names).
> Replace invocation of LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS with
> direct access to partial_load_store_bias member of vec_info.
> Pass a pointer to vec_info to vect_get_len, vect_get_mask and
> prepare_vec_mask instead of passing a loop_vec_info.
> (vectorizable_load): As above.
> (vectorizable_condition): Stop passing a vec_loop_masks or
> vec_loop_lens pointer as an explicit argument to
> vect_record_loop_len, vect_record_loop_mask, vect_get_loop_len,
> or vect_get_loop_mask (old function names).
> Replace uses of local vec_loop_masks and vec_loop_lens pointers
> for flow control with Boolean local variables that can
> be true for BB SLP if using partial vectors.
> (vectorizable_early_exit): Stop passing a vec_loop_masks or
> vec_loop_lens pointer as an explicit argument to
> vect_record_loop_len, vect_record_loop_mask,
> vect_get_loop_mask, prepare_vec_mask and
> vect_gen_loop_len_mask.
> * tree-vectorizer.cc (vec_info::vec_info):
> Add initializers for vectorization_factor,
> rgroup_compare_type, rgroup_iv_type,
> partial_vector_style, partial_load_store_bias.
> * tree-vectorizer.h (struct scalar_cond_masked_key):
> Update reference to vect_record_loop_mask.
> (class vec_info): Move members vectorization_factor,
> masks, lens, vec_cond_masked_set, rgroup_compare_type,
> rgroup_iv_type, partial_vector_style and
> partial_load_store_bias from _loop_vec_info to its
> superclass.
> (class _loop_vec_info): As above.
> (struct vec_loop_masks): Rename as vec_masks.
> (vec_loop_lens): Rename as vec_lens.
> (vect_record_loop_mask): Rename as vect_record_mask.
> (vect_get_loop_mask): Rename as vect_get_mask.
> (vect_record_loop_len): Rename as vect_record_len.
> (vect_get_loop_len): Rename as vect_get_len.
> (vect_record_mask): New name of vect_record_loop_mask.
> Replace first parameter of type loop_vec_info with a
> pointer to vec_info.
> Don't require the caller to pass a pointer to vec_loop_masks.
> Add an optional parameter of type slp_tree.
> (vect_get_mask): New name of vect_get_loop_mask.
> Replace first parameter of type loop_vec_info with a pointer
> to vec_info.
> Don't require the caller to pass a pointer to vec_loop_masks.
> Add an optional parameter of type slp_tree.
> (vect_record_len): New name of vect_record_loop_len.
> Replace first parameter of type loop_vec_info with a pointer
> to vec_info.
> Don't require the caller to pass a pointer to vec_loop_lens.
> Add an optional parameter of type slp_tree.
> (vect_get_len): New name of vect_get_loop_len.
> Replace first parameter of type loop_vec_info with a pointer
> to vec_info.
> Don't require the caller to pass a pointer to vec_loop_lens.
> Add an optional parameter of type slp_tree.
> (vect_gen_loop_len_mask): Don't require the caller to pass a
> pointer to vec_loop_lens.
> (prepare_vec_mask): Change the type of the first parameter from
> loop_vec_info into a pointer to vec_info.
> ---
> gcc/tree-vect-loop.cc | 175 ++++++++++++------------
> gcc/tree-vect-stmts.cc | 290 ++++++++++++++++-----------------------
> gcc/tree-vectorizer.cc | 7 +-
> gcc/tree-vectorizer.h | 302 +++++++++++++++++++++--------------------
> 4 files changed, 363 insertions(+), 411 deletions(-)
>
> diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
> index 8cda9da7f1a..d15675b2f35 100644
> --- a/gcc/tree-vect-loop.cc
> +++ b/gcc/tree-vect-loop.cc
> @@ -732,7 +732,6 @@ _loop_vec_info::_loop_vec_info (class loop *loop_in, vec_info_shared *shared)
> scalar_costs (nullptr),
> th (0),
> versioning_threshold (0),
> - vectorization_factor (0),
> main_loop_edge (nullptr),
> skip_main_loop_edge (nullptr),
> skip_this_loop_edge (nullptr),
> @@ -741,9 +740,7 @@ _loop_vec_info::_loop_vec_info (class loop *loop_in, vec_info_shared *shared)
> max_vectorization_factor (0),
> mask_skip_niters (NULL_TREE),
> mask_skip_niters_pfa_offset (NULL_TREE),
> - rgroup_compare_type (NULL_TREE),
> simd_if_cond (NULL_TREE),
> - partial_vector_style (vect_partial_vectors_none),
> unaligned_dr (NULL),
> peeling_for_alignment (0),
> ptr_mask (0),
> @@ -759,7 +756,6 @@ _loop_vec_info::_loop_vec_info (class loop *loop_in, vec_info_shared *shared)
> using_decrementing_iv_p (false),
> using_select_vl_p (false),
> allow_mutual_alignment (false),
> - partial_load_store_bias (0),
> peeling_for_gaps (false),
> peeling_for_niter (false),
> early_breaks (false),
> @@ -1011,7 +1007,7 @@ vect_verify_full_masking (loop_vec_info loop_vinfo)
> /* Produce the rgroup controls. */
> for (auto mask : LOOP_VINFO_MASKS (loop_vinfo).mask_set)
> {
> - vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
> + vec_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
> tree vectype = mask.first;
> unsigned nvectors = mask.second;
>
> @@ -1150,7 +1146,7 @@ vect_verify_full_masking_avx512 (loop_vec_info loop_vinfo)
> /* Produce the rgroup controls. */
> for (auto const &mask : LOOP_VINFO_MASKS (loop_vinfo).mask_set)
> {
> - vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
> + vec_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
> tree vectype = mask.first;
> unsigned nvectors = mask.second;
>
> @@ -1189,7 +1185,7 @@ vect_verify_full_masking_avx512 (loop_vec_info loop_vinfo)
> if (!mask_type)
> continue;
>
> - /* For now vect_get_loop_mask only supports integer mode masks
> + /* For now vect_get_mask only supports integer mode masks
> when we need to split it. */
> if (GET_MODE_CLASS (TYPE_MODE (mask_type)) != MODE_INT
> || TYPE_PRECISION (TREE_TYPE (mask_type)) != 1)
> @@ -4207,7 +4203,7 @@ vect_estimate_min_profitable_iters (loop_vec_info loop_vinfo,
> out of mask registers for example. */
>
> /* ??? We are also failing to account for smaller vector masks
> - we generate by splitting larger masks in vect_get_loop_mask. */
> + we generate by splitting larger masks in vect_get_mask. */
>
> /* In the worst case, we need to generate each mask in the prologue
> and in the loop body. We need one splat per group and one
> @@ -6414,12 +6410,9 @@ get_masked_reduction_fn (internal_fn reduc_fn, tree vectype_in)
> static bool
> vectorize_fold_left_reduction (loop_vec_info loop_vinfo,
> stmt_vec_info stmt_info,
> - gimple_stmt_iterator *gsi,
> - slp_tree slp_node,
> + gimple_stmt_iterator *gsi, slp_tree slp_node,
> code_helper code, internal_fn reduc_fn,
> - int num_ops, tree vectype_in,
> - int reduc_index, vec_loop_masks *masks,
> - vec_loop_lens *lens)
> + int num_ops, tree vectype_in, int reduc_index)
> {
> class loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> tree vectype_out = SLP_TREE_VECTYPE (slp_node);
> @@ -6500,8 +6493,8 @@ vectorize_fold_left_reduction (loop_vec_info loop_vinfo,
> tree bias = NULL_TREE;
> if (LOOP_VINFO_FULLY_MASKED_P (loop_vinfo))
> {
> - tree loop_mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
> - vec_num, vectype_in, i);
> + tree loop_mask
> + = vect_get_mask (loop_vinfo, gsi, vec_num, vectype_in, i);
> if (is_cond_op)
> mask = prepare_vec_mask (loop_vinfo, TREE_TYPE (loop_mask),
> loop_mask, vec_opmask[i], gsi);
> @@ -6512,8 +6505,8 @@ vectorize_fold_left_reduction (loop_vec_info loop_vinfo,
> mask = vec_opmask[i];
> if (LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo))
> {
> - len = vect_get_loop_len (loop_vinfo, gsi, lens, vec_num, vectype_in,
> - i, 1, false);
> + len
> + = vect_get_len (loop_vinfo, gsi, vec_num, vectype_in, i, 1, false);
> signed char biasval = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
> bias = build_int_cst (intQI_type_node, biasval);
> if (!is_cond_op)
> @@ -6753,14 +6746,12 @@ vect_reduction_update_partial_vector_usage (loop_vec_info loop_vinfo,
> {
> internal_fn mask_reduc_fn
> = get_masked_reduction_fn (reduc_fn, vectype_in);
> - vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
> - vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo);
> unsigned nvectors = vect_get_num_copies (loop_vinfo, slp_node);
>
> if (mask_reduc_fn == IFN_MASK_LEN_FOLD_LEFT_PLUS)
> - vect_record_loop_len (loop_vinfo, lens, nvectors, vectype_in, 1);
> + vect_record_len (loop_vinfo, nvectors, vectype_in, 1);
> else
> - vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype_in, NULL);
> + vect_record_mask (loop_vinfo, nvectors, vectype_in, NULL_TREE);
> }
> }
>
> @@ -7988,8 +7979,6 @@ vect_transform_reduction (loop_vec_info loop_vinfo,
> && internal_fn_mask_index ((internal_fn)code) != -1)
> ? (internal_fn)code : get_conditional_internal_fn (code, op.type));
>
> - vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
> - vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo);
> bool mask_by_cond_expr = use_mask_by_cond_expr_p (code, cond_fn, vectype_in);
>
> /* Transform. */
> @@ -8021,10 +8010,10 @@ vect_transform_reduction (loop_vec_info loop_vinfo,
> {
> internal_fn reduc_fn = VECT_REDUC_INFO_FN (reduc_info);
> gcc_assert (code.is_tree_code () || cond_fn_p);
> - return vectorize_fold_left_reduction
> - (loop_vinfo, stmt_info, gsi, slp_node,
> - code, reduc_fn, op.num_ops, vectype_in,
> - reduc_index, masks, lens);
> + return vectorize_fold_left_reduction (loop_vinfo, stmt_info, gsi,
> + slp_node, code, reduc_fn,
> + op.num_ops, vectype_in,
> + reduc_index);
> }
>
> bool single_defuse_cycle = VECT_REDUC_INFO_FORCE_SINGLE_CYCLE (reduc_info);
> @@ -8234,9 +8223,8 @@ vect_transform_reduction (loop_vec_info loop_vinfo,
> yet. */
> gcc_assert (!lane_reducing);
>
> - tree mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
> - vec_num, vectype_in,
> - mask_index++);
> + tree mask = vect_get_mask (loop_vinfo, gsi, vec_num, vectype_in,
> + mask_index++);
> gcall *call;
> if (code.is_internal_fn () && cond_fn_p)
> {
> @@ -8264,9 +8252,8 @@ vect_transform_reduction (loop_vec_info loop_vinfo,
>
> if (masked_loop_p && mask_by_cond_expr)
> {
> - tree mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
> - vec_num, vectype_in,
> - mask_index++);
> + tree mask = vect_get_mask (loop_vinfo, gsi, vec_num, vectype_in,
> + mask_index++);
> build_vect_cond_expr (code, vop, mask, gsi);
> }
>
> @@ -10127,9 +10114,7 @@ vectorizable_live_operation_1 (loop_vec_info loop_vinfo, basic_block exit_bb,
> gcc_assert (SLP_TREE_LANES (slp_node) == 1);
> gimple_seq tem = NULL;
> gimple_stmt_iterator gsi = gsi_last (tem);
> - tree len = vect_get_loop_len (loop_vinfo, &gsi,
> - &LOOP_VINFO_LENS (loop_vinfo),
> - 1, vectype, 0, 1, false);
> + tree len = vect_get_len (loop_vinfo, &gsi, 1, vectype, 0, 1, false);
> gimple_seq_add_seq (&stmts, tem);
>
> /* LAST_INDEX = LEN - 1. */
> @@ -10156,9 +10141,7 @@ vectorizable_live_operation_1 (loop_vec_info loop_vinfo, basic_block exit_bb,
> tree scalar_type = TREE_TYPE (vectype);
> gimple_seq tem = NULL;
> gimple_stmt_iterator gsi = gsi_last (tem);
> - tree mask = vect_get_loop_mask (loop_vinfo, &gsi,
> - &LOOP_VINFO_MASKS (loop_vinfo),
> - 1, vectype, 0);
> + tree mask = vect_get_mask (loop_vinfo, &gsi, 1, vectype, 0);
> tree scalar_res;
> gimple_seq_add_seq (&stmts, tem);
>
> @@ -10319,14 +10302,10 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info,
> {
> if (direct_internal_fn_supported_p (IFN_EXTRACT_LAST, vectype,
> OPTIMIZE_FOR_SPEED))
> - vect_record_loop_mask (loop_vinfo,
> - &LOOP_VINFO_MASKS (loop_vinfo),
> - 1, vectype, NULL);
> + vect_record_mask (loop_vinfo, 1, vectype, NULL_TREE);
> else if (can_vec_extract_var_idx_p (
> TYPE_MODE (vectype), TYPE_MODE (TREE_TYPE (vectype))))
> - vect_record_loop_len (loop_vinfo,
> - &LOOP_VINFO_LENS (loop_vinfo),
> - 1, vectype, 1);
> + vect_record_len (loop_vinfo, 1, vectype, 1);
> else
> {
> if (dump_enabled_p ())
> @@ -10587,18 +10566,23 @@ vect_double_mask_nunits (tree old_type, machine_mode new_mode)
> return build_truth_vector_type_for_mode (nunits, new_mode);
> }
>
> -/* Record that a fully-masked version of LOOP_VINFO would need MASKS to
> - contain a sequence of NVECTORS masks that each control a vector of type
> - VECTYPE. If SCALAR_MASK is nonnull, the fully-masked loop would AND
> - these vector masks with the vector version of SCALAR_MASK. */
> +/* Record that a fully-masked version of the loop or SLP region represented by
> + VINFO would need a sequence of NVECTORS masks that each control a
> + vector of type VECTYPE. If SCALAR_MASK is nonnull and VINFO represents a
> + loop, the fully-masked loop would AND these vector masks with the vector
> + version of SCALAR_MASK. SLP_NODE can be null unless VINFO represents an
> + SLP region. */
>
> void
> -vect_record_loop_mask (loop_vec_info loop_vinfo, vec_loop_masks *masks,
> - unsigned int nvectors, tree vectype, tree scalar_mask)
> +vect_record_mask (vec_info *vinfo, unsigned int nvectors,
> + tree vectype, tree scalar_mask, slp_tree slp_node)
> {
> + gcc_assert (!is_a<bb_vec_info> (vinfo) || slp_node != nullptr);
> gcc_assert (nvectors != 0);
> + vec_masks *masks = &vinfo->masks;
> + loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
>
> - if (scalar_mask)
> + if (loop_vinfo && scalar_mask)
> {
> scalar_cond_masked_key cond (scalar_mask, nvectors);
> loop_vinfo->scalar_cond_masked_set.add (cond);
> @@ -10607,20 +10591,24 @@ vect_record_loop_mask (loop_vec_info loop_vinfo, vec_loop_masks *masks,
> masks->mask_set.add (std::make_pair (vectype, nvectors));
> }
>
> -/* Given a complete set of masks MASKS, extract mask number INDEX
> - for an rgroup that operates on NVECTORS vectors of type VECTYPE,
> - where 0 <= INDEX < NVECTORS. Insert any set-up statements before GSI.
> +/* Given a complete set of masks for the loop or SLP region represented by
> + VINFO, extract mask number INDEX for an rgroup that operates on NVECTORS
> + vectors of type VECTYPE, where 0 <= INDEX < NVECTORS. Insert any set-up
> + statements before GSI. SLP_NODE can be null unless VINFO represents a basic
> + block SLP region.
>
> - See the comment above vec_loop_masks for more details about the mask
> + See the comment above vec_masks for more details about the mask
> arrangement. */
>
> tree
> -vect_get_loop_mask (loop_vec_info loop_vinfo,
> - gimple_stmt_iterator *gsi, vec_loop_masks *masks,
> - unsigned int nvectors, tree vectype, unsigned int index)
> +vect_get_mask (vec_info *vinfo, gimple_stmt_iterator *gsi,
> + unsigned int nvectors, tree vectype, unsigned int index,
> + slp_tree slp_node)
> {
> - if (LOOP_VINFO_PARTIAL_VECTORS_STYLE (loop_vinfo)
> - == vect_partial_vectors_while_ult)
> + gcc_assert (!is_a<bb_vec_info> (vinfo) || slp_node != nullptr);
> + vec_masks *masks = &LOOP_VINFO_MASKS (vinfo);
> +
> + if (vinfo->partial_vector_style == vect_partial_vectors_while_ult)
> {
> rgroup_controls *rgm = &(masks->rgc_vec)[nvectors - 1];
> tree mask_type = rgm->type;
> @@ -10659,14 +10647,14 @@ vect_get_loop_mask (loop_vec_info loop_vinfo,
> }
> return mask;
> }
> - else if (LOOP_VINFO_PARTIAL_VECTORS_STYLE (loop_vinfo)
> - == vect_partial_vectors_avx512)
> + else if (vinfo->partial_vector_style == vect_partial_vectors_avx512)
> {
> /* The number of scalars per iteration and the number of vectors are
> both compile-time constants. */
> unsigned int nscalars_per_iter
> = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
> - LOOP_VINFO_VECT_FACTOR (loop_vinfo)).to_constant ();
> + vinfo->vectorization_factor)
> + .to_constant ();
>
> rgroup_controls *rgm = &masks->rgc_vec[nscalars_per_iter - 1];
>
> @@ -10727,16 +10715,19 @@ vect_get_loop_mask (loop_vec_info loop_vinfo,
> gcc_unreachable ();
> }
>
> -/* Record that LOOP_VINFO would need LENS to contain a sequence of NVECTORS
> - lengths for controlling an operation on VECTYPE. The operation splits
> - each element of VECTYPE into FACTOR separate subelements, measuring the
> - length as a number of these subelements. */
> +/* Record that the loop or SLP region represented by VINFO would need a sequence
> + of NVECTORS lengths for controlling an operation on VECTYPE. The operation
> + splits each element of VECTYPE into FACTOR separate subelements, measuring
> + the length as a number of these subelements. SLP_NODE can be null unless
> + VINFO represents a basic block SLP region. */
>
> void
> -vect_record_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens,
> - unsigned int nvectors, tree vectype, unsigned int factor)
> +vect_record_len (vec_info *vinfo, unsigned int nvectors, tree vectype,
> + unsigned int factor, slp_tree slp_node)
> {
> + gcc_assert (!is_a<bb_vec_info> (vinfo) || slp_node != nullptr);
> gcc_assert (nvectors != 0);
> + vec_lens *lens = &vinfo->lens;
> if (lens->length () < nvectors)
> lens->safe_grow_cleared (nvectors, true);
> rgroup_controls *rgl = &(*lens)[nvectors - 1];
> @@ -10745,7 +10736,8 @@ vect_record_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens,
> the number of vectors are both compile-time constants. */
> unsigned int nscalars_per_iter
> = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
> - LOOP_VINFO_VECT_FACTOR (loop_vinfo)).to_constant ();
> + vinfo->vectorization_factor)
> + .to_constant ();
>
> if (rgl->max_nscalars_per_iter < nscalars_per_iter)
> {
> @@ -10761,20 +10753,23 @@ vect_record_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens,
> }
> }
>
> -/* Given a complete set of lengths LENS, extract length number INDEX
> - for an rgroup that operates on NVECTORS vectors of type VECTYPE,
> - where 0 <= INDEX < NVECTORS. Return a value that contains FACTOR
> - multiplied by the number of elements that should be processed.
> - Insert any set-up statements before GSI. */
> +/* Given a complete set of lengths for the loop or SLP region represented by
> + VINFO, extract length number INDEX for an rgroup that operates on NVECTORS
> + vectors of type VECTYPE, where 0 <= INDEX < NVECTORS. Return a value that
> + contains FACTOR multiplied by the number of elements that should be
> + processed. Insert any set-up statements before GSI. SLP_NODE can be null
> + unless VINFO represents a basic block SLP region. */
>
> tree
> -vect_get_loop_len (loop_vec_info loop_vinfo, gimple_stmt_iterator *gsi,
> - vec_loop_lens *lens, unsigned int nvectors, tree vectype,
> - unsigned int index, unsigned int factor, bool adjusted)
> +vect_get_len (vec_info *vinfo, gimple_stmt_iterator *gsi, unsigned int nvectors,
> + tree vectype, unsigned int index, unsigned int factor,
> + bool adjusted, slp_tree slp_node)
> {
> + gcc_assert (!is_a<bb_vec_info> (vinfo) || slp_node != nullptr);
> + vec_lens *lens = &vinfo->lens;
> rgroup_controls *rgl = &(*lens)[nvectors - 1];
> bool use_bias_adjusted_len =
> - LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo) != 0;
> + vinfo->partial_load_store_bias != 0;
>
> /* Populate the rgroup's len array, if this is the first time we've
> used it. */
> @@ -10783,7 +10778,7 @@ vect_get_loop_len (loop_vec_info loop_vinfo, gimple_stmt_iterator *gsi,
> rgl->controls.safe_grow_cleared (nvectors, true);
> for (unsigned int i = 0; i < nvectors; ++i)
> {
> - tree len_type = LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo);
> + tree len_type = vinfo->rgroup_compare_type;
> gcc_assert (len_type != NULL_TREE);
>
> tree len = make_temp_ssa_name (len_type, NULL, "loop_len");
> @@ -10818,7 +10813,7 @@ vect_get_loop_len (loop_vec_info loop_vinfo, gimple_stmt_iterator *gsi,
> are N times bigger than X's. */
> gcc_assert (multiple_p (nunits1, nunits2));
> factor = exact_div (nunits1, nunits2).to_constant ();
> - tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
> + tree iv_type = vinfo->rgroup_iv_type;
> gimple_seq seq = NULL;
> loop_len = gimple_build (&seq, EXACT_DIV_EXPR, iv_type, loop_len,
> build_int_cst (iv_type, factor));
> @@ -10832,13 +10827,13 @@ vect_get_loop_len (loop_vec_info loop_vinfo, gimple_stmt_iterator *gsi,
> the number of vectors are both compile-time constants. */
> unsigned int nscalars_per_iter
> = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
> - LOOP_VINFO_VECT_FACTOR (loop_vinfo)).to_constant ();
> + vinfo->vectorization_factor).to_constant ();
> unsigned int rglvecsize = rgl->factor * rgl->max_nscalars_per_iter;
> unsigned int vecsize = nscalars_per_iter * factor;
> if (rglvecsize > vecsize)
> {
> unsigned int fac = rglvecsize / vecsize;
> - tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
> + tree iv_type = vinfo->rgroup_iv_type;
> gimple_seq seq = NULL;
> loop_len = gimple_build (&seq, EXACT_DIV_EXPR, iv_type, loop_len,
> build_int_cst (iv_type, fac));
> @@ -10848,7 +10843,7 @@ vect_get_loop_len (loop_vec_info loop_vinfo, gimple_stmt_iterator *gsi,
> else if (rglvecsize < vecsize)
> {
> unsigned int fac = vecsize / rglvecsize;
> - tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
> + tree iv_type = vinfo->rgroup_iv_type;
> gimple_seq seq = NULL;
> loop_len = gimple_build (&seq, MULT_EXPR, iv_type, loop_len,
> build_int_cst (iv_type, fac));
> @@ -10866,14 +10861,14 @@ vect_get_loop_len (loop_vec_info loop_vinfo, gimple_stmt_iterator *gsi,
> */
> tree
> vect_gen_loop_len_mask (loop_vec_info loop_vinfo, gimple_stmt_iterator *gsi,
> - gimple_stmt_iterator *cond_gsi, vec_loop_lens *lens,
> - unsigned int nvectors, tree vectype, tree stmt,
> - unsigned int index, unsigned int factor)
> + gimple_stmt_iterator *cond_gsi, unsigned int nvectors,
> + tree vectype, tree stmt, unsigned int index,
> + unsigned int factor)
> {
> tree all_one_mask = build_all_ones_cst (vectype);
> tree all_zero_mask = build_zero_cst (vectype);
> - tree len = vect_get_loop_len (loop_vinfo, gsi, lens, nvectors, vectype, index,
> - factor, true);
> + tree len
> + = vect_get_len (loop_vinfo, gsi, nvectors, vectype, index, factor, true);
> tree bias = build_int_cst (intQI_type_node,
> LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo));
> tree len_mask = make_temp_ssa_name (TREE_TYPE (stmt), NULL, "vec_len_mask");
> diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
> index c0c3d750217..e76778c8b94 100644
> --- a/gcc/tree-vect-stmts.cc
> +++ b/gcc/tree-vect-stmts.cc
> @@ -1440,8 +1440,6 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype,
> }
>
> unsigned int nvectors = vect_get_num_copies (loop_vinfo, slp_node);
> - vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
> - vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo);
> machine_mode vecmode = TYPE_MODE (vectype);
> bool is_load = (vls_type == VLS_LOAD);
> if (memory_access_type == VMAT_LOAD_STORE_LANES)
> @@ -1452,10 +1450,9 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype,
> elsvals)
> : vect_store_lanes_supported (vectype, group_size, true));
> if (ifn == IFN_MASK_LEN_LOAD_LANES || ifn == IFN_MASK_LEN_STORE_LANES)
> - vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, 1);
> + vect_record_len (loop_vinfo, nvectors, vectype, 1);
> else if (ifn == IFN_MASK_LOAD_LANES || ifn == IFN_MASK_STORE_LANES)
> - vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype,
> - scalar_mask);
> + vect_record_mask (loop_vinfo, nvectors, vectype, scalar_mask);
> else
> {
> if (dump_enabled_p ())
> @@ -1497,14 +1494,13 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype,
> memory_type,
> off_vectype, scale,
> elsvals))
> - vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, 1);
> + vect_record_len (loop_vinfo, nvectors, vectype, 1);
> else if (internal_gather_scatter_fn_supported_p (ifn, vectype,
> memory_type,
> off_vectype, scale,
> elsvals)
> || memory_access_type == VMAT_GATHER_SCATTER_LEGACY)
> - vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype,
> - scalar_mask);
> + vect_record_mask (loop_vinfo, nvectors, vectype, scalar_mask);
> else
> {
> if (dump_enabled_p ())
> @@ -1560,7 +1556,7 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype,
> {
> nvectors = group_memory_nvectors (group_size * vf, nunits);
> unsigned factor = (vecmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vecmode);
> - vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, factor);
> + vect_record_len (loop_vinfo, nvectors, vectype, factor);
> using_partial_vectors_p = true;
> }
> else if (targetm.vectorize.get_mask_mode (vecmode).exists (&mask_mode)
> @@ -1568,7 +1564,7 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype,
> elsvals))
> {
> nvectors = group_memory_nvectors (group_size * vf, nunits);
> - vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype, scalar_mask);
> + vect_record_mask (loop_vinfo, nvectors, vectype, scalar_mask);
> using_partial_vectors_p = true;
> }
>
> @@ -1584,30 +1580,31 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype,
> }
>
> /* Return the mask input to a masked load or store. VEC_MASK is the vectorized
> - form of the scalar mask condition and LOOP_MASK, if nonnull, is the mask
> - that needs to be applied to all loads and stores in a vectorized loop.
> - Return VEC_MASK if LOOP_MASK is null or if VEC_MASK is already masked,
> - otherwise return VEC_MASK & LOOP_MASK.
> + form of the scalar mask condition and LOOP_OR_TAIL_MASK, if nonnull, is the
> + mask that needs to be applied because partial vectors are used in the loop or
> + SLP region represented by VINFO. Return VEC_MASK if LOOP_OR_TAIL_MASK is
> + null or if VEC_MASK is already masked, otherwise return VEC_MASK &
> + LOOP_OR_TAIL_MASK.
>
> MASK_TYPE is the type of both masks. If new statements are needed,
> insert them before GSI. */
>
> tree
> -prepare_vec_mask (loop_vec_info loop_vinfo, tree mask_type, tree loop_mask,
> +prepare_vec_mask (vec_info *vinfo, tree mask_type, tree loop_or_tail_mask,
> tree vec_mask, gimple_stmt_iterator *gsi)
> {
> gcc_assert (useless_type_conversion_p (mask_type, TREE_TYPE (vec_mask)));
> - if (!loop_mask)
> + if (!loop_or_tail_mask)
> return vec_mask;
>
> - gcc_assert (TREE_TYPE (loop_mask) == mask_type);
> + gcc_assert (TREE_TYPE (loop_or_tail_mask) == mask_type);
>
> - if (loop_vinfo->vec_cond_masked_set.contains ({ vec_mask, loop_mask }))
> + if (vinfo->vec_cond_masked_set.contains ({ vec_mask, loop_or_tail_mask }))
> return vec_mask;
>
> tree and_res = make_temp_ssa_name (mask_type, NULL, "vec_mask_and");
> - gimple *and_stmt = gimple_build_assign (and_res, BIT_AND_EXPR,
> - vec_mask, loop_mask);
> + gimple *and_stmt
> + = gimple_build_assign (and_res, BIT_AND_EXPR, vec_mask, loop_or_tail_mask);
>
> gsi_insert_before (gsi, and_stmt, GSI_SAME_STMT);
> return and_res;
> @@ -3682,8 +3679,6 @@ vectorizable_call (vec_info *vinfo,
> internal_fn cond_fn = (internal_fn_mask_index (ifn) != -1
> ? ifn : get_conditional_internal_fn (ifn));
> internal_fn cond_len_fn = get_len_internal_fn (cond_fn);
> - vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) : NULL);
> - vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) : NULL);
> unsigned int nvectors = vect_get_num_copies (vinfo, slp_node);
> if (cost_vec) /* transformation not required. */
> {
> @@ -3727,11 +3722,10 @@ vectorizable_call (vec_info *vinfo,
> if (cond_len_fn != IFN_LAST
> && direct_internal_fn_supported_p (cond_len_fn, vectype_out,
> OPTIMIZE_FOR_SPEED))
> - vect_record_loop_len (loop_vinfo, lens, nvectors, vectype_out,
> - 1);
> + vect_record_len (loop_vinfo, nvectors, vectype_out, 1);
> else
> - vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype_out,
> - scalar_mask);
> + vect_record_mask (loop_vinfo, nvectors, vectype_out,
> + scalar_mask);
> }
> }
> return true;
> @@ -3811,9 +3805,8 @@ vectorizable_call (vec_info *vinfo,
> if (masked_loop_p)
> {
> unsigned int vec_num = vec_oprnds0.length ();
> - vargs[varg++] = vect_get_loop_mask (loop_vinfo, gsi,
> - masks, vec_num,
> - vectype_out, i);
> + vargs[varg++] = vect_get_mask (loop_vinfo, gsi, vec_num,
> + vectype_out, i);
> }
> else
> {
> @@ -3872,8 +3865,8 @@ vectorizable_call (vec_info *vinfo,
> if (len_opno >= 0 && len_loop_p)
> {
> unsigned int vec_num = vec_oprnds0.length ();
> - tree len = vect_get_loop_len (loop_vinfo, gsi, lens,
> - vec_num, vectype_out, i, 1, true);
> + tree len = vect_get_len (loop_vinfo, gsi, vec_num,
> + vectype_out, i, 1, true);
> signed char biasval
> = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
> tree bias = build_int_cst (intQI_type_node, biasval);
> @@ -3883,8 +3876,8 @@ vectorizable_call (vec_info *vinfo,
> else if (mask_opno >= 0 && masked_loop_p)
> {
> unsigned int vec_num = vec_oprnds0.length ();
> - tree mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
> - vec_num, vectype_out, i);
> + tree mask = vect_get_mask (loop_vinfo, gsi, vec_num,
> + vectype_out, i);
> vargs[mask_opno]
> = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask,
> vargs[mask_opno], gsi);
> @@ -4455,9 +4448,7 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info,
> if (masked_call_offset)
> /* When there is an explicit mask we require the
> number of elements to match up. */
> - vect_record_loop_mask (loop_vinfo,
> - &LOOP_VINFO_MASKS (loop_vinfo),
> - ncopies_in, vectype, NULL_TREE);
> + vect_record_mask (loop_vinfo, ncopies_in, vectype, NULL_TREE);
> else
> {
> /* When there is no explicit mask on the call we have
> @@ -4488,9 +4479,8 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info,
> callee_nelements = TYPE_VECTOR_SUBPARTS (masktype);
> }
> auto o = vector_unroll_factor (nunits, callee_nelements);
> - vect_record_loop_mask (loop_vinfo,
> - &LOOP_VINFO_MASKS (loop_vinfo),
> - ncopies * o, masktype, NULL_TREE);
> + vect_record_mask (loop_vinfo, ncopies * o, masktype,
> + NULL_TREE);
> }
> }
> else if ((bestn->simdclone->args[i].arg_type
> @@ -4688,12 +4678,9 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info,
> if (loop_vinfo
> && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo))
> {
> - vec_loop_masks *loop_masks
> - = &LOOP_VINFO_MASKS (loop_vinfo);
> tree loop_mask
> - = vect_get_loop_mask (loop_vinfo, gsi,
> - loop_masks, ncopies_in,
> - vectype, j);
> + = vect_get_mask (loop_vinfo, gsi, ncopies_in,
> + vectype, j);
> vec_oprnd0
> = prepare_vec_mask (loop_vinfo,
> TREE_TYPE (loop_mask),
> @@ -4701,7 +4688,6 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info,
> gsi);
> loop_vinfo->vec_cond_masked_set.add ({ vec_oprnd0,
> loop_mask });
> -
> }
> vec_oprnd0
> = build3 (VEC_COND_EXPR, atype, vec_oprnd0,
> @@ -4749,12 +4735,9 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info,
> if (loop_vinfo
> && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo))
> {
> - vec_loop_masks *loop_masks
> - = &LOOP_VINFO_MASKS (loop_vinfo);
> tree loop_mask
> - = vect_get_loop_mask (loop_vinfo, gsi,
> - loop_masks, ncopies_in,
> - vectype, j);
> + = vect_get_mask (loop_vinfo, gsi, ncopies_in,
> + vectype, j);
> vec_oprnd0
> = prepare_vec_mask (loop_vinfo,
> TREE_TYPE (loop_mask),
> @@ -4901,11 +4884,8 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info,
> for (m = j * o; m < (j + 1) * o; m++)
> {
> if (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo))
> - {
> - vec_loop_masks *loop_masks = &LOOP_VINFO_MASKS (loop_vinfo);
> - mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
> - ncopies * o, mask_vectype, m);
> - }
> + mask = vect_get_mask (loop_vinfo, gsi, ncopies * o,
> + mask_vectype, m);
> else
> mask = vect_build_all_ones_mask (vinfo, stmt_info,
> mask_argtype);
> @@ -6783,8 +6763,6 @@ vectorizable_operation (vec_info *vinfo,
> }
>
> int reduc_idx = SLP_TREE_REDUC_IDX (slp_node);
> - vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) : NULL);
> - vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) : NULL);
> internal_fn cond_fn = get_conditional_internal_fn (code);
> internal_fn cond_len_fn = get_conditional_len_internal_fn (code);
>
> @@ -6808,13 +6786,11 @@ vectorizable_operation (vec_info *vinfo,
> if (cond_len_fn != IFN_LAST
> && direct_internal_fn_supported_p (cond_len_fn, vectype,
> OPTIMIZE_FOR_SPEED))
> - vect_record_loop_len (loop_vinfo, lens, vec_num, vectype,
> - 1);
> + vect_record_len (loop_vinfo, vec_num, vectype, 1);
> else if (cond_fn != IFN_LAST
> && direct_internal_fn_supported_p (cond_fn, vectype,
> OPTIMIZE_FOR_SPEED))
> - vect_record_loop_mask (loop_vinfo, masks, vec_num,
> - vectype, NULL);
> + vect_record_mask (loop_vinfo, vec_num, vectype, NULL_TREE);
> else
> {
> if (dump_enabled_p ())
> @@ -7074,8 +7050,7 @@ vectorizable_operation (vec_info *vinfo,
> {
> tree mask;
> if (masked_loop_p)
> - mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
> - vec_num, vectype, i);
> + mask = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
> else
> /* Dummy mask. */
> mask = build_minus_one_cst (truth_type_for (vectype));
> @@ -7101,8 +7076,8 @@ vectorizable_operation (vec_info *vinfo,
> }
> if (len_loop_p)
> {
> - tree len = vect_get_loop_len (loop_vinfo, gsi, lens,
> - vec_num, vectype, i, 1, true);
> + tree len
> + = vect_get_len (loop_vinfo, gsi, vec_num, vectype, i, 1, true);
> signed char biasval
> = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
> tree bias = build_int_cst (intQI_type_node, biasval);
> @@ -7131,18 +7106,14 @@ vectorizable_operation (vec_info *vinfo,
> {
> if (loop_vinfo->scalar_cond_masked_set.contains ({ op0, vec_num }))
> {
> - mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
> - vec_num, vectype, i);
> -
> + mask = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
> vop0 = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask,
> vop0, gsi);
> }
>
> if (loop_vinfo->scalar_cond_masked_set.contains ({ op1, vec_num }))
> {
> - mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
> - vec_num, vectype, i);
> -
> + mask = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
> vop1 = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask,
> vop1, gsi);
> }
> @@ -8671,14 +8642,8 @@ vectorizable_store (vec_info *vinfo,
> }
>
> gcc_assert (alignment_support_scheme);
> - vec_loop_masks *loop_masks
> - = (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)
> - ? &LOOP_VINFO_MASKS (loop_vinfo)
> - : NULL);
> - vec_loop_lens *loop_lens
> - = (loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo)
> - ? &LOOP_VINFO_LENS (loop_vinfo)
> - : NULL);
> + bool masks_p = loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
> + bool lens_p = loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo);
>
> /* The vect_transform_stmt and vect_analyze_stmt will go here but there
> are some difference here. We cannot enable both the lens and masks
> @@ -8686,14 +8651,14 @@ vectorizable_store (vec_info *vinfo,
> Shouldn't go with length-based approach if fully masked. */
> if (cost_vec == NULL)
> /* The cost_vec is NULL during transform. */
> - gcc_assert ((!loop_lens || !loop_masks));
> + gcc_assert ((!lens_p || !masks_p));
>
> /* Targets with store-lane instructions must not require explicit
> realignment. vect_supportable_dr_alignment always returns either
> dr_aligned or dr_unaligned_supported for masked operations. */
> gcc_assert ((memory_access_type != VMAT_LOAD_STORE_LANES
> && !mask_node
> - && !loop_masks)
> + && !masks_p)
> || alignment_support_scheme == dr_aligned
> || alignment_support_scheme == dr_unaligned_supported);
>
> @@ -8831,20 +8796,19 @@ vectorizable_store (vec_info *vinfo,
> tree final_mask = NULL;
> tree final_len = NULL;
> tree bias = NULL;
> - if (loop_masks)
> - final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
> - ncopies, vectype, j);
> + if (masks_p)
> + final_mask
> + = vect_get_mask (loop_vinfo, gsi, ncopies, vectype, j);
> if (vec_mask)
> final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, final_mask,
> vec_mask, gsi);
>
> if (lanes_ifn == IFN_MASK_LEN_STORE_LANES)
> {
> - if (loop_lens)
> - final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
> - ncopies, vectype, j, 1, true);
> - else
> - final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
> + if (lens_p)
> + final_len = vect_get_len (loop_vinfo, gsi, ncopies, vectype, j,
> + 1, true);
> + else final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
> signed char biasval
> = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
> bias = build_int_cst (intQI_type_node, biasval);
> @@ -8973,10 +8937,9 @@ vectorizable_store (vec_info *vinfo,
> tree bias = NULL_TREE;
> if (!costing_p)
> {
> - if (loop_masks)
> - final_mask = vect_get_loop_mask (loop_vinfo, gsi,
> - loop_masks, num_stmts,
> - vectype, j);
> + if (masks_p)
> + final_mask
> + = vect_get_mask (loop_vinfo, gsi, num_stmts, vectype, j);
> if (vec_mask)
> final_mask = prepare_vec_mask (loop_vinfo, mask_vectype,
> final_mask, vec_mask, gsi);
> @@ -9042,10 +9005,9 @@ vectorizable_store (vec_info *vinfo,
>
> if (ls.gs.ifn == IFN_MASK_LEN_SCATTER_STORE)
> {
> - if (loop_lens)
> - final_len = vect_get_loop_len (loop_vinfo, gsi,
> - loop_lens, num_stmts,
> - vectype, j, 1, true);
> + if (lens_p)
> + final_len = vect_get_len (loop_vinfo, gsi, num_stmts,
> + vectype, j, 1, true);
> else
> final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
>
> @@ -9337,7 +9299,7 @@ vectorizable_store (vec_info *vinfo,
> = STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) != 0;
> if (!costing_p
> && simd_lane_access_p
> - && !loop_masks
> + && !lens_p
> && TREE_CODE (DR_BASE_ADDRESS (first_dr_info->dr)) == ADDR_EXPR
> && VAR_P (TREE_OPERAND (DR_BASE_ADDRESS (first_dr_info->dr), 0))
> && integer_zerop (get_dr_vinfo_offset (vinfo, first_dr_info))
> @@ -9391,9 +9353,8 @@ vectorizable_store (vec_info *vinfo,
> tree final_mask = NULL_TREE;
> tree final_len = NULL_TREE;
> tree bias = NULL_TREE;
> - if (loop_masks)
> - final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
> - vec_num, vectype, i);
> + if (masks_p)
> + final_mask = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
> if (vec_mask)
> vec_mask = vec_masks[i];
> if (vec_mask)
> @@ -9422,19 +9383,19 @@ vectorizable_store (vec_info *vinfo,
> set_ptr_info_alignment (get_ptr_info (dataref_ptr), align, misalign);
> align = least_bit_hwi (misalign | align);
>
> - /* Compute IFN when LOOP_LENS or final_mask valid. */
> + /* Compute IFN when LEN_LOOP_P or final_mask valid. */
> machine_mode vmode = TYPE_MODE (vectype);
> machine_mode new_vmode = vmode;
> internal_fn partial_ifn = IFN_LAST;
> - if (loop_lens)
> + if (lens_p)
> {
> opt_machine_mode new_ovmode
> = get_len_load_store_mode (vmode, false, &partial_ifn);
> new_vmode = new_ovmode.require ();
> unsigned factor
> = (new_ovmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vmode);
> - final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
> - vec_num, vectype, i, factor, true);
> + final_len
> + = vect_get_len (loop_vinfo, gsi, vec_num, vectype, i, factor, true);
> }
> else if (final_mask)
> {
> @@ -9449,7 +9410,7 @@ vectorizable_store (vec_info *vinfo,
> if (!final_len)
> {
> /* Pass VF value to 'len' argument of
> - MASK_LEN_STORE if LOOP_LENS is invalid. */
> + MASK_LEN_STORE if lens_p is false. */
> final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
> }
> if (!final_mask)
> @@ -10581,14 +10542,8 @@ vectorizable_load (vec_info *vinfo,
> ref_type = reference_alias_ptr_type (DR_REF (first_dr_info->dr));
> }
>
> - vec_loop_masks *loop_masks
> - = (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)
> - ? &LOOP_VINFO_MASKS (loop_vinfo)
> - : NULL);
> - vec_loop_lens *loop_lens
> - = (loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo)
> - ? &LOOP_VINFO_LENS (loop_vinfo)
> - : NULL);
> + bool masks_p = loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
> + bool lens_p = loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo);
>
> /* The vect_transform_stmt and vect_analyze_stmt will go here but there
> are some difference here. We cannot enable both the lens and masks
> @@ -10596,7 +10551,7 @@ vectorizable_load (vec_info *vinfo,
> Shouldn't go with length-based approach if fully masked. */
> if (cost_vec == NULL)
> /* The cost_vec is NULL during transform. */
> - gcc_assert ((!loop_lens || !loop_masks));
> + gcc_assert ((!lens_p || !masks_p));
>
> /* Targets with store-lane instructions must not require explicit
> realignment. vect_supportable_dr_alignment always returns either
> @@ -10604,7 +10559,7 @@ vectorizable_load (vec_info *vinfo,
> operations. */
> gcc_assert ((memory_access_type != VMAT_LOAD_STORE_LANES
> && !mask_node
> - && !loop_masks)
> + && !masks_p)
> || mat_gather_scatter_p (memory_access_type)
> || alignment_support_scheme == dr_aligned
> || alignment_support_scheme == dr_unaligned_supported);
> @@ -10808,18 +10763,18 @@ vectorizable_load (vec_info *vinfo,
> tree final_mask = NULL_TREE;
> tree final_len = NULL_TREE;
> tree bias = NULL_TREE;
> - if (loop_masks)
> - final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
> - ncopies, vectype, j);
> + if (masks_p)
> + final_mask
> + = vect_get_mask (loop_vinfo, gsi, ncopies, vectype, j);
> if (vec_mask)
> final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, final_mask,
> vec_mask, gsi);
>
> if (lanes_ifn == IFN_MASK_LEN_LOAD_LANES)
> {
> - if (loop_lens)
> - final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
> - ncopies, vectype, j, 1, true);
> + if (lens_p)
> + final_len
> + = vect_get_len (loop_vinfo, gsi, ncopies, vectype, j, 1, true);
> else
> final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
> signed char biasval
> @@ -10958,9 +10913,9 @@ vectorizable_load (vec_info *vinfo,
> {
> if (mask_node)
> vec_mask = vec_masks[i];
> - if (loop_masks)
> - final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
> - vec_num, vectype, i);
> + if (masks_p)
> + final_mask
> + = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
> if (vec_mask)
> final_mask = prepare_vec_mask (loop_vinfo, mask_vectype,
> final_mask, vec_mask, gsi);
> @@ -11030,9 +10985,9 @@ vectorizable_load (vec_info *vinfo,
>
> if (ls.gs.ifn == IFN_MASK_LEN_GATHER_LOAD)
> {
> - if (loop_lens)
> - final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
> - vec_num, vectype, i, 1, true);
> + if (lens_p)
> + final_len = vect_get_len (loop_vinfo, gsi, vec_num, vectype,
> + i, 1, true);
> else
> final_len = build_int_cst (sizetype,
> TYPE_VECTOR_SUBPARTS (vectype));
> @@ -11432,9 +11387,9 @@ vectorizable_load (vec_info *vinfo,
> {
> if (mask_node)
> vec_mask = vec_masks[i];
> - if (loop_masks)
> - final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
> - vec_num, vectype, i);
> + if (masks_p)
> + final_mask
> + = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
> if (vec_mask)
> final_mask = prepare_vec_mask (loop_vinfo, mask_vectype,
> final_mask, vec_mask, gsi);
> @@ -11471,19 +11426,19 @@ vectorizable_load (vec_info *vinfo,
> misalign);
> align = least_bit_hwi (misalign | align);
>
> - /* Compute IFN when LOOP_LENS or final_mask valid. */
> + /* Compute IFN when LEN_LOOP_P or final_mask valid. */
> machine_mode vmode = TYPE_MODE (vectype);
> machine_mode new_vmode = vmode;
> internal_fn partial_ifn = IFN_LAST;
> - if (loop_lens)
> + if (lens_p)
> {
> opt_machine_mode new_ovmode
> = get_len_load_store_mode (vmode, true, &partial_ifn);
> new_vmode = new_ovmode.require ();
> unsigned factor
> = (new_ovmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vmode);
> - final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
> - vec_num, vectype, i, factor, true);
> + final_len = vect_get_len (loop_vinfo, gsi, vec_num, vectype, i,
> + factor, true);
> }
> else if (final_mask)
> {
> @@ -11499,7 +11454,7 @@ vectorizable_load (vec_info *vinfo,
> if (!final_len)
> {
> /* Pass VF value to 'len' argument of
> - MASK_LEN_LOAD if LOOP_LENS is invalid. */
> + MASK_LEN_LOAD if LEN_LOOP_P is false. */
> final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
> }
> if (!final_mask)
> @@ -12351,13 +12306,9 @@ vectorizable_condition (vec_info *vinfo,
> {
> if (direct_internal_fn_supported_p (IFN_LEN_FOLD_EXTRACT_LAST,
> vectype, OPTIMIZE_FOR_SPEED))
> - vect_record_loop_len (loop_vinfo,
> - &LOOP_VINFO_LENS (loop_vinfo),
> - vec_num, vectype, 1);
> + vect_record_len (loop_vinfo, vec_num, vectype, 1);
> else
> - vect_record_loop_mask (loop_vinfo,
> - &LOOP_VINFO_MASKS (loop_vinfo),
> - vec_num, vectype, NULL);
> + vect_record_mask (loop_vinfo, vec_num, vectype, NULL_TREE);
> }
> /* Extra inactive lanes should be safe for vect_nested_cycle. */
> else if (!nested_cycle_p)
> @@ -12387,22 +12338,22 @@ vectorizable_condition (vec_info *vinfo,
> /* See whether another part of the vectorized code applies a loop
> mask to the condition, or to its inverse. */
>
> - vec_loop_masks *masks = NULL;
> - vec_loop_lens *lens = NULL;
> + bool fully_masked_p = false;
> + bool fully_with_length_p = false;
The names fully_masked_p and fully_with_length_p sound like local copies
of LOOP_VINFO_FULLY_MASKED_P and LOOP_VINFO_FULLY_WITH_LENGTH_P, but
they actually indicate something more. Could they be renamed?
> if (loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo))
> {
> if (reduction_type == EXTRACT_LAST_REDUCTION)
> - lens = &LOOP_VINFO_LENS (loop_vinfo);
> + fully_with_length_p = true;
> }
> else if (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo))
> {
> if (reduction_type == EXTRACT_LAST_REDUCTION)
> - masks = &LOOP_VINFO_MASKS (loop_vinfo);
> + fully_masked_p = true;
> else
> {
> scalar_cond_masked_key cond (cond_expr, 1);
> if (loop_vinfo->scalar_cond_masked_set.contains (cond))
> - masks = &LOOP_VINFO_MASKS (loop_vinfo);
> + fully_masked_p = true;
> else
> {
> bool honor_nans = HONOR_NANS (TREE_TYPE (cond.op0));
> @@ -12410,7 +12361,7 @@ vectorizable_condition (vec_info *vinfo,
> cond.code = invert_tree_comparison (cond.code, honor_nans);
> if (!masked && loop_vinfo->scalar_cond_masked_set.contains (cond))
> {
> - masks = &LOOP_VINFO_MASKS (loop_vinfo);
> + fully_masked_p = true;
> cond_code = cond.code;
> swap_cond_operands = true;
> }
> @@ -12423,7 +12374,7 @@ vectorizable_condition (vec_info *vinfo,
> cond.code = orig_code;
> if (loop_vinfo->scalar_cond_masked_set.contains (cond))
> {
> - masks = &LOOP_VINFO_MASKS (loop_vinfo);
> + fully_masked_p = true;
> cond_code = cond.code;
> swap_cond_operands = true;
> must_invert_cmp_result = true;
> @@ -12536,7 +12487,8 @@ vectorizable_condition (vec_info *vinfo,
> in cases where that's necessary. */
>
> tree len = NULL_TREE, bias = NULL_TREE;
> - if (masks || lens || reduction_type == EXTRACT_LAST_REDUCTION)
> + if (fully_masked_p || fully_with_length_p
> + || reduction_type == EXTRACT_LAST_REDUCTION)
> {
> if (!is_gimple_val (vec_compare))
> {
> @@ -12560,12 +12512,12 @@ vectorizable_condition (vec_info *vinfo,
> if (direct_internal_fn_supported_p (IFN_LEN_FOLD_EXTRACT_LAST,
> vectype, OPTIMIZE_FOR_SPEED))
> {
> - if (lens)
> + if (fully_with_length_p)
> {
> /* ??? Do we really want the adjusted LEN here? Isn't this
> based on number of elements? */
> - len = vect_get_loop_len (loop_vinfo, gsi, lens,
> - vec_num, vectype, i, 1, true);
> + len = vect_get_len (loop_vinfo, gsi, vec_num, vectype, i, 1,
> + true);
> signed char biasval
> = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
> bias = build_int_cst (intQI_type_node, biasval);
> @@ -12576,11 +12528,10 @@ vectorizable_condition (vec_info *vinfo,
> bias = build_int_cst (intQI_type_node, 0);
> }
> }
> - if (masks)
> + if (fully_masked_p)
> {
> tree loop_mask
> - = vect_get_loop_mask (loop_vinfo, gsi, masks, vec_num,
> - vectype, i);
> + = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
> tree tmp2 = make_ssa_name (vec_cmp_type);
> gassign *g
> = gimple_build_assign (tmp2, BIT_AND_EXPR, vec_compare,
> @@ -13013,8 +12964,6 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, stmt_vec_info stmt_info,
> machine_mode mode = TYPE_MODE (vectype);
> int vec_num = vect_get_num_copies (loop_vinfo, slp_node);
>
> - vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
> - vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo);
> bool masked_loop_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
> bool len_loop_p = LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo);
>
> @@ -13088,9 +13037,9 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, stmt_vec_info stmt_info,
> {
> if (direct_internal_fn_supported_p (IFN_VCOND_MASK_LEN, vectype,
> OPTIMIZE_FOR_SPEED))
> - vect_record_loop_len (loop_vinfo, lens, vec_num, vectype, 1);
> + vect_record_len (loop_vinfo, vec_num, vectype, 1);
> else
> - vect_record_loop_mask (loop_vinfo, masks, vec_num, vectype, NULL);
> + vect_record_mask (loop_vinfo, vec_num, vectype, NULL_TREE);
> }
>
> if (!vect_compute_type_for_early_break_scalar_iv (loop_vinfo))
> @@ -13158,19 +13107,17 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, stmt_vec_info stmt_info,
> for (unsigned i = 0; i < stmts.length (); i++)
> {
> tree stmt_mask
> - = vect_get_loop_mask (loop_vinfo, gsi, masks, vec_num,
> - vectype, i);
> - stmt_mask
> - = prepare_vec_mask (loop_vinfo, TREE_TYPE (stmt_mask), stmt_mask,
> - stmts[i], &cond_gsi);
> + = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
> + stmt_mask = prepare_vec_mask (loop_vinfo, TREE_TYPE (stmt_mask),
> + stmt_mask, stmts[i], &cond_gsi);
> workset.quick_push (stmt_mask);
> }
> else if (len_loop_p)
> for (unsigned i = 0; i < stmts.length (); i++)
> {
> - tree len_mask = vect_gen_loop_len_mask (loop_vinfo, gsi, &cond_gsi,
> - lens, vec_num,
> - vectype, stmts[i], i, 1);
> + tree len_mask
> + = vect_gen_loop_len_mask (loop_vinfo, gsi, &cond_gsi, vec_num,
> + vectype, stmts[i], i, 1);
>
> workset.quick_push (len_mask);
> }
> @@ -13205,14 +13152,13 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, stmt_vec_info stmt_info,
> new_temp = stmts[0];
> if (masked_loop_p)
> {
> - tree mask
> - = vect_get_loop_mask (loop_vinfo, gsi, masks, 1, vectype, 0);
> + tree mask = vect_get_mask (loop_vinfo, gsi, 1, vectype, 0);
> new_temp = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask,
> new_temp, &cond_gsi);
> }
> else if (len_loop_p)
> - new_temp = vect_gen_loop_len_mask (loop_vinfo, gsi, &cond_gsi, lens,
> - 1, vectype, new_temp, 0, 1);
> + new_temp = vect_gen_loop_len_mask (loop_vinfo, gsi, &cond_gsi, 1,
> + vectype, new_temp, 0, 1);
> }
>
> gcc_assert (new_temp);
> diff --git a/gcc/tree-vectorizer.cc b/gcc/tree-vectorizer.cc
> index 8c824e4ee33..87204ba54dc 100644
> --- a/gcc/tree-vectorizer.cc
> +++ b/gcc/tree-vectorizer.cc
> @@ -470,7 +470,12 @@ vec_info::vec_info (vec_info::vec_kind kind_in, vec_info_shared *shared_)
> stmt_vec_info_ro (false),
> bbs (NULL),
> nbbs (0),
> - inv_pattern_def_seq (NULL)
> + inv_pattern_def_seq (NULL),
> + vectorization_factor (1),
> + rgroup_compare_type (NULL_TREE),
> + rgroup_iv_type (NULL_TREE),
> + partial_vector_style (vect_partial_vectors_none),
> + partial_load_store_bias (0)
> {
> stmt_vec_infos.create (50);
> }
> diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
> index 94ef81383f4..29ce87e1fb0 100644
> --- a/gcc/tree-vectorizer.h
> +++ b/gcc/tree-vectorizer.h
> @@ -499,7 +499,7 @@ enum vect_partial_vector_style {
>
> /* Key for map that records association between
> scalar conditions and corresponding loop mask, and
> - is populated by vect_record_loop_mask. */
> + is populated by vect_record_mask. */
>
> struct scalar_cond_masked_key
> {
> @@ -618,104 +618,6 @@ public:
> vec<ddr_p> ddrs;
> };
>
> -/* Vectorizer state common between loop and basic-block vectorization. */
> -class vec_info {
> -public:
> - typedef hash_set<int_hash<machine_mode, E_VOIDmode, E_BLKmode> > mode_set;
> - enum vec_kind { bb, loop };
> -
> - vec_info (vec_kind, vec_info_shared *);
> - ~vec_info ();
> -
> - stmt_vec_info add_stmt (gimple *);
> - stmt_vec_info add_pattern_stmt (gimple *, stmt_vec_info);
> - stmt_vec_info resync_stmt_addr (gimple *);
> - stmt_vec_info lookup_stmt (gimple *);
> - stmt_vec_info lookup_def (tree);
> - stmt_vec_info lookup_single_use (tree);
> - class dr_vec_info *lookup_dr (data_reference *);
> - void move_dr (stmt_vec_info, stmt_vec_info);
> - void remove_stmt (stmt_vec_info);
> - void replace_stmt (gimple_stmt_iterator *, stmt_vec_info, gimple *);
> - void insert_on_entry (stmt_vec_info, gimple *);
> - void insert_seq_on_entry (stmt_vec_info, gimple_seq);
> -
> - /* The type of vectorization. */
> - vec_kind kind;
> -
> - /* Shared vectorizer state. */
> - vec_info_shared *shared;
> -
> - /* The mapping of GIMPLE UID to stmt_vec_info. */
> - vec<stmt_vec_info> stmt_vec_infos;
> - /* Whether the above mapping is complete. */
> - bool stmt_vec_info_ro;
> -
> - /* Whether we've done a transform we think OK to not update virtual
> - SSA form. */
> - bool any_known_not_updated_vssa;
> -
> - /* The SLP graph. */
> - auto_vec<slp_instance> slp_instances;
> -
> - /* Maps base addresses to an innermost_loop_behavior that gives the maximum
> - known alignment for that base. */
> - vec_base_alignments base_alignments;
> -
> - /* All interleaving chains of stores, represented by the first
> - stmt in the chain. */
> - auto_vec<stmt_vec_info> grouped_stores;
> -
> - /* The set of vector modes used in the vectorized region. */
> - mode_set used_vector_modes;
> -
> - /* The argument we should pass to related_vector_mode when looking up
> - the vector mode for a scalar mode, or VOIDmode if we haven't yet
> - made any decisions about which vector modes to use. */
> - machine_mode vector_mode;
> -
> - /* The basic blocks in the vectorization region. For _loop_vec_info,
> - the memory is internally managed, while for _bb_vec_info, it points
> - to element space of an external auto_vec<>. This inconsistency is
> - not a good class design pattern. TODO: improve it with an unified
> - auto_vec<> whose lifetime is confined to vec_info object. */
> - basic_block *bbs;
> -
> - /* The count of the basic blocks in the vectorization region. */
> - unsigned int nbbs;
> -
> - /* Used to keep a sequence of def stmts of a pattern stmt that are loop
> - invariant if they exists.
> - The sequence is emitted in the loop preheader should the loop be vectorized
> - and are reset when undoing patterns. */
> - gimple_seq inv_pattern_def_seq;
> -
> -private:
> - stmt_vec_info new_stmt_vec_info (gimple *stmt);
> - void set_vinfo_for_stmt (gimple *, stmt_vec_info, bool = true);
> - void free_stmt_vec_infos ();
> - void free_stmt_vec_info (stmt_vec_info);
> -};
> -
> -class _loop_vec_info;
> -class _bb_vec_info;
> -
> -template<>
> -template<>
> -inline bool
> -is_a_helper <_loop_vec_info *>::test (vec_info *i)
> -{
> - return i->kind == vec_info::loop;
> -}
> -
> -template<>
> -template<>
> -inline bool
> -is_a_helper <_bb_vec_info *>::test (vec_info *i)
> -{
> - return i->kind == vec_info::bb;
> -}
> -
> /* In general, we can divide the vector statements in a vectorized loop
> into related groups ("rgroups") and say that for each rgroup there is
> some nS such that the rgroup operates on nS values from one scalar
> @@ -831,7 +733,7 @@ struct rgroup_controls {
> tree bias_adjusted_ctrl;
> };
>
> -struct vec_loop_masks
> +struct vec_masks
> {
> bool is_empty () const { return mask_set.is_empty (); }
>
> @@ -843,7 +745,146 @@ struct vec_loop_masks
> auto_vec<rgroup_controls> rgc_vec;
> };
>
> -typedef auto_vec<rgroup_controls> vec_loop_lens;
> +typedef auto_vec<rgroup_controls> vec_lens;
> +
> +/* Vectorizer state common between loop and basic-block vectorization. */
> +class vec_info {
> +public:
> + typedef hash_set<int_hash<machine_mode, E_VOIDmode, E_BLKmode> > mode_set;
> + enum vec_kind { bb, loop };
> +
> + vec_info (vec_kind, vec_info_shared *);
> + ~vec_info ();
> +
> + stmt_vec_info add_stmt (gimple *);
> + stmt_vec_info add_pattern_stmt (gimple *, stmt_vec_info);
> + stmt_vec_info resync_stmt_addr (gimple *);
> + stmt_vec_info lookup_stmt (gimple *);
> + stmt_vec_info lookup_def (tree);
> + stmt_vec_info lookup_single_use (tree);
> + class dr_vec_info *lookup_dr (data_reference *);
> + void move_dr (stmt_vec_info, stmt_vec_info);
> + void remove_stmt (stmt_vec_info);
> + void replace_stmt (gimple_stmt_iterator *, stmt_vec_info, gimple *);
> + void insert_on_entry (stmt_vec_info, gimple *);
> + void insert_seq_on_entry (stmt_vec_info, gimple_seq);
> +
> + /* The type of vectorization. */
> + vec_kind kind;
> +
> + /* Shared vectorizer state. */
> + vec_info_shared *shared;
> +
> + /* The mapping of GIMPLE UID to stmt_vec_info. */
> + vec<stmt_vec_info> stmt_vec_infos;
> + /* Whether the above mapping is complete. */
> + bool stmt_vec_info_ro;
> +
> + /* Whether we've done a transform we think OK to not update virtual
> + SSA form. */
> + bool any_known_not_updated_vssa;
> +
> + /* The SLP graph. */
> + auto_vec<slp_instance> slp_instances;
> +
> + /* Maps base addresses to an innermost_loop_behavior that gives the maximum
> + known alignment for that base. */
> + vec_base_alignments base_alignments;
> +
> + /* All interleaving chains of stores, represented by the first
> + stmt in the chain. */
> + auto_vec<stmt_vec_info> grouped_stores;
> +
> + /* The set of vector modes used in the vectorized region. */
> + mode_set used_vector_modes;
> +
> + /* The argument we should pass to related_vector_mode when looking up
> + the vector mode for a scalar mode, or VOIDmode if we haven't yet
> + made any decisions about which vector modes to use. */
> + machine_mode vector_mode;
> +
> + /* The basic blocks in the vectorization region. For _loop_vec_info,
> + the memory is internally managed, while for _bb_vec_info, it points
> + to element space of an external auto_vec<>. This inconsistency is
> + not a good class design pattern. TODO: improve it with an unified
> + auto_vec<> whose lifetime is confined to vec_info object. */
> + basic_block *bbs;
> +
> + /* The count of the basic blocks in the vectorization region. */
> + unsigned int nbbs;
> +
> + /* Used to keep a sequence of def stmts of a pattern stmt that are loop
> + invariant if they exists.
> + The sequence is emitted in the loop preheader should the loop be vectorized
> + and are reset when undoing patterns. */
> + gimple_seq inv_pattern_def_seq;
> +
> + /* Set of vector conditions that have loop mask applied. */
> + vec_cond_masked_set_type vec_cond_masked_set;
> +
> + /* The masks that a vectorization region should use to avoid operating
> + on inactive scalars. */
> + vec_masks masks;
> +
> + /* The lengths that a vectorization region should use to avoid operating
> + on inactive scalars. */
> + vec_lens lens;
> +
> + /* Unrolling factor. In case of suitable super-word parallelism
> + it can be that no unrolling is needed, and thus this is 1. */
> + poly_uint64 vectorization_factor;
> +
> + /* The type that the loop control IV should be converted to before
> + testing which of the VF scalars are active and inactive.
> + For _loop_vec_info, this is only meaningful if
> + LOOP_VINFO_USING_PARTIAL_VECTORS_P is true.
> + For _bb_vec_info, this is the type of the active lane count.
> + */
> + tree rgroup_compare_type;
> +
> + /* The type that the vector loop control IV should have.
> + For _loop_vec_info, this is only meaningful if
> + LOOP_VINFO_USING_PARTIAL_VECTORS_P is true.
> + For _bb_vec_info, this is not actually the type of a
> + vector loop control IV but facilitates code reuse.
> + */
> + tree rgroup_iv_type;
> +
> + /* The style used for implementing partial vectors.
> + For _loop_vec_info, this is only meaningful if
> + LOOP_VINFO_USING_PARTIAL_VECTORS_P is true. */
> + vect_partial_vector_style partial_vector_style;
> +
> + /* The bias for len_load and len_store. For now, only 0 and -1 are
> + supported. -1 must be used when a backend does not support
> + len_load/len_store with a length of zero. */
> + signed char partial_load_store_bias;
> +
> +private:
> + stmt_vec_info new_stmt_vec_info (gimple *stmt);
> + void set_vinfo_for_stmt (gimple *, stmt_vec_info, bool = true);
> + void free_stmt_vec_infos ();
> + void free_stmt_vec_info (stmt_vec_info);
> +};
> +
> +class _loop_vec_info;
> +class _bb_vec_info;
> +
> +template<>
> +template<>
> +inline bool
> +is_a_helper <_loop_vec_info *>::test (vec_info *i)
> +{
> + return i->kind == vec_info::loop;
> +}
> +
> +template<>
> +template<>
> +inline bool
> +is_a_helper <_bb_vec_info *>::test (vec_info *i)
> +{
> + return i->kind == vec_info::bb;
> +}
>
> typedef auto_vec<std::pair<data_reference*, tree> > drs_init_vec;
>
> @@ -975,10 +1016,6 @@ public:
> used. */
> poly_uint64 versioning_threshold;
>
> - /* Unrolling factor. In case of suitable super-word parallelism
> - it can be that no unrolling is needed, and thus this is 1. */
> - poly_uint64 vectorization_factor;
> -
> /* Gimple operand for the number of scalar iteration handed per loop
> iteration, and therefore how much to increment each IV by. */
> tree iv_increment;
> @@ -1018,20 +1055,9 @@ public:
> if there is no particular limit. */
> unsigned HOST_WIDE_INT max_vectorization_factor;
>
> - /* The masks that a fully-masked loop should use to avoid operating
> - on inactive scalars. */
> - vec_loop_masks masks;
> -
> - /* The lengths that a loop with length should use to avoid operating
> - on inactive scalars. */
> - vec_loop_lens lens;
> -
> /* Set of scalar conditions that have loop mask applied. */
> scalar_cond_masked_set_type scalar_cond_masked_set;
>
> - /* Set of vector conditions that have loop mask applied. */
> - vec_cond_masked_set_type vec_cond_masked_set;
> -
> /* If we are using a loop mask to align memory addresses, this variable
> contains the number of vector elements that we should skip in the
> first iteration of the vector loop (i.e. the number of leading
> @@ -1043,11 +1069,6 @@ public:
> were skipped during the initial iteration of the loop. */
> tree mask_skip_niters_pfa_offset;
>
> - /* The type that the loop control IV should be converted to before
> - testing which of the VF scalars are active and inactive.
> - Only meaningful if LOOP_VINFO_USING_PARTIAL_VECTORS_P. */
> - tree rgroup_compare_type;
> -
> /* For #pragma omp simd if (x) loops the x expression. If constant 0,
> the loop should not be vectorized, if constant non-zero, simd_if_cond
> shouldn't be set and loop vectorized normally, if SSA_NAME, the loop
> @@ -1055,14 +1076,6 @@ public:
> is false and vectorized loop otherwise. */
> tree simd_if_cond;
>
> - /* The type that the vector loop control IV should have when
> - LOOP_VINFO_USING_PARTIAL_VECTORS_P is true. */
> - tree rgroup_iv_type;
> -
> - /* The style used for implementing partial vectors when
> - LOOP_VINFO_USING_PARTIAL_VECTORS_P is true. */
> - vect_partial_vector_style partial_vector_style;
> -
> /* Unknown DRs according to which loop was peeled. */
> class dr_vec_info *unaligned_dr;
>
> @@ -1164,11 +1177,6 @@ public:
> without peeling. */
> bool allow_mutual_alignment;
>
> - /* The bias for len_load and len_store. For now, only 0 and -1 are
> - supported. -1 must be used when a backend does not support
> - len_load/len_store with a length of zero. */
> - signed char partial_load_store_bias;
> -
> /* When we have grouped data accesses with gaps, we may introduce invalid
> memory accesses. We peel the last iteration of the loop to prevent
> this. */
> @@ -2695,20 +2703,18 @@ extern void vect_gen_vector_loop_niters (loop_vec_info, tree, tree *,
> extern tree vect_get_loop_iv_increment (loop_vec_info);
> extern tree vect_halve_mask_nunits (tree, machine_mode);
> extern tree vect_double_mask_nunits (tree, machine_mode);
> -extern void vect_record_loop_mask (loop_vec_info, vec_loop_masks *,
> - unsigned int, tree, tree);
> -extern tree vect_get_loop_mask (loop_vec_info, gimple_stmt_iterator *,
> - vec_loop_masks *,
> - unsigned int, tree, unsigned int);
> -extern void vect_record_loop_len (loop_vec_info, vec_loop_lens *, unsigned int,
> - tree, unsigned int);
> -extern tree vect_get_loop_len (loop_vec_info, gimple_stmt_iterator *,
> - vec_loop_lens *, unsigned int, tree,
> - unsigned int, unsigned int, bool);
> +extern void vect_record_mask (vec_info *, unsigned int, tree, tree,
> + slp_tree = nullptr);
> +extern tree vect_get_mask (vec_info *, gimple_stmt_iterator *, unsigned int,
> + tree, unsigned int, slp_tree = nullptr);
> +extern void vect_record_len (vec_info *, unsigned int, tree, unsigned int,
> + slp_tree = nullptr);
> +extern tree vect_get_len (vec_info *, gimple_stmt_iterator *, unsigned int,
> + tree, unsigned int, unsigned int, bool,
> + slp_tree = nullptr);
> extern tree vect_gen_loop_len_mask (loop_vec_info, gimple_stmt_iterator *,
> - gimple_stmt_iterator *, vec_loop_lens *,
> - unsigned int, tree, tree, unsigned int,
> - unsigned int);
> + gimple_stmt_iterator *, unsigned int, tree,
> + tree, unsigned int, unsigned int);
> extern gimple_seq vect_gen_len (tree, tree, tree, tree);
> extern vect_reduc_info info_for_reduction (loop_vec_info, slp_tree);
> extern bool reduction_fn_for_scalar_code (code_helper, internal_fn *);
> @@ -2801,7 +2807,7 @@ extern void vect_free_slp_tree (slp_tree);
> extern bool compatible_calls_p (gcall *, gcall *, bool);
> extern int vect_slp_child_index_for_operand (const stmt_vec_info, int op);
>
> -extern tree prepare_vec_mask (loop_vec_info, tree, tree, tree,
> +extern tree prepare_vec_mask (vec_info *, tree, tree, tree,
> gimple_stmt_iterator *);
> extern tree vect_get_mask_load_else (int, tree);
> extern bool vect_load_perm_consecutive_p (slp_tree, unsigned = UINT_MAX);