[Bug tree-optimization/126789] Bogus bool vect_recog_mask_conversion_pattern with AVX512 style masks

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=126789

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned at gcc dot gnu.org      |rguenth at gcc dot gnu.org
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2026-08-11
             Status|UNCONFIRMED                 |ASSIGNED

--- Comment #2 from Richard Biener <rguenth at gcc dot gnu.org> ---
OK, so the odd vector type comes from get_mask_type_for_scalar_type
which expects a scalar, not a mask type and goes via determining
vector(4) unsigned char as vectype for bool:1 and a vector(4) bool:8
as corresponding mask type (because the backend says so, for V4QI).

That's vect_get_vector_types_for_stmt doing

  else if (vect_use_mask_type_p (stmt_info))
    {
      unsigned int precision = stmt_info->mask_precision;
      scalar_type = build_nonstandard_integer_type (precision, 1);
      vectype = get_mask_type_for_scalar_type (vinfo, scalar_type, group_size);

I have a fix.