[Bug tree-optimization/126789] New: Bogus bool vect_recog_mask_conversion_pattern with AVX512 style masks

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=126789

            Bug ID: 126789
           Summary: Bogus bool vect_recog_mask_conversion_pattern with
                    AVX512 style masks
           Product: gcc
           Version: 17.0
            Status: UNCONFIRMED
          Keywords: missed-optimization
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: rguenth at gcc dot gnu.org
            Blocks: 53947
  Target Milestone: ---

int foo (double g, int f, double *r, int *s)
{
  int hu = 0;
  bool test0 = r[0] < g;
  bool test1 = r[1] < g;
  bool test2 = r[2] < g;
  bool test3 = r[3] < g;
  bool test4 = s[0] < f;
  bool test5 = s[1] < f;
  bool test6 = s[2] < f;
  bool test7 = s[3] < f;
  hu += (test0 & test4) + (test1 & test5) + (test2 & test6) + (test3 & test7);
  return hu;
}

shows (with -O2 -march=znver5):

t3.c:12:61: note:   node 0x21d85f70 (max_nunits=4, refcnt=2) vector(4)
<signed-boolean:8>
t3.c:12:61: note:   op template: patt_42 = (<signed-boolean:1>) test3_23;
t3.c:12:61: note:       stmt 0 patt_42 = (<signed-boolean:1>) test3_23;
t3.c:12:61: note:       stmt 1 patt_39 = (<signed-boolean:1>) test2_22;
t3.c:12:61: note:       stmt 2 patt_33 = (<signed-boolean:1>) test0_20;
t3.c:12:61: note:       stmt 3 patt_36 = (<signed-boolean:1>) test1_21;
t3.c:12:61: note:       children 0x21d86028

...

t3.c:12:61: note:   ==> examining statement: patt_42 = (<signed-boolean:1>)
test3_23;
...
t3.c:1:5: missed:   not vectorized: relevant stmt not supported: patt_42 =
(<signed-boolean:1>) test3_23;

that's the attempt to perform mask unpacking during pattern recog, but that
falls short of BB SLP which does not use an uniform vector type.  Interestingly
enough it works fine for traditional masking where we also insert the
unpacking.  So somehow vect_recog_mask_conversion_pattern is confused by
mask types here.

On x86 the following woudl add the twist of mixing integer and vector masks
when AVX512 is enabled since MMX-with-SSE forces traditional masking:

int foo (double g, int f, double *r, int *s)
{
  int hu = 0;
  bool test0 = r[0] < g;
  bool test1 = r[1] < g;
  bool test2 = s[0] < f;
  bool test3 = s[1] < f;
  hu += (test0 & test2) + (test1 & test3);
  return hu;
}


Referenced Bugs:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53947
[Bug 53947] [meta-bug] vectorizer missed-optimizations