[Bug target/126429] New: [i386] _mm512_{mask,maskz}_dpbf16_ps only honor the low 8 bits of the __mmask16 (kmovb instead of kmovw)

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=126429

            Bug ID: 126429
           Summary: [i386] _mm512_{mask,maskz}_dpbf16_ps only honor the
                    low 8 bits of the __mmask16 (kmovb instead of kmovw)
           Product: gcc
           Version: 16.1.0
            Status: UNCONFIRMED
          Keywords: wrong-code
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: michael.crusoe at gmail dot com
  Target Milestone: ---
            Target: x86_64--

The below report is written by https://github.com/fo40225 (from
https://github.com/simd-everywhere/simde/pull/1419#issuecomment-5078585078 )
and I affirm that their workaround fixes a real problem I've seen as the SIMDe
maintainer.

---

The AVX512-BF16 masked dot-product intrinsics _mm512_mask_dpbf16_ps and
_mm512_maskz_dpbf16_ps take a __mmask16 -- the zmm destination holds 16
float32 lanes -- but the writemask operand of the underlying insn
patterns is QImode, so the upper 8 mask bits are dropped.  Lanes 8-15
always behave as if their mask bit were clear: the merge form copies src
through and the zeroing form zeroes the lane, whatever the mask says.

The visible symptom is a kmovb where kmovw is required.

*** Please enable AVX512DQ when reproducing. ***

kmovb is an AVX512DQ instruction, so the QImode truncation only becomes
architecturally observable once AVX512DQ is on.  Under a bare
-mavx512bf16 the QImode mask move is lowered to kmovw, which transfers
all 16 bits and hides the defect.  All ten -march= values that imply
AVX512BF16 in GCC 16.1 also imply AVX512DQ -- cooperlake,
sapphirerapids, emeraldrapids, graniterapids, graniterapids-d,
diamondrapids, novalake, znver4, znver5, znver6 -- and every one of them
produces the kmovb.  The wrong code is therefore what real builds get;
-mavx512bf16 on its own is the only configuration that escapes it.

This is a silent wrong-code bug present in every GCC release with
AVX512BF16 support.  Reproduced on the official docker images 10.5.0,
11.5.0, 12.5.0, 13.4.0, 14.4.0, 15.3.0 and 16.1.0; on 16.1.0 the kmovb
comes out for all ten -march= values above at each of -O0, -O1, -O2, -O3
and -Os.  The three operands responsible are unchanged on trunk
(f6b00aefc25a).  Clang emits kmovd and is correct (checked 18.1.3 and
20.1.2).

It is not latent in practice: GitHub Actions' ubuntu-24.04 runner fleet
now contains AVX512-BF16-capable machines (AMD EPYC 9V74 / Zen4, Intel
Emerald Rapids 8573C, Granite Rapids 6973P-C), so any -march=native
build that lands on one of them silently computes wrong results.  It was
found through intermittent CI failures in SIMDe.

https://github.com/simd-everywhere/simde/issues/902
https://github.com/simd-everywhere/simde/issues/988
https://github.com/simd-everywhere/simde/issues/1095
https://github.com/simd-everywhere/simde/issues/1201
https://github.com/simd-everywhere/simde/pull/1276

=== Reproducer (compile only) ===

$ cat asm.c
#include <immintrin.h>
__m512 f_mask(__m512 src, __mmask16 k, __m512bh a, __m512bh b) {
  return _mm512_mask_dpbf16_ps(src, k, a, b);
}
__m512 f_maskz(__mmask16 k, __m512 src, __m512bh a, __m512bh b) {
  return _mm512_maskz_dpbf16_ps(k, src, a, b);
}

$ gcc-16 -O2 -mavx512bf16 -mavx512dq -S -o - asm.c
f_mask:
        kmovb   %edi, %k1                        <-- should be kmovw
        vdpbf16ps       %zmm2, %zmm1, %zmm0{%k1}
        ret
f_maskz:
        kmovb   %edi, %k1                        <-- should be kmovw
        vdpbf16ps       %zmm2, %zmm1, %zmm0{%k1}{z}
        ret

-march=sapphirerapids / znver4 / cooperlake / ... give the same kmovb.
clang emits kmovd for all of them.

=== Reproducer (runtime, wrong-code) ===

$ cat runtime.c
#include <immintrin.h>
#include <stdio.h>
#include <string.h>

volatile unsigned short k_mask  = 0xFFFF;  /* opaque so the mask is not folded
*/
volatile unsigned short k_maskz = 0xFF00;

int main(void) {
  float srcf[16], af[16], bf[16], r[16];
  for (int i = 0; i < 16; i++) { srcf[i] = 1000.0f + i; af[i] = 2.0f; bf[i] =
3.0f; }
  /* each f32 lane of a/b holds bf16 pair {lo=0, hi=2.0/3.0} -> dot adds 6.0 */
  __m512 src = _mm512_loadu_ps(srcf);
  __m512bh a, b;
  memcpy(&a, af, 64); memcpy(&b, bf, 64);
  int bad = 0;

  _mm512_storeu_ps(r, _mm512_mask_dpbf16_ps(src, (__mmask16)k_mask, a, b));
  for (int i = 8; i < 16; i++)      /* k=0xFFFF: expect src+6 everywhere */
    if (r[i] != srcf[i] + 6.0f) { bad++; printf("mask  lane %2d: got %g want
%g\n", i, r[i], srcf[i]+6.0f); }

  _mm512_storeu_ps(r, _mm512_maskz_dpbf16_ps((__mmask16)k_maskz, src, a, b));
  for (int i = 8; i < 16; i++)      /* k=0xFF00: expect src+6 in lanes 8-15 */
    if (r[i] != srcf[i] + 6.0f) { bad++; printf("maskz lane %2d: got %g want
%g\n", i, r[i], srcf[i]+6.0f); }

  printf(bad ? "FAIL: %d wrong lanes\n" : "OK\n", bad);
  return bad != 0;
}

$ gcc-16 -O2 -march=native runtime.c && ./a.out   # on avx512_bf16 hardware
mask  lane  8: got 1008 want 1014     (src passthrough: mask bit read as 0)
...
maskz lane  8: got 0 want 1014        (zeroed: mask bit read as 0)
...
FAIL: 16 wrong lanes

All 16 lanes are wrong under Intel SDE 10.8.0 -gnr, and identically so
for -march=graniterapids, and on an AMD EPYC 9474F (Zen4) running -march=znver4
on real silicon. The same binary built with a bare -mavx512bf16 (the kmovw
form)
prints OK, so the defect is in the generated code, not in any CPU or emulator.

=== Root cause ===

(Line numbers from trunk, commit f6b00aefc25a.)

All three masked dpbf16 patterns carry the writemask in the half-width
mask mode <avx512fmaskhalfmode>:

1. gcc/config/i386/sse.md:32651, define_insn
   "avx512f_dpbf16ps_<mode>_mask":

     (vec_merge:VF1_AVX512VL
       (unspec:VF1_AVX512VL [...] UNSPEC_VDPBF16PS)
       (match_dup 1)
       (match_operand:<avx512fmaskhalfmode> 4 "register_operand" "Yk"))

2. gcc/config/i386/sse.md:32628, define_expand
   "avx512f_dpbf16ps_<mode>_maskz": operand 4 is
   (match_operand:<avx512fmaskhalfmode> 4 "register_operand").

3. gcc/config/i386/sse.md:32641, define_insn
   "avx512f_dpbf16ps_<mode><maskz_half_name>": the maskz variant is
   generated by the dedicated define_subst "maskz_half"
   (gcc/config/i386/subst.md:473-484), whose mask operand is likewise
   (match_operand:<avx512fmaskhalfmode> 3 "register_operand" "Yk").

For V16SF <avx512fmaskhalfmode> is QImode, so only 8 mask bits survive.
The writemask governs the 16 float32 *output* lanes, so it must be
<avx512fmaskmode> (HImode for V16SF); "<avx512fmaskhalfmode>" appears to
have been chosen because the BF16 *inputs* have 32 elements, but
per-input-element masking does not exist for this instruction.

The builtin prototypes in i386-builtin.def already use UHI correctly
(i386-builtin.def:2874-2875, V16SF_FTYPE_V16SF_V32BF_V32BF_UHI); only
the insn operand mode is wrong.  Comparing the two mode attributes:

  avx512fmaskmode     (sse.md:1087):  V16SF -> HI,  V8SF -> QI,  V4SF -> QI
  avx512fmaskhalfmode (sse.md:1113):  V16SF -> QI,  V8SF -> QI,  V4SF -> QI

For V4SF/V8SF the two attributes coincide (QImode covers the 4/8 output
lanes), so the 128/256-bit variants are accidentally correct; only the
V16SF (512-bit) case misbehaves.

Apart from its definition, those three operands are the only uses of
<avx512fmaskhalfmode> anywhere in the i386 backend's machine
descriptions, and dpbf16 is the only user of the "maskz_half" subst.

Suggested fix: switch the writemask operands to <avx512fmaskmode> in all
three places -- the _mask insn, the _maskz expander, and the "maskz_half"
subst (which is dpbf16-only, so it can be corrected in place or the
pattern moved to the standard "maskz" subst machinery) -- plus a
wrong-code testcase like the runtime reproducer above.  Once that is
done, both the "maskz_half" subst and the "avx512fmaskhalfmode" mode
attribute become entirely unused and can be removed.