[PATCH] x86emul: V{,P}{COMPRESS,EXPAND}* can (wrongly) trigger assertion

Jan Beulich <[email protected]>
Newsgroups org.xenproject.lists.xen-devel
Message-ID <[email protected]>
AFL has pointed out that the op_bytes-is-not-0 assertion in common SIMD
handling can trigger for these insns. Indeed when the (relevant part of)
the controlling mask register is 0, no memory is accessed at all. Leave
op_bytes unaltered in this case, to engage the short-circuiting in common
SIMD handling when fault_suppression is true and op_bytes is 0.

While there also correct a related typo in the test harness.

Fixes: 65f82d4ce1ea ("x86emul: support AVX512{F,_VBMI2} compress/expand insns")
Signed-off-by: Jan Beulich <[email protected]>
---
In a release build, due to op_mask being 0 when n is 0, the short-circuit
mentioned will prevent any damage.

--- a/tools/tests/x86_emulator/predicates.c
+++ b/tools/tests/x86_emulator/predicates.c
@@ -1946,7 +1946,7 @@ static const struct evex {
     { { 0x83 }, 2, T, R, pfx_66, W1, Ln }, /* vpmultishiftqb */
     { { 0x88 }, 2, T, R, pfx_66, Wn, Ln }, /* vpexpandp{s,d} */
     { { 0x89 }, 2, T, R, pfx_66, Wn, Ln }, /* vpexpand{d,q} */
-    { { 0x8a }, 2, T, W, pfx_66, Wn, Ln }, /* vpcompressp{s,d} */
+    { { 0x8a }, 2, T, W, pfx_66, Wn, Ln }, /* vcompressp{s,d} */
     { { 0x8b }, 2, T, W, pfx_66, Wn, Ln }, /* vpcompress{d,q} */
     { { 0x8d }, 2, F, R, pfx_66, Wn, Ln }, /* vperm{b,w} */
     { { 0x8f }, 2, F, R, pfx_66, W0, Ln }, /* vpshufbitqmb */
--- a/xen/arch/x86/x86_emulate/x86_emulate.c
+++ b/xen/arch/x86/x86_emulate/x86_emulate.c
@@ -6236,9 +6236,11 @@ x86_emulate(
             ASSERT(op_bytes == n * elem_bytes);
             op_mask &= ~0ULL >> (64 - n);
             n = hweight64(op_mask);
-            op_bytes = n * elem_bytes;
             if ( n )
+            {
+                op_bytes = n * elem_bytes;
                 op_mask = ~0ULL >> (64 - n);
+            }
         }
         goto simd_zmm;
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.