[x86 SSE PATCH] PR target/126619: Improve V4SF vector initialization.
"Roger Sayle" <[email protected]>
| Newsgroups | gmane.comp.gcc.patches |
|---|---|
| Message-ID | <[email protected]> |
This patch addresses PR target/126619, a performance regression caused by
my recent update of SSE vector initialization on x86_64. The new idiom
attempts to benefit from the implicit zero extension provided by movss
and movd, but this causes problems for (V4SF) cases were the (SFmode)
value is already in a register. Without newer extensions, GCC's approach
to zero extension (i.e. vec_init of (V4SF){ x, 0, 0, 0 }) is to perform
an interunit transfer to a general purpose integer register, and then
transfer the value back again. Inter-unit moves are expensive, especially
on older microarchitectures.
The problem is fixed in several ways. The first is to tweak register
preferencing in vec_set<mode>_0's define_insn, so that general registers
are only used if the source/destination is already an integer GPR.
This changes reload from generating (two instructions):
movd %xmm0, %eax
movd %eax, %xmm0
to instead using:
pxor %xmm1, %xmm1
movss %xmm0, %xmm1
movaps %xmm1, %xmm0
which requires 3 instructions, and 1 extra register, but requires
no inter-unit moves. This matches what clang/llvm does.
However, it's possible to do better, borrowing an idiom from
the middle-end's expansion of integer zero-extensions.
pslldq $12, %xmm0
psrldq $12, %xmm0
uses two instructions, and doesn't require an extra register.
Indeed, initializing the vector (V4SF){ 0, 0, 0, x } can be
done in a single instruction, as it doesn't require a "right"
shift.
Additionally, for cases such as (V4SF){ a, b, c, d }, where
there is no benefit from zero extension, we should continue
using GCC's original CONCAT of V2SF idiom, avoiding any overhead
of zero extension (Hongtao's suggestion in the Bugzilla PR).
Additionally, there are some additional V4SF initialization
tweaks. When loading from memory, where zero extension is
free "onevar_perm"s should construct { x, 0, 0, 0 } then
perform a shuffle using shufps, but when the source is a
register, it should construct { 0, 0, 0, x } (using the
single shift instruction described above), and perform a
modified shuffle using shufps from there.
With TARGET_SSE4_1, the first insertps can be used to clear
(initialize) all the other elements to zero, and the remaining
non-zero elements can be inserted with regular insertps.
As explained above, optimal code generation depends upon
knowing whether the source elements are in memory or in
registers. Currently this decision is made during RTL
expansion even though the final allocations/sources aren't
known until reload [CSE can convert a MEM to a REG, and
reload can spill a REG to a MEM]. Things work fine when the
tree-ssa optimizers correctly predict things well, but there
are one two cases than could still be improved (in either
combine or peephole2) where late changes are made to the
RTL.
This patch has been tested on x86_64-pc-linux-gnu with make bootstrap
and make -k check, both with and without --target_board=unix{-m32}
with no new failures. Ok for mainline?
2026-08-09 Roger Sayle <[email protected]>
Hongtao Liu <[email protected]>
gcc/ChangeLog
PR target/126619
(ix86_expand_vector_init_one_nonzero) <case E_V4SFmode>:
Improved initialization of one non-zero element V4SF vectors.
(ix86_expand_vector_init_v4sf): Reuse the above function
ix86_expand_vector_init_one_nonzero where possible. Various
improvements. Fall back to using (the original)
ix86_expand_vector_init_concat in the general case, when
SSE 4.1 instructions aren't available.
* config/i386/sse.md (vec_set<mode>_0): Prefer to avoid
inter-unit moves to general purpose registers in reload.
(vec_setv4sf_sse4_1): Remove asterisk to expose to i386-expand.
(sse4_1_insertps_<mode>_init): Variant of insertps that clears
all the other elements of the destination to zero.
(sse2_insertps_v4sf_3): Implementation of the above instruction
available on SSE2 by using the pslldq instruction.
gcc/testsuite/ChangeLog
PR target/126619
* gcc.target/i386/avx-init-v4sf-1.c: Update test case.
* gcc.target/i386/avx-init-v4sf-2.c: Likewise.
* gcc.target/i386/avx2-init-v4sf-1.c: Likewise.
* gcc.target/i386/sse-init-v4sf-2.c: Likewise.
* gcc.target/i386/sse-init-v4sf-3.c: Likewise.
* gcc.target/i386/sse2-init-v4sf-1.c: Likewise.
* gcc.target/i386/sse2-init-v4sf-2.c: Likewise.
* gcc.target/i386/sse4_1-init-v4sf-2.c: Likewise.
* gcc.target/i386/sse4_1-init-v4sf-3.c: Likewise.
Thanks in advance, and apologies for any inconvenience.
Roger
--
patchvi6.txt
(text/plain, 18.7 KB)
diff --git a/gcc/config/i386/i386-expand.cc b/gcc/config/i386/i386-expand.cc
index 6ddc402e9ad..d2873c044db 100644
--- a/gcc/config/i386/i386-expand.cc
+++ b/gcc/config/i386/i386-expand.cc
@@ -18089,21 +18089,45 @@ ix86_expand_vector_init_one_nonzero (bool mmx_ok, machine_mode mode,
}
return true;
case E_V4SFmode:
- var = force_reg (SFmode, var);
- x = gen_rtx_VEC_DUPLICATE (V4SFmode, var);
- x = gen_rtx_VEC_MERGE (V4SFmode, x, CONST0_RTX (V4SFmode), const1_rtx);
+ if (TARGET_SSE4_1)
+ {
+ if (!REG_P (var) && !MEM_P (var))
+ var = force_reg (SFmode, var);
+ if (one_var)
+ emit_insn (gen_sse4_1_insertps_v4sf_init (target, var,
+ CONST0_RTX (V4SFmode),
+ GEN_INT (1 << one_var)));
+ else
+ emit_insn (gen_vec_setv4sf_0 (target, CONST0_RTX (V4SFmode), var));
+ return true;
+ }
+ if (TARGET_SSE2 && REG_P (var))
+ {
+ if (one_var == 3)
+ emit_insn (gen_sse2_insertps_v4sf_3 (target, var,
+ CONST0_RTX (V4SFmode)));
+ else
+ {
+ rtx tmp = gen_reg_rtx (V4SFmode);
+ emit_insn (gen_sse2_insertps_v4sf_3 (tmp, var,
+ CONST0_RTX (V4SFmode)));
+ emit_insn (gen_vec_shr_v4sf (target, tmp,
+ GEN_INT ((3 - one_var) * 32)));
+ }
+ return true;
+ }
if (!one_var)
- emit_insn (gen_rtx_SET (target, x));
+ emit_insn (gen_vec_setv4sf_0 (target, CONST0_RTX (V4SFmode), var));
else if (TARGET_SSE2)
{
rtx tmp = gen_reg_rtx (V4SFmode);
- emit_insn (gen_rtx_SET (tmp, x));
+ emit_insn (gen_vec_setv4sf_0 (tmp, CONST0_RTX (V4SFmode), var));
emit_insn (gen_vec_shl_v4sf (target, tmp, GEN_INT (one_var * 32)));
}
else
{
rtx tmp = gen_reg_rtx (V4SFmode);
- emit_insn (gen_rtx_SET (tmp, x));
+ emit_insn (gen_vec_setv4sf_0 (tmp, CONST0_RTX (V4SFmode), var));
emit_insn (gen_sse_shufps_v4sf (target, tmp, tmp,
const1_rtx,
GEN_INT (one_var == 1 ? 0 : 1),
@@ -19205,12 +19229,7 @@ ix86_expand_vector_init_v4sf (rtx target, rtx *ops)
else if (ops[1] == CONST0_RTX (SFmode)
&& ops[2] == CONST0_RTX (SFmode)
&& ops[3] == CONST0_RTX (SFmode))
- {
- rtx val = ops[0];
- if (!REG_P (val) && !MEM_P (val))
- val = force_reg (SFmode, val);
- emit_insn (gen_vec_setv4sf_0 (target, CONST0_RTX (V4SFmode), val));
- }
+ ix86_expand_vector_init_one_nonzero (false, V4SFmode, target, ops[0], 0);
else if (rtx_equal_p (ops[0], ops[1])
&& rtx_equal_p (ops[0], ops[2])
&& rtx_equal_p (ops[0], ops[3]))
@@ -19228,33 +19247,100 @@ ix86_expand_vector_init_v4sf (rtx target, rtx *ops)
rtx vec = gen_rtx_CONST_VECTOR (V4SFmode, gen_rtvec_v (4, ops));
emit_move_insn (target, vec);
}
+ else if (ops[0] == CONST0_RTX (SFmode)
+ && ops[2] == CONST0_RTX (SFmode)
+ && ops[3] == CONST0_RTX (SFmode))
+ ix86_expand_vector_init_one_nonzero (false, V4SFmode, target, ops[1], 1);
+ else if (ops[0] == CONST0_RTX (SFmode)
+ && ops[1] == CONST0_RTX (SFmode)
+ && ops[3] == CONST0_RTX (SFmode))
+ ix86_expand_vector_init_one_nonzero (false, V4SFmode, target, ops[2], 2);
+ else if (ops[0] == CONST0_RTX (SFmode)
+ && ops[1] == CONST0_RTX (SFmode)
+ && ops[2] == CONST0_RTX (SFmode))
+ ix86_expand_vector_init_one_nonzero (false, V4SFmode, target, ops[3], 3);
else if (onevar_perm_p (ops, 4, perm, vars))
{
rtx tmp = gen_reg_rtx (V4SFmode);
- vars[1] = CONST0_RTX (SFmode);
- vars[2] = CONST0_RTX (SFmode);
- vars[3] = CONST0_RTX (SFmode);
- ix86_expand_vector_init_v4sf (tmp, vars);
- emit_insn (gen_sse_shufps_v4sf (target, tmp, tmp,
- GEN_INT (perm[0]),
- GEN_INT (perm[1]),
- GEN_INT (perm[2] + 4),
- GEN_INT (perm[3] + 4)));
+ if (TARGET_SSE2 && !TARGET_SSE4_1 && REG_P (ops[0]))
+ {
+ emit_insn (gen_sse2_insertps_v4sf_3 (tmp, ops[0],
+ CONST0_RTX (V4SFmode)));
+ emit_insn (gen_sse_shufps_v4sf (target, tmp, tmp,
+ GEN_INT (perm[0] ? 0 : 3),
+ GEN_INT (perm[1] ? 0 : 3),
+ GEN_INT (perm[2] ? 4 : 7),
+ GEN_INT (perm[3] ? 4 : 7)));
+ }
+ else
+ {
+ vars[1] = CONST0_RTX (SFmode);
+ vars[2] = CONST0_RTX (SFmode);
+ vars[3] = CONST0_RTX (SFmode);
+ ix86_expand_vector_init_v4sf (tmp, vars);
+ emit_insn (gen_sse_shufps_v4sf (target, tmp, tmp,
+ GEN_INT (perm[0]),
+ GEN_INT (perm[1]),
+ GEN_INT (perm[2] + 4),
+ GEN_INT (perm[3] + 4)));
+ }
}
else if (ops[2] == CONST0_RTX (SFmode)
&& ops[3] == CONST0_RTX (SFmode))
{
- rtx tmp1 = gen_reg_rtx (V4SFmode);
- vars[0] = ops[0];
- vars[1] = CONST0_RTX (SFmode);
- vars[2] = CONST0_RTX (SFmode);
- vars[3] = CONST0_RTX (SFmode);
- ix86_expand_vector_init_v4sf (tmp1, vars);
+ if (TARGET_SSE4_1)
+ {
+ rtx tmp = gen_reg_rtx (V4SFmode);
+ rtx val = ops[0];
+ if (!REG_P (val) && !MEM_P (val))
+ val = force_reg (SFmode, val);
+ emit_insn (gen_vec_setv4sf_0 (tmp, CONST0_RTX (V4SFmode), val));
+ val = ops[1];
+ if (!REG_P (val) && !MEM_P (val))
+ val = force_reg (SFmode, val);
+ emit_insn (gen_vec_setv4sf_sse4_1 (target, tmp, val, GEN_INT (2)));
+ }
+ else if (MEM_P (ops[0]) && MEM_P (ops[1]))
+ {
+ rtx tmp1 = gen_reg_rtx (V4SFmode);
+ vars[0] = ops[0];
+ vars[1] = CONST0_RTX (SFmode);
+ vars[2] = CONST0_RTX (SFmode);
+ vars[3] = CONST0_RTX (SFmode);
+ ix86_expand_vector_init_v4sf (tmp1, vars);
+ rtx tmp2 = gen_reg_rtx (V4SFmode);
+ vars[0] = ops[1];
+ ix86_expand_vector_init_v4sf (tmp2, vars);
+ emit_insn (gen_vec_interleave_lowv4sf (target, tmp1, tmp2));
+ }
+ else if (TARGET_SSE2)
+ {
+ rtx tmp = gen_reg_rtx (V2SFmode);
+ ix86_expand_vector_init_concat (V2SFmode, tmp, ops, 2);
+ tmp = gen_rtx_VEC_CONCAT (V4SFmode, tmp, CONST0_RTX (V2SFmode));
+ emit_insn (gen_rtx_SET (target, tmp));
+ }
+ else
+ {
+ rtx tmp1 = gen_reg_rtx (V2SFmode);
+ ix86_expand_vector_init_concat (V2SFmode, tmp1, ops, 2);
+ rtx tmp2 = gen_reg_rtx (V2SFmode);
+ emit_insn (gen_rtx_SET (tmp2, CONST0_RTX (V2SFmode)));
+ rtx tmp = gen_rtx_VEC_CONCAT (V4SFmode, tmp1, tmp2);
+ emit_insn (gen_rtx_SET (target, tmp));
+ }
+ }
+ else if (MEM_P (ops[0])
+ && ops[1] == CONST0_RTX (SFmode)
+ && MEM_P (ops[2])
+ && ops[3] == CONST0_RTX (SFmode))
+ {
+ rtx tmp1 = gen_reg_rtx (V4SFmode);
+ ix86_expand_vector_init_one_nonzero (false, V4SFmode, tmp1, ops[0], 0);
rtx tmp2 = gen_reg_rtx (V4SFmode);
- vars[0] = ops[1];
- ix86_expand_vector_init_v4sf (tmp2, vars);
- emit_insn (gen_vec_interleave_lowv4sf (target, tmp1, tmp2));
+ ix86_expand_vector_init_one_nonzero (false, V4SFmode, tmp2, ops[2], 0);
+ emit_insn (gen_sse_movlhps (target, tmp1, tmp2));
}
else if (twovar_perm_p (ops, 4, perm, vars))
{
@@ -19290,7 +19376,32 @@ ix86_expand_vector_init_v4sf (rtx target, rtx *ops)
emit_move_insn (tmp2, vec);
emit_insn (gen_rtx_SET (target, gen_rtx_IOR (V4SFmode, tmp1, tmp2)));
}
- else
+ else if (TARGET_SSE4_1)
+ {
+ int i;
+ rtx tmp = gen_reg_rtx (V4SFmode);
+ bool first_p = true;
+ for (i = 0; i < 4; i++)
+ if (ops[i] != CONST0_RTX (SFmode))
+ {
+ rtx idx = GEN_INT (1 << i);
+ rtx val = ops[i];
+ if (!REG_P (val) && !MEM_P (val))
+ val = force_reg (SFmode, val);
+ rtx pat = first_p
+ ? gen_sse4_1_insertps_v4sf_init (tmp, val,
+ CONST0_RTX (V4SFmode),
+ idx)
+ : gen_vec_setv4sf_sse4_1 (tmp, tmp, val, idx);
+ emit_insn (pat);
+ first_p = false;
+ }
+ emit_move_insn (target, tmp);
+ }
+ else if (MEM_P (ops[0])
+ && MEM_P (ops[1])
+ && MEM_P (ops[2])
+ && MEM_P (ops[3]))
{
rtx tmp1 = gen_reg_rtx (V4SFmode);
rtx tmp2 = gen_reg_rtx (V4SFmode);
@@ -19306,6 +19417,8 @@ ix86_expand_vector_init_v4sf (rtx target, rtx *ops)
const0_rtx, const1_rtx,
GEN_INT (4), GEN_INT (5)));
}
+ else
+ ix86_expand_vector_init_concat (V4SFmode, target, ops, 4);
}
/* A subroutine of ix86_expand_vector_init for V8HImode. */
diff --git a/gcc/config/i386/sse.md b/gcc/config/i386/sse.md
index 7d71f8ba546..473103fc927 100644
--- a/gcc/config/i386/sse.md
+++ b/gcc/config/i386/sse.md
@@ -12545,7 +12545,7 @@
;; see comment above inline_secondary_memory_needed function in i386.cc
(define_insn "vec_set<mode>_0"
[(set (match_operand:VI4F_128 0 "nonimmediate_operand"
- "=Yr,*x,v,v,v,v,x,x,v,Yr ,?x ,x ,m ,m ,m")
+ "=Yr,*x,v,v,v,!v,x,x,v,Yr ,?x ,x ,m ,m ,m")
(vec_merge:VI4F_128
(vec_duplicate:VI4F_128
(match_operand:<ssescalarmode> 2 "general_operand"
@@ -12807,7 +12807,7 @@
(set_attr "mode" "HF,HI")])
;; A subset is vec_setv4sf.
-(define_insn "*vec_setv4sf_sse4_1"
+(define_insn "vec_setv4sf_sse4_1"
[(set (match_operand:V4SF 0 "register_operand" "=Yr,*x,v")
(vec_merge:V4SF
(vec_duplicate:V4SF
@@ -12971,6 +12971,56 @@
(set_attr "prefix" "orig,maybe_evex")
(set_attr "mode" "V4SF")])
+;; Use sse4_1_insertps_v4s[if] to vector_init one non-zero value.
+(define_insn "sse4_1_insertps_<mode>_init"
+ [(set (match_operand:VI4F_128 0 "register_operand" "=Yr,*x,v")
+ (vec_merge:VI4F_128
+ (vec_duplicate:VI4F_128
+ (match_operand:<ssescalarmode> 1 "nonimmediate_operand" "Yrjm,*xjm,vm"))
+ (match_operand:VI4F_128 2 "const0_operand")
+ (match_operand:SI 3 "const248_operand")))]
+ "TARGET_SSE4_1"
+{
+ int op3 = INTVAL (operands[3]);
+ operands[3] = GEN_INT ((exact_log2 (op3) << 4) + (op3 ^ 15));
+ switch (which_alternative)
+ {
+ case 0:
+ case 1:
+ return "insertps\t{%3, %1, %0|%0, %1, %3}";
+ case 2:
+ return "vinsertps\t{%3, %1, %0, %0|%0, %0, %1, %3}";
+ default:
+ gcc_unreachable ();
+ }
+}
+ [(set_attr "isa" "noavx,noavx,avx")
+ (set_attr "type" "sselog")
+ (set_attr "addr" "gpr16,gpr16,*")
+ (set_attr "prefix_data16" "1,1,*")
+ (set_attr "prefix_extra" "1")
+ (set_attr "length_immediate" "1")
+ (set_attr "prefix" "orig,orig,maybe_evex")
+ (set_attr "mode" "V4SF")])
+
+;; Use SSE2's pslldq to vector_init v4sf one non-zero value at 3.
+(define_insn "sse2_insertps_v4sf_3"
+ [(set (match_operand:V4SF 0 "register_operand" "=x")
+ (vec_merge:V4SF
+ (vec_duplicate:V4SF
+ (match_operand:SF 1 "register_operand" "0"))
+ (match_operand:V4SF 2 "const0_operand")
+ (const_int 8)))]
+ "TARGET_SSE2 && !TARGET_SSE4_1"
+ "pslldq\t{$12, %0|%0, 12}"
+ [(set_attr "isa" "noavx")
+ (set_attr "type" "sseishft")
+ (set_attr "length_immediate" "1")
+ (set_attr "atom_unit" "sishuf")
+ (set_attr "prefix_data16" "1")
+ (set_attr "prefix" "orig")
+ (set_attr "mode" "V4SF")])
+
(define_split
[(set (match_operand:VI4F_128 0 "memory_operand")
(vec_merge:VI4F_128
diff --git a/gcc/testsuite/gcc.target/i386/avx-init-v4sf-1.c b/gcc/testsuite/gcc.target/i386/avx-init-v4sf-1.c
index a1f63c6a44a..962d487d051 100644
--- a/gcc/testsuite/gcc.target/i386/avx-init-v4sf-1.c
+++ b/gcc/testsuite/gcc.target/i386/avx-init-v4sf-1.c
@@ -25,9 +25,7 @@ v4sf fm0m0() { return (v4sf){m,0.0f,m,0.0f}; }
v4sf fmmmm() { return (v4sf){m,m,m,m}; }
/* { dg-final { scan-assembler-times "vinsertps" 16 } } */
-/* { dg-final { scan-assembler-times "vpslldq" 3 } } */
/* { dg-final { scan-assembler-times "vshufps" 3 } } */
-/* { dg-final { scan-assembler-times "vmovlhps" 4 } } */
-/* { dg-final { scan-assembler-times "vunpcklps" 5 } } */
+/* { dg-final { scan-assembler-times "vmovlhps" 3 } } */
/* { dg-final { scan-assembler-times "vmovss" 2 } } */
/* { dg-final { scan-assembler-times "vbroadcastss" 1 } } */
diff --git a/gcc/testsuite/gcc.target/i386/avx-init-v4sf-2.c b/gcc/testsuite/gcc.target/i386/avx-init-v4sf-2.c
index 6d3ff179d16..666d1b2fcc9 100644
--- a/gcc/testsuite/gcc.target/i386/avx-init-v4sf-2.c
+++ b/gcc/testsuite/gcc.target/i386/avx-init-v4sf-2.c
@@ -24,9 +24,8 @@ v4sf fm000() { return (v4sf){m,0.0f,0.0f,0.0f}; }
v4sf fm0m0() { return (v4sf){m,0.0f,m,0.0f}; }
v4sf fmmmm() { return (v4sf){m,m,m,m}; }
-/* { dg-final { scan-assembler-times "vmovss" 18 } } */
-/* { dg-final { scan-assembler-times "vpslldq" 3 } } */
+/* { dg-final { scan-assembler-times "vmovss" 9 } } */
+/* { dg-final { scan-assembler-times "vinsertps" 9 } } */
/* { dg-final { scan-assembler-times "vshufps" 2 } } */
-/* { dg-final { scan-assembler-times "vmovlhps" 4 } } */
+/* { dg-final { scan-assembler-times "vmovlhps" 3 } } */
/* { dg-final { scan-assembler-times "vbroadcastss" 2 } } */
-/* { dg-final { scan-assembler-times "vunpcklps" 5 } } */
diff --git a/gcc/testsuite/gcc.target/i386/avx2-init-v4sf-1.c b/gcc/testsuite/gcc.target/i386/avx2-init-v4sf-1.c
index 23ff0e76836..b549b9a68b2 100644
--- a/gcc/testsuite/gcc.target/i386/avx2-init-v4sf-1.c
+++ b/gcc/testsuite/gcc.target/i386/avx2-init-v4sf-1.c
@@ -25,9 +25,7 @@ v4sf fm0m0() { return (v4sf){m,0.0f,m,0.0f}; }
v4sf fmmmm() { return (v4sf){m,m,m,m}; }
/* { dg-final { scan-assembler-times "vinsertps" 16 } } */
-/* { dg-final { scan-assembler-times "vpslldq" 3 } } */
/* { dg-final { scan-assembler-times "vshufps" 2 } } */
-/* { dg-final { scan-assembler-times "vmovlhps" 4 } } */
-/* { dg-final { scan-assembler-times "vunpcklps" 5 } } */
+/* { dg-final { scan-assembler-times "vmovlhps" 3 } } */
/* { dg-final { scan-assembler-times "vmovss" 2 } } */
/* { dg-final { scan-assembler-times "vbroadcastss" 2 } } */
diff --git a/gcc/testsuite/gcc.target/i386/sse-init-v4sf-2.c b/gcc/testsuite/gcc.target/i386/sse-init-v4sf-2.c
index c958ae9e7f8..086ddbbc2a6 100644
--- a/gcc/testsuite/gcc.target/i386/sse-init-v4sf-2.c
+++ b/gcc/testsuite/gcc.target/i386/sse-init-v4sf-2.c
@@ -24,9 +24,9 @@ v4sf fm000() { return (v4sf){m,0.0f,0.0f,0.0f}; }
v4sf fm0m0() { return (v4sf){m,0.0f,m,0.0f}; }
v4sf fmmmm() { return (v4sf){m,m,m,m}; }
-/* { dg-final { scan-assembler-times "xorps" 16 } } */
-/* { dg-final { scan-assembler-times "movss" 19 } } */
-/* { dg-final { scan-assembler-times "movaps" 10 } } */
+/* { dg-final { scan-assembler-times "xorps" 9 } } */
+/* { dg-final { scan-assembler-times "movss" 9 } } */
+/* { dg-final { scan-assembler-times "movaps" 6 } } */
/* { dg-final { scan-assembler-times "shufps" 7 } } */
-/* { dg-final { scan-assembler-times "movlhps" 4 } } */
+/* { dg-final { scan-assembler-times "movlhps" 7 } } */
/* { dg-final { scan-assembler-times "unpcklps" 5 } } */
diff --git a/gcc/testsuite/gcc.target/i386/sse-init-v4sf-3.c b/gcc/testsuite/gcc.target/i386/sse-init-v4sf-3.c
index d98386e9ffc..ad1777c476b 100644
--- a/gcc/testsuite/gcc.target/i386/sse-init-v4sf-3.c
+++ b/gcc/testsuite/gcc.target/i386/sse-init-v4sf-3.c
@@ -24,7 +24,8 @@ v4sf fm000() { return (v4sf){m,0.0f,0.0f,0.0f}; }
v4sf fm0m0() { return (v4sf){m,0.0f,m,0.0f}; }
v4sf fmmmm() { return (v4sf){m,m,m,m}; }
+/* { dg-final { scan-assembler-times "movlhps" 7 } } */
/* { dg-final { scan-assembler-times "movss" 20 } } */
/* { dg-final { scan-assembler-times "shufps" 7 } } */
-/* { dg-final { scan-assembler-times "movlhps" 4 } } */
/* { dg-final { scan-assembler-times "unpcklps" 5 } } */
+/* { dg-final { scan-assembler-times "xorps" 3 } } */
diff --git a/gcc/testsuite/gcc.target/i386/sse2-init-v4sf-1.c b/gcc/testsuite/gcc.target/i386/sse2-init-v4sf-1.c
index 716d55b8161..9fb63dfce5a 100644
--- a/gcc/testsuite/gcc.target/i386/sse2-init-v4sf-1.c
+++ b/gcc/testsuite/gcc.target/i386/sse2-init-v4sf-1.c
@@ -24,9 +24,10 @@ v4sf fm000() { return (v4sf){m,0.0f,0.0f,0.0f}; }
v4sf fm0m0() { return (v4sf){m,0.0f,m,0.0f}; }
v4sf fmmmm() { return (v4sf){m,m,m,m}; }
-/* { dg-final { scan-assembler-times "movd" 32 } } */
-/* { dg-final { scan-assembler-times "pslldq" 3 } } */
-/* { dg-final { scan-assembler-times "shufps" 4 } } */
-/* { dg-final { scan-assembler-times "movlhps" 4 } } */
+/* { dg-final { scan-assembler-times "pslldq" 7 } } */
+/* { dg-final { scan-assembler-times "psrldq" 3 } } */
+/* { dg-final { scan-assembler-times "shufps" 6 } } */
+/* { dg-final { scan-assembler-times "movlhps" 2 } } */
/* { dg-final { scan-assembler-times "unpcklps" 5 } } */
+/* { dg-final { scan-assembler-times "movq" 3 } } */
/* { dg-final { scan-assembler-times "movss" 3 } } */
diff --git a/gcc/testsuite/gcc.target/i386/sse2-init-v4sf-2.c b/gcc/testsuite/gcc.target/i386/sse2-init-v4sf-2.c
index 44ed9944ef2..5e55ce14f49 100644
--- a/gcc/testsuite/gcc.target/i386/sse2-init-v4sf-2.c
+++ b/gcc/testsuite/gcc.target/i386/sse2-init-v4sf-2.c
@@ -24,8 +24,10 @@ v4sf fm000() { return (v4sf){m,0.0f,0.0f,0.0f}; }
v4sf fm0m0() { return (v4sf){m,0.0f,m,0.0f}; }
v4sf fmmmm() { return (v4sf){m,m,m,m}; }
-/* { dg-final { scan-assembler-times "movss" 20 } } */
-/* { dg-final { scan-assembler-times "pslldq" 3 } } */
-/* { dg-final { scan-assembler-times "shufps" 4 } } */
-/* { dg-final { scan-assembler-times "movlhps" 4 } } */
+/* { dg-final { scan-assembler-times "pslldq" 7 } } */
+/* { dg-final { scan-assembler-times "psrldq" 3 } } */
+/* { dg-final { scan-assembler-times "shufps" 6 } } */
+/* { dg-final { scan-assembler-times "movlhps" 2 } } */
/* { dg-final { scan-assembler-times "unpcklps" 5 } } */
+/* { dg-final { scan-assembler-times "movq" 3 } } */
+/* { dg-final { scan-assembler-times "movss" 20 } } */
diff --git a/gcc/testsuite/gcc.target/i386/sse4_1-init-v4sf-2.c b/gcc/testsuite/gcc.target/i386/sse4_1-init-v4sf-2.c
index b4e7f7293a9..250ee1fd795 100644
--- a/gcc/testsuite/gcc.target/i386/sse4_1-init-v4sf-2.c
+++ b/gcc/testsuite/gcc.target/i386/sse4_1-init-v4sf-2.c
@@ -25,8 +25,8 @@ v4sf fm0m0() { return (v4sf){m,0.0f,m,0.0f}; }
v4sf fmmmm() { return (v4sf){m,m,m,m}; }
/* { dg-final { scan-assembler-times "insertps" 16 } } */
-/* { dg-final { scan-assembler-times "pslldq" 3 } } */
-/* { dg-final { scan-assembler-times "shufps" 4 } } */
-/* { dg-final { scan-assembler-times "movlhps" 4 } } */
-/* { dg-final { scan-assembler-times "unpcklps" 5 } } */
+/* { dg-final { scan-assembler-times "movlhps" 3 } } */
/* { dg-final { scan-assembler-times "movss" 3 } } */
+/* { dg-final { scan-assembler-not "pslldq" } } */
+/* { dg-final { scan-assembler-times "shufps" 4 } } */
+/* { dg-final { scan-assembler-not "unpcklps" } } */
diff --git a/gcc/testsuite/gcc.target/i386/sse4_1-init-v4sf-3.c b/gcc/testsuite/gcc.target/i386/sse4_1-init-v4sf-3.c
index 9a51d1b7dcf..d128b745b1c 100644
--- a/gcc/testsuite/gcc.target/i386/sse4_1-init-v4sf-3.c
+++ b/gcc/testsuite/gcc.target/i386/sse4_1-init-v4sf-3.c
@@ -24,8 +24,9 @@ v4sf fm000() { return (v4sf){m,0.0f,0.0f,0.0f}; }
v4sf fm0m0() { return (v4sf){m,0.0f,m,0.0f}; }
v4sf fmmmm() { return (v4sf){m,m,m,m}; }
-/* { dg-final { scan-assembler-times "movss" 20 } } */
-/* { dg-final { scan-assembler-times "pslldq" 3 } } */
+/* { dg-final { scan-assembler-times "insertps" 9 } } */
+/* { dg-final { scan-assembler-times "movss" 11 } } */
+/* { dg-final { scan-assembler-times "movlhps" 3 } } */
+/* { dg-final { scan-assembler-not "pslldq" } } */
/* { dg-final { scan-assembler-times "shufps" 4 } } */
-/* { dg-final { scan-assembler-times "movlhps" 4 } } */
-/* { dg-final { scan-assembler-times "unpcklps" 5 } } */
+/* { dg-final { scan-assembler-not "unpcklps" } } */