[PATCH 01/20] alpha: use prefetch-with-modify-intent in memset on EV7

The unrolled loop in the ev6 memset issues wh64 two cache lines ahead of
the stores to avoid read-for-ownership on whole-line stores.  That is
tuned for the 21264 (EV6, EV67, EV68) with an external Bcache.  On the
21364 (EV7), with its on-chip memory controller and much longer memory
latency, two lines is far too close: the hint neither overlaps the fill
nor avoids the Mbox replay traps it provokes in the store stream.

Dispatch on IMPLVER (2 for the 21264 family, 3 for the 21364 family) and
give EV7 its own unrolled loop that issues a software prefetch with
modify intent (LDS to F31, a ReadBlkMod that allocates the line
writeable) six cache lines ahead of the stores.  On EV7 this beats wh64
at every distance and, unlike wh64, only reads, so prefetching past the
end of the region is harmless and no overrun guard is needed.  The 21264
code path is unchanged.

Measured on an AlphaServer ES47 (EV7, 1.3GHz), cycles/call:
  size     cold before  cold after     hot before  hot after
  1024            2056        1566 -24%        220        204
  4096           13167       11036 -16%        807        731
  65536         111438       76719 -31%       8495       7729
  262144        423219      285094 -33%     102067      54753 -46%
Mbox replay traps (perf -e r4) were used to identify the wh64 prefetch
distance as the bottleneck.
---
 sysdeps/alpha/alphaev6/memset.S | 51 +++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)

diff --git ./sysdeps/alpha/alphaev6/memset.S ./sysdeps/alpha/alphaev6/memset.S
index 6057274f1a..5c1e307cdf 100644
--- ./sysdeps/alpha/alphaev6/memset.S
+++ ./sysdeps/alpha/alphaev6/memset.S
@@ -103,6 +103,26 @@ $aligned:
 	 * $3	Number quads to write
 	 */
 
+	/*
+	 * Dispatch on the CPU implementation.  The 21264 (EV6, EV67, EV68) uses
+	 * wh64 two cache lines ahead to avoid read-for-ownership on whole-line
+	 * stores.  That distance is far too short for the 21364 (EV7), whose
+	 * on-chip memory controller has a much longer latency: there the hint
+	 * neither overlaps the fill nor avoids Mbox replay traps.  Issuing a
+	 * software prefetch-with-modify-intent (LDS to F31, a ReadBlkMod that
+	 * allocates the line writeable) six lines ahead instead recovers a
+	 * large cold-store win -- about -30% at >=64KB -- and, unlike wh64,
+	 * does not zero the line, so no end-guard against overrun is needed.
+	 * EV7 therefore takes its own unrolled loop.  IMPLVER returns 2 for the
+	 * 21264 family (EV6/67/68) and 3 for the 21364 family (EV7/EV79).
+	 */
+	subq	$3, 16, $4	# E : Only try to unroll if > 128 bytes
+	blt	$4, $loop	# U : too small -- scalar tail loop
+	implver	$2		# E : 2 = 21264 (EV6/67/68), 3 = 21364 (EV7)
+	subq	$2, 3, $2	# E : EV7 -> 0
+
+	beq	$2, $ev7_unroll	# U : EV7 takes the prefetch-modify path
+
 	and	$16, 0x3f, $2	# E : Forward work (only useful for unrolled loop)
 	subq	$3, 16, $4	# E : Only try to unroll if > 128 bytes
 	subq	$2, 0x40, $1	# E : bias counter (aligning stuff 0mod64)
@@ -174,6 +194,37 @@ $do_wh64:
 	nop
 	nop
 	beq	$3, $no_quad	# U : Might have finished already
+	br	$31, $loop	# U : 1..7 trailing quads
+
+	/*
+	 * EV7 (21364): 64-byte unrolled store loop that issues a software
+	 * prefetch-with-modify-intent (LDS to F31) six cache lines (384 bytes)
+	 * ahead of the current block.  No 0mod64 pre-align is needed.  The
+	 * prefetch only reads -- it never writes -- so prefetching past the end
+	 * of the region is harmless (a prefetch that misses the TLB or hits an
+	 * unwriteable page is simply dropped), and no end-guard is required.
+	 * $3 - quads left, $5 - dest (0mod8), $17 - store value, $7 scratch.
+	 */
+	.align 4
+$ev7_unroll:
+	lds	$f31, 6*64($5)	# L : prefetch w/ modify intent, six lines ahead
+	stq	$17, 0($5)	# L :
+	stq	$17, 8($5)	# L :
+	stq	$17, 16($5)	# L :
+	stq	$17, 24($5)	# L :
+	stq	$17, 32($5)	# L :
+	stq	$17, 40($5)	# L :
+	stq	$17, 48($5)	# L :
+	stq	$17, 56($5)	# L :
+
+	subq	$3, 8, $3	# E : decrement quad count
+	addq	$5, 64, $5	# E : advance dest
+	subq	$3, 8, $7	# E : at least 8 more quads?
+	bge	$7, $ev7_unroll	# U :
+
+	beq	$3, $no_quad	# U : exact multiple, done
+	nop
+	nop
 
 	.align 4
 	/*
-- 
2.54.0