[PATCH 09/20] alpha: add EV6/EV7 assembly memcmp
Matt Turner <[email protected]>
| Newsgroups | gmane.comp.lib.glibc.alpha |
|---|---|
| Message-ID | <1f888a256c172062afcbe683dee419aac5d279e3.1786497497.git.mattst88@gmail.com> |
Alpha had no assembly memcmp and fell back to the generic C implementation. Add one for the alphaev6 directory (shared by the 21264 and 21364). When both operands share alignment, byte-align to a quadword and then compare 64 bytes per iteration, OR-accumulating the per-quad xors; a non-zero accumulator means the difference is somewhere in the block, which a byte scan from the block base pinpoints to return the correct sign. Operands of differing alignment use a byte compare. The loop issues a plain read prefetch (LDL to R31) on both streams eight cache lines ahead. On the 21364 (EV7), whose on-chip memory controller has a long memory latency, this hides the Bcache misses that otherwise stall the compare; on the 21264 it is at worst a dropped hint, so no IMPLVER dispatch is needed. A read prefetch -- rather than the modify-intent LDS/LDT used by memset/memcpy -- is deliberate: memcmp never writes its operands, so requesting the lines for ownership would only add coherence traffic on a multiprocessor. The 1 to 7 bytes left after the loops are compared with a single masked quadword rather than a byte at a time, which would cost about eight issue slots per byte and dominate any length that is not a multiple of eight. Both operands are 0mod8 by then, since the head aligned them and everything since has advanced by whole quadwords, and an aligned quadword cannot cross a page boundary, so a whole one can be loaded at either pointer even though only some of it belongs to the compare; masking the difference discards the rest. Nothing is stored, so reading past the end of the operands is not observable. A mismatch still falls into the byte loop, which is what determines which byte differed and in which direction. Measured on an AlphaServer ES47 (EV7, 1.3GHz), cold cyc/call vs the generic C: size generic asm 1024 3617 1502 2.35x 4096 14527 6423 2.24x 65536 231362 96424 2.40x 262144 916913 386234 2.37x bcmp is provided as a weak alias, as in the generic implementation. Verified on an EV68CB against the C memcmp for every length from 0 to 300 at 24 alignments, both for equal operands and with a difference planted at every position in turn, comparing the sign of the result. __memcmpeq is aliased to memcmp here too. string/memcmp.c provides that alias for the C implementation and string/memcmpeq.c is deliberately empty, so an assembly memcmp that replaces the C one has to define it, as sparc64 and s390 do. Without it an --host=alphaev6 or later build links no __memcmpeq at all, which drops a symbol the ABI lists and fails to link string/inl-tester. --- sysdeps/alpha/alphaev6/memcmp.S | 226 ++++++++++++++++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 sysdeps/alpha/alphaev6/memcmp.S diff --git ./sysdeps/alpha/alphaev6/memcmp.S ./sysdeps/alpha/alphaev6/memcmp.S new file mode 100644 index 0000000000..0a3d2e11e6 --- /dev/null +++ ./sysdeps/alpha/alphaev6/memcmp.S @@ -0,0 +1,226 @@ +/* Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + <https://www.gnu.org/licenses/>. */ + +/* + * EV6/EV7 memcmp. When both operands share alignment, byte-align to a + * quadword and compare 64 bytes per iteration, OR-accumulating the per-quad + * xors; a non-zero accumulator means a difference lies in the block, which a + * byte scan from the block base then pinpoints to return the correct sign. + * Operands of differing alignment fall back to a byte compare. + * + * The loop issues a plain read prefetch (LDL to R31) on both streams several + * cache lines ahead. On the 21364 (EV7), whose on-chip memory controller has + * a long latency, this hides the Bcache misses that otherwise stall the + * compare; on the 21264 it is at worst a dropped hint. A read prefetch (not + * the modify-intent LDS/LDT) is deliberate: memcmp never writes its operands, + * so requesting lines for ownership would only add coherence traffic on SMP. + * + * Temp usage: + * $0 - result + * $1,$2 - scratch + * $3,$4 - loaded data + * $5,$6 - xor / accumulator + */ + +#include <sysdep.h> + + .arch ev6 + .set noreorder + .set noat + +/* Read prefetch distance, in bytes (8 cache lines ahead). The CWG asks for + 8/n cache blocks for n streams against the 21264's 8-entry MAF, so four for + the two streams here; measured on an EV68CB the curve is flat from four + blocks out to eight, and only a distance below two costs anything. + + These must stay plain read prefetches (LDL to R31). Asking for the lines + with modify intent instead (LDS to F31) measures 2x slower on an EV68CB -- + 756446 cycles against 368989 for a cold 256KB compare -- because ReadBlkMod + provokes about twice the Mbox replay traps on a stream that is only ever + read. Bcache misses are unchanged, so it is not extra memory traffic. + ReadBlkModEN (LDT to F31) measures identical to the plain prefetch, since + the evict-next variants are 21364-only and degrade here. */ +#define PF (8 * 64) + +ENTRY(memcmp) + .prologue 0 + + mov $31, $0 # E : default result 0 (equal) + beq $18, $ret # U : n == 0 -> equal + xor $16, $17, $1 # E : do the operands share alignment? + and $1, 7, $1 # E : ... mod 8 + + bne $1, $bytewise # U : different alignment -> byte compare + and $16, 7, $1 # E : already 0mod8? + beq $1, $aligned # U : yes + nop # E : + + /* Same misalignment: byte-compare up to a 0mod8 boundary. */ +$head: + ldbu $3, 0($16) # L : a byte + ldbu $4, 0($17) # L : b byte + lda $16, 1($16) # E : a++ + lda $17, 1($17) # E : b++ + + subq $3, $4, $0 # E : difference (unsigned bytes) + bne $0, $ret # U : differ -> return sign + subq $18, 1, $18 # E : n-- + beq $18, $reteq # U : consumed all -> equal + + and $16, 7, $1 # E : aligned now? + bne $1, $head # U : keep going + +$aligned: + cmple $18, 63, $1 # E : at least one 64B block? + bne $1, $quadtail # U : no + nop # E : + nop # E : + + .align 4 +$loop: + ldl $31, PF($16) # L : read prefetch a, PF bytes ahead + ldl $31, PF($17) # L : read prefetch b + ldq $3, 0($16) # L : a[0..7] + ldq $4, 0($17) # L : b[0..7] + + xor $3, $4, $5 # E : seed the accumulator + ldq $3, 8($16) # L : + ldq $4, 8($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : + ldq $3, 16($16) # L : + ldq $4, 16($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : + ldq $3, 24($16) # L : + ldq $4, 24($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : + ldq $3, 32($16) # L : + ldq $4, 32($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : + ldq $3, 40($16) # L : + ldq $4, 40($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : + ldq $3, 48($16) # L : + ldq $4, 48($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : + ldq $3, 56($16) # L : + ldq $4, 56($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : full 64B xor summary + bne $5, $bytewise # U : a difference is in this block + subq $18, 64, $18 # E : n -= 64 + lda $16, 64($16) # E : a += 64 + + lda $17, 64($17) # E : b += 64 + cmple $18, 63, $1 # E : another full block? + beq $1, $loop # U : + nop # E : + + /* 0..63 bytes left, still 0mod8: compare remaining quads. */ +$quadtail: + subq $18, 8, $2 # E : (n - 8) + blt $2, $tailquad # U : < 8 left -> masked quad + +$qt: + ldq $3, 0($16) # L : + ldq $4, 0($17) # L : + xor $3, $4, $5 # E : + bne $5, $bytewise # U : differ in this quad (n unchanged) + + lda $16, 8($16) # E : a += 8 + lda $17, 8($17) # E : b += 8 + mov $2, $18 # E : commit n -= 8 + subq $18, 8, $2 # E : (n - 8) + + bge $2, $qt # U : another quad + nop # E : + nop # E : + nop # E : + + /* + * 1..7 bytes left, both operands 0mod8 -- the head aligned them and + * everything since has advanced by whole quadwords. An aligned + * quadword cannot cross a page boundary, so loading a whole one at + * either pointer is safe even though only $18 bytes belong to the + * compare; masking the difference down to those bytes discards + * whatever followed. Nothing is stored, so reading past the end of + * the operands is not observable. + * + * This is only the tail. A mismatch still falls into the byte loop + * below, which is what works out which byte differed and in which + * direction, and which the block and quad loops above also branch to. + */ +$tailquad: + beq $18, $reteq # U : nothing left -> equal + nop # E : + nop # E : + nop # E : + + ldq $3, 0($16) # L : + ldq $4, 0($17) # L : + xor $3, $4, $5 # E : + mskql $5, $18, $5 # U : only the bytes being compared + + beq $5, $reteq # U : all equal + nop # E : + nop # E : + nop # E : + + /* Tail bytes, and the unaligned / locate path. On entry from a block + or quad mismatch, the first differing byte lies within the bytes + scanned here, so the loop returns its sign before running off. */ +$bytewise: + beq $18, $reteq # U : nothing left -> equal + nop # E : + +$bl: + ldbu $3, 0($16) # L : + ldbu $4, 0($17) # L : + lda $16, 1($16) # E : + lda $17, 1($17) # E : + + subq $3, $4, $0 # E : unsigned byte difference + bne $0, $ret # U : differ -> sign + subq $18, 1, $18 # E : n-- + bne $18, $bl # U : + +$reteq: + mov $31, $0 # E : equal +$ret: + ret $31, ($26), 1 # L0 : + +END(memcmp) +libc_hidden_builtin_def (memcmp) +#ifdef weak_alias +# undef bcmp +weak_alias (memcmp, bcmp) +#endif +#undef __memcmpeq +strong_alias (memcmp, __memcmpeq) +libc_hidden_def (__memcmpeq) -- 2.54.0