NUMA balancing degrading performance

--RnlQjJ0d97Da+TV1
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hello,

I'm experimenting with the kernel's automatic NUMA balancing and I'm
experiencing performance loss when I turn the balancing on.

Through the sysctl kernel.numa_balancing, I've ran some benchmarks and
the performance with the balancing turned *off* was consistently better.
A general figure was 50% slower with balancing on, but in one particular
case it got 10 times slower.

AFAIK, none of the benchmarks are NUMA-aware, so I was expecting some
kind of performance gain. In the case any are in fact NUMA-aware, I
could understand a slight performance loss, but just slight: since the
scan period adapts to the ratio of local/remote faults, it should
quickly understand that memory placement is already optimal and increase
the scan period so that the overhead is small.

I would appreciate if someone could help me find what's causing this
behavior.

I'm running on CentOS 6.5 with Linux 3.17.1 (self compiled).

Below is some stuff that might be useful. If you need anything else,
just tell me.

Thank you,
Martin

$ for x in /proc/sys/kernel/numa_balancing* ; do echo $x ; cat $x; done
/proc/sys/kernel/numa_balancing
32768
/proc/sys/kernel/numa_balancing_scan_delay_ms
1000
/proc/sys/kernel/numa_balancing_scan_period_max_ms
60000
/proc/sys/kernel/numa_balancing_scan_period_min_ms
1000
/proc/sys/kernel/numa_balancing_scan_size_mb
256
(I tried playing with differents values of scan size, but it didn't made mu=
ch=20
of a difference)

$ numactl --hardware
available: 4 nodes (0,2,4,6)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 16076 MB
node 0 free: 15713 MB
node 2 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
node 2 size: 16157 MB
node 2 free: 15903 MB
node 4 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
node 4 size: 16157 MB
node 4 free: 15998 MB
node 6 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 6 size: 16141 MB
node 6 free: 15941 MB
node distances:
node   0   2   4   6=20
  0:  10  16  16  16=20
  2:  16  10  16  16=20
  4:  16  16  10  16=20
  6:  16  16  16  10

$ grep NUMA .config
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=3Dy
CONFIG_ARCH_WANTS_PROT_NUMA_PROT_NONE=3Dy
CONFIG_ARCH_USES_NUMA_PROT_NONE=3Dy
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=3Dy
CONFIG_NUMA_BALANCING=3Dy
# CONFIG_X86_NUMACHIP is not set
CONFIG_NUMA=3Dy
CONFIG_AMD_NUMA=3Dy
CONFIG_X86_64_ACPI_NUMA=3Dy
# CONFIG_NUMA_EMU is not set
CONFIG_USE_PERCPU_NUMA_NODE_ID=3Dy
CONFIG_ACPI_NUMA=3Dy

$ cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 1
model name      : AMD Opteron(TM) Processor 6272
stepping        : 2
microcode       : 0x6000629
cpu MHz         : 1400.000
cache size      : 2048 KB
physical id     : 0
siblings        : 16
core id         : 0
cpu cores       : 8
apicid          : 32
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc
extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 cx16 sse4_1
sse4_2 popcnt aes xsave avx lahf_lm cmp_lega
cy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs
xop skinit wdt lwp fma4 nodeid_msr topoext perfctr_core perfctr_nb arat
cpb hw_pstate npt lbrv svm_lock nrip_save tsc_scale vmcb_clean
flushbyasid decodeassists pausefilter pfthreshold
bugs            : fxsave_leak
bogomips        : 4200.06
TLB size        : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb
(63 other just like this)

$ cat /proc/meminfo
MemTotal:       66083340 kB
MemFree:        64445100 kB
MemAvailable:   64477352 kB
Buffers:          172848 kB
Cached:           132988 kB
SwapCached:            0 kB
Active:           857468 kB
Inactive:          70176 kB
Active(anon):     621996 kB
Inactive(anon):        8 kB
Active(file):     235472 kB
Inactive(file):    70168 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      65535996 kB
SwapFree:       65535996 kB
Dirty:                28 kB
Writeback:             0 kB
AnonPages:        692268 kB
Mapped:            21468 kB
Shmem:               200 kB
Slab:             182152 kB
SReclaimable:      96752 kB
SUnreclaim:        85400 kB
KernelStack:       13120 kB
PageTables:         5928 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    98577664 kB
Committed_AS:    1363304 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      400824 kB
VmallocChunk:   34309010120 kB
HardwareCorrupted:     0 kB
AnonHugePages:    665600 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      160128 kB
DirectMap2M:     5064704 kB
DirectMap1G:    61865984 kB

--RnlQjJ0d97Da+TV1
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAABCAAGBQJUTonHAAoJEKAgcdgOx8KDLocP/0+qN435yF+Nj8ht7DuaPk4v
PJOTcOknrarixOq9ABmCAmskKFUlWB+4Jf9hEZARwWYpqUP0jRQW9lXG7D8BpOBV
LpDaSoBuuQIeX5i1KTyndEEpuwbD1XQWRLOak7LV4x26pTn1IgoGgdNof/OGMt8s
xXvbjcceoT8gEGBAuvZ+ofCHGGG5Z5MoJErOl+jTaDqzg9NzqXz+05M1wZbL6P0Z
Q3Uup+2FP5qdI5R4NBUxsFvuiaf6AFoUm0fqlRAoxa6znnTfBlcmePI+hu2bAWCY
erYeEWjXuSjWnk4//+Tod3qRu7ra1ogWspR5d9dHbNi/YQjIOeJ02oGPPHRfgmR5
dx7bi7zbhs/WeIAXN7OLzEFHJqcSwLK+DNGdcrMhggFuiHlRZ4cfPYtw0OMI2Tme
Zje78uTbrt3SWcgXwbNKaAR+Vx6/bJf4pxgUC8EqbQkYJApKNmKsry+ITF/o4Gm5
oWnREWKtQ0EtJ/V5UF/O1OtUwy7QDb7G4bUqlaK4pmnQZmHawEn3bw2ZV7pssFRS
sS5HrEAGcV10oxBgADrhqcws+3ertBRniybku5tbbZVydaUWqFDHNg3rBhCvEUrG
jxOsTMaIN6ffKaiQbA1FQ6w/rjt0cgdocOO4tCOqFn05dIkJE/CJg4RFLbddJjC1
gxKMV3zZ5kLpXMwOVaJr
=ys5w
-----END PGP SIGNATURE-----

--RnlQjJ0d97Da+TV1--