Re: NUMA balancing degrading performance

Hi Martin,

I had some time to run NPB.

LU-HP is not available in NPB 3.3.1, so I used LU instead.

Here are my results:

LU Benchmark Completed.
 Class           =                        C
 Size            =            162x 162x 162
 Iterations      =                      250
 Time in seconds =                    XX.XX
 Total threads   =                       80
 Avail threads   =                       80
 Mop/s total     =                 51420.84
 Mop/s/thread    =                   642.76
 Operation type  =           floating point
 Verification    =               SUCCESSFUL
 Version         =                    3.3.1
 Compile date    =              28 Oct 2014

 Compile options:
    F77          = gfortran
    FLINK        = $(F77)
    F_LIB        = (none)
    F_INC        = (none)
    FFLAGS       = -O3 -fopenmp -mcmodel=medium
    FLINKFLAGS   = -O3 -fopenmp
    RAND         = (none)

With numa balance disabled :
(sudo bash -c "echo 0 > /proc/sys/kernel/numa_balancing"):

1st run:  Time in seconds =                    39.65
2nd run:  Time in seconds =                    39.47
3rd run:  Time in seconds =                    41.31
4th run:  Time in seconds =                    40.42

The measurements without numa balance are stable and around 40 sec.

With numa balance enabled
(sudo bash -c "echo 1 > /proc/sys/kernel/numa_balancing"):

1st run: Time in seconds =                    53.89
2nd run: Time in seconds =                    51.95
3rd run: Time in seconds =                    56.22
4th run: Time in seconds =                    64.20

Enabling this option increases the runtime by more then 50 % in the worst case.

Here is some information about the hardware:

Kernel: Linux inwest 3.16.4-1-ARCH #1 SMP PREEMPT Mon Oct 6 08:22:27
CEST 2014 x86_64 GNU/Linux
CPU: Intel(R) Xeon(R) CPU E7- 4850  @ 2.00GHz

numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 40 41 42 43 44 45 46 47 48 49
node 0 size: 64427 MB
node 0 free: 63912 MB
node 1 cpus: 10 11 12 13 14 15 16 17 18 19 50 51 52 53 54 55 56 57 58 59
node 1 size: 64509 MB
node 1 free: 64066 MB
node 2 cpus: 20 21 22 23 24 25 26 27 28 29 60 61 62 63 64 65 66 67 68 69
node 2 size: 64509 MB
node 2 free: 63987 MB
node 3 cpus: 30 31 32 33 34 35 36 37 38 39 70 71 72 73 74 75 76 77 78 79
node 3 size: 64509 MB
node 3 free: 64035 MB
node distances:
node   0   1   2   3
  0:  10  21  21  21
  1:  21  10  21  21
  2:  21  21  10  21
  3:  21  21  21  10

Regards,
Andreas

2014-10-27 22:27 GMT+01:00 Martin Ichilevici de Oliveira
<[email protected]>:
> Hello Andreas,
>
> Thank you for your reply. Please check my comments inline.
>
>> it would be good to know which applications/benchmarks you were running.
>>
>> Have you tried out some well known and open source benchmarks?
>>
>> NAS Parallel Benchmarks -
>> http://www.nas.nasa.gov/publications/npb.html (Fortran Code)
>> NPB2.3-omp-C.tgz (C version NPB in OpenMP) -
>> http://www.hpcs.cs.tsukuba.ac.jp/omni-compiler/download/NPB2.3-omp-C.tgz
>> Stream - http://www.cs.virginia.edu/stream/FTP/Code/stream.c
>
> Sorry, I should have mentioned that. I tried some NAS benchmarks:
> bt, sp and lu-hp. bt and sp were around 60% slower with the balancing
> turned on, and lu-hp was 10 times slower.
>
> I also ran Lulesh, which was roughly 100% slower with the balancing
> turned on.
>
>> Do you have "numad" running on your machine? If it is running you
>> should stop it.
>
> I checked and it's not running.
>
> Cheers,
> Martin