Re: numa_alloc_onnode does not allocate on node passed as argument

Hi Andres,

I've tried out your code and the numa calls work for me as expected.

In all cases and with or without (numa_set_preferred(1)) it returns nod=
e 1.

Some info on my machine:

uname -a
Linux inwest 3.16.4-1-ARCH #1 SMP PREEMPT Mon Oct 6 08:22:27 CEST 2014
x86_64 GNU/Linux

CPU: model name      : Intel(R) Xeon(R) CPU E7- 4850  @ 2.00GHz


numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 40 41 42 43 44 45 46 47 48 49
node 0 size: 64427 MB
node 0 free: 63884 MB
node 1 cpus: 10 11 12 13 14 15 16 17 18 19 50 51 52 53 54 55 56 57 58 5=
9
node 1 size: 64509 MB
node 1 free: 64046 MB
node 2 cpus: 20 21 22 23 24 25 26 27 28 29 60 61 62 63 64 65 66 67 68 6=
9
node 2 size: 64509 MB
node 2 free: 63972 MB
node 3 cpus: 30 31 32 33 34 35 36 37 38 39 70 71 72 73 74 75 76 77 78 7=
9
node 3 size: 64509 MB
node 3 free: 64036 MB
node distances:
node   0   1   2   3
  0:  10  21  21  21
  1:  21  10  21  21
  2:  21  21  10  21
  3:  21  21  21  10

extra/numactl 2.0.10-1 [installed]
    Simple NUMA policy support

2014-10-31 0:21 GMT+01:00 Andres N=3DC3=3DB6tzli <[email protected]>=
:
> Hi Elena,
>
> Thank you so much for looking into this issue. It is good to hear tha=
t yo=3D
u are getting the same strange result.
>
> I posted the output of /proc/pid/numa_maps here: https://gist.github.=
com/=3D
4tXJ7f/5e89f466e29cd1f7f1aa
>
> I hope this helps.
>
> Thanks again,
> Andres
>
>> On 29 Oct 2014, at 21:33, Elena Ufimtseva <[email protected]> wrot=
e:
>>
>> Hello Andres
>>
>> I looked at the example you gave, had multiple variations running an=
d
>> have same strange results.
>> The default local policy should be in use when there is no other pol=
icy =3D
defined.
>> The only thing what comes to my mind its the shared library libnuma
>> which has its data on different node then the node I try to run
>> the test process on.
>> Can you take a look and check what node is used by libnuma in
>> /proc/pid/numa_maps?
>>
>> I will keep searching for an answer, its rather interesting topic.
>> Or maybe someone else will give more details on this.
>>
>> Thank you!
>>
>>
>> On Thu, Oct 23, 2014 at 1:17 PM, Andres N=3DC3=3DB6tzli <noetzli@sta=
nford.ed=3D
u> wrote:
>>> Hi Elena,
>>>
>>> That would be great! I created a gist with the kernel config (cat /=
boot=3D
/config-$(uname -r)): https://gist.github.com/4tXJ7f/408a562abe5d4f2865=
6d
>>>
>>> Please let me know if you need anything else.
>>>
>>> Thank you very much,
>>> Andres
>>>
>>>> On 23 Oct 2014, at 06:15, Elena Ufimtseva <[email protected]> wr=
ote:
>>>>
>>>> Hi Andres
>>>>
>>>> I will poke around this on the weekend on my NUMA machine.
>>>> Can you also attach your kernel config please?
>>>>
>>>> Thank you.
>>>>
>>>> On Wed, Oct 22, 2014 at 12:40 PM, Andres N=3DC3=3DB6tzli <noetzli@=
stanford=3D
=2Eedu> wrote:
>>>>> Hi Elena,
>>>>>
>>>>> Thank you very much for your quick reply! numa_set_strict(1) and =
numa=3D
_set_strict(0) both result in the wrong output. I did not change the de=
faul=3D
t policy.
>>>>>
>>>>> numa_get_membind returns 1 for all nodes before and after numa_ru=
n_on=3D
_node.
>>>>> numa_get_interleave_mask returns 0 for all nodes.
>>>>> numa_get_run_node_mask is all 1s before and 0010 after numa_run_o=
n_no=3D
de.
>>>>>
>>>>> The machine config (the CPUs are all Intel(R) Xeon(R) CPU E5-4657=
L v2=3D
 @ 2.40GHz):
>>>>>
>>>>> $ numactl --hardware
>>>>> available: 4 nodes (0-3)
>>>>> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 48 49 50 51 52 53 54 55 56=
 57 =3D
58 59
>>>>> node 0 size: 262093 MB
>>>>> node 0 free: 966 MB
>>>>> node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 60 61 62 63 64 6=
5 66=3D
 67 68 69 70 71
>>>>> node 1 size: 262144 MB
>>>>> node 1 free: 82 MB
>>>>> node 2 cpus: 24 25 26 27 28 29 30 31 32 33 34 35 72 73 74 75 76 7=
7 78=3D
 79 80 81 82 83
>>>>> node 2 size: 262144 MB
>>>>> node 2 free: 102 MB
>>>>> node 3 cpus: 36 37 38 39 40 41 42 43 44 45 46 47 84 85 86 87 88 8=
9 90=3D
 91 92 93 94 95
>>>>> node 3 size: 262144 MB
>>>>> node 3 free: 113 MB
>>>>> node distances:
>>>>> node   0   1   2   3
>>>>> 0:  10  20  30  20
>>>>> 1:  20  10  20  30
>>>>> 2:  30  20  10  20
>>>>> 3:  20  30  20  10
>>>>>
>>>>> Thanks again,
>>>>> Andres
>>>>>
>>>>>> On 22 Oct 2014, at 06:12, Elena Ufimtseva <[email protected]> =
wrot=3D
e:
>>>>>>
>>>>>> On Tue, Oct 21, 2014 at 11:47 PM, Andres N=3DC3=3DB6tzli <noetzl=
i@stanfo=3D
rd.edu> wrote:
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I am experiencing a weird problem. When using numa_alloc_onnode=
 rep=3D
eatedly to allocate memory, it does not allocate memory on the node pas=
sed =3D
as an argument.
>>>>>>>
>>>>>>> Sample code:
>>>>>>> #include <numa.h>
>>>>>>> #include <numaif.h>
>>>>>>> #include <iostream>
>>>>>>> using namespace std;
>>>>>>>
>>>>>>> void find_memory_node_for_addr(void* ptr) {
>>>>>>> int numa_node =3D3D -1;
>>>>>>> if(get_mempolicy(&numa_node, NULL, 0, ptr, MPOL_F_NODE | MPOL_F=
_ADD=3D
R) < 0)
>>>>>>>    cout << "WARNING: get_mempolicy failed" << endl;
>>>>>>> cout << numa_node << endl;
>>>>>>> }
>>>>>>>
>>>>>>> int main() {
>>>>>>> int64_t* x;
>>>>>>> int64_t n =3D3D 5000;
>>>>>>> //numa_set_preferred(1);
>>>>>>>
>>>>>>> numa_run_on_node(2);
>>>>>>> for(int i =3D3D 0; i < 20; i++) {
>>>>>>>    size_t s =3D3D n * sizeof(int64_t);
>>>>>>>    x =3D3D (int64_t*)numa_alloc_onnode(s, 1);
>>>>>>>    for(int j =3D3D 0; j < n; j++)
>>>>>>>       x[j] =3D3D j + i;
>>>>>>>    find_memory_node_for_addr(x);
>>>>>>> }
>>>>>>>
>>>>>>> return 0;
>>>>>>> }
>>>>>>>
>>>>>>> Output:
>>>>>>> 1
>>>>>>> 1
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>>
>>>>>>> When uncommenting the line "numa_set_preferred(1);=3DE2=3D80=3D=
9D, the ou=3D
tput is all 1s as expected. Am I doing something wrong? Have you seen s=
imil=3D
ar issues?
>>>>>>>
>>>>>>> I am running Ubuntu 12.04.5 LTS:
>>>>>>> $ cat /proc/version
>>>>>>> Linux version 3.2.0-29-generic (buildd@allspice) (gcc version 4=
=2E6.3=3D
 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #46-Ubuntu SMP Fri Jul 27 17:03:23 UT=
C 20=3D
12
>>>>>>>
>>>>>>> I am using libnuma 2.0.10 but I=3DE2=3D80=3D99ve had the same p=
roblem wit=3D
h 2.0.8~rc3-1.
>>>>>>>
>>>>>>> Thank you very much,
>>>>>>> Andres
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux=
-num=3D
a" in
>>>>>>> the body of a message to [email protected]
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.h=
tml
>>>>>>
>>>>>> Hi Andres
>>>>>>
>>>>>> Can you try to use strict policy by calling numa_set_strict?
>>>>>>
>>>>>> If you comment out setting the preferred node, the default polic=
y is
>>>>>> in action (I assume you did no change it, not for the process, n=
ot
>>>>>> system wide) which is preferred also.
>>>>>> But here you set preferred to a specific node and manual says, t=
he
>>>>>> default for process is to allocate on the node it runs.
>>>>>> So I wonder what is the cpu affinity for this process looks like=
=2E..
>>>>>> Also maybe just to confirm you can check the policy from within =
your
>>>>>> running code?
>>>>>>
>>>>>> Can you also post the machine NUMA config?
>>>>>>
>>>>>> --
>>>>>> Elena
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Elena
>>>
>>>
>>
>>
>>
>> --
>> Elena
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-numa"=
 in
> the body of a message to [email protected]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

2014-10-31 0:21 GMT+01:00 Andres N=C3=B6tzli <[email protected]>:
> Hi Elena,
>
> Thank you so much for looking into this issue. It is good to hear tha=
t you are getting the same strange result.
>
> I posted the output of /proc/pid/numa_maps here: https://gist.github.=
com/4tXJ7f/5e89f466e29cd1f7f1aa
>
> I hope this helps.
>
> Thanks again,
> Andres
>
>> On 29 Oct 2014, at 21:33, Elena Ufimtseva <[email protected]> wrot=
e:
>>
>> Hello Andres
>>
>> I looked at the example you gave, had multiple variations running an=
d
>> have same strange results.
>> The default local policy should be in use when there is no other pol=
icy defined.
>> The only thing what comes to my mind its the shared library libnuma
>> which has its data on different node then the node I try to run
>> the test process on.
>> Can you take a look and check what node is used by libnuma in
>> /proc/pid/numa_maps?
>>
>> I will keep searching for an answer, its rather interesting topic.
>> Or maybe someone else will give more details on this.
>>
>> Thank you!
>>
>>
>> On Thu, Oct 23, 2014 at 1:17 PM, Andres N=C3=B6tzli <noetzli@stanfor=
d.edu> wrote:
>>> Hi Elena,
>>>
>>> That would be great! I created a gist with the kernel config (cat /=
boot/config-$(uname -r)): https://gist.github.com/4tXJ7f/408a562abe5d4f=
28656d
>>>
>>> Please let me know if you need anything else.
>>>
>>> Thank you very much,
>>> Andres
>>>
>>>> On 23 Oct 2014, at 06:15, Elena Ufimtseva <[email protected]> wr=
ote:
>>>>
>>>> Hi Andres
>>>>
>>>> I will poke around this on the weekend on my NUMA machine.
>>>> Can you also attach your kernel config please?
>>>>
>>>> Thank you.
>>>>
>>>> On Wed, Oct 22, 2014 at 12:40 PM, Andres N=C3=B6tzli <noetzli@stan=
ford.edu> wrote:
>>>>> Hi Elena,
>>>>>
>>>>> Thank you very much for your quick reply! numa_set_strict(1) and =
numa_set_strict(0) both result in the wrong output. I did not change th=
e default policy.
>>>>>
>>>>> numa_get_membind returns 1 for all nodes before and after numa_ru=
n_on_node.
>>>>> numa_get_interleave_mask returns 0 for all nodes.
>>>>> numa_get_run_node_mask is all 1s before and 0010 after numa_run_o=
n_node.
>>>>>
>>>>> The machine config (the CPUs are all Intel(R) Xeon(R) CPU E5-4657=
L v2 @ 2.40GHz):
>>>>>
>>>>> $ numactl --hardware
>>>>> available: 4 nodes (0-3)
>>>>> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 48 49 50 51 52 53 54 55 56=
 57 58 59
>>>>> node 0 size: 262093 MB
>>>>> node 0 free: 966 MB
>>>>> node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 60 61 62 63 64 6=
5 66 67 68 69 70 71
>>>>> node 1 size: 262144 MB
>>>>> node 1 free: 82 MB
>>>>> node 2 cpus: 24 25 26 27 28 29 30 31 32 33 34 35 72 73 74 75 76 7=
7 78 79 80 81 82 83
>>>>> node 2 size: 262144 MB
>>>>> node 2 free: 102 MB
>>>>> node 3 cpus: 36 37 38 39 40 41 42 43 44 45 46 47 84 85 86 87 88 8=
9 90 91 92 93 94 95
>>>>> node 3 size: 262144 MB
>>>>> node 3 free: 113 MB
>>>>> node distances:
>>>>> node   0   1   2   3
>>>>> 0:  10  20  30  20
>>>>> 1:  20  10  20  30
>>>>> 2:  30  20  10  20
>>>>> 3:  20  30  20  10
>>>>>
>>>>> Thanks again,
>>>>> Andres
>>>>>
>>>>>> On 22 Oct 2014, at 06:12, Elena Ufimtseva <[email protected]> =
wrote:
>>>>>>
>>>>>> On Tue, Oct 21, 2014 at 11:47 PM, Andres N=C3=B6tzli <noetzli@st=
anford.edu> wrote:
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I am experiencing a weird problem. When using numa_alloc_onnode=
 repeatedly to allocate memory, it does not allocate memory on the node=
 passed as an argument.
>>>>>>>
>>>>>>> Sample code:
>>>>>>> #include <numa.h>
>>>>>>> #include <numaif.h>
>>>>>>> #include <iostream>
>>>>>>> using namespace std;
>>>>>>>
>>>>>>> void find_memory_node_for_addr(void* ptr) {
>>>>>>> int numa_node =3D -1;
>>>>>>> if(get_mempolicy(&numa_node, NULL, 0, ptr, MPOL_F_NODE | MPOL_F=
_ADDR) < 0)
>>>>>>>    cout << "WARNING: get_mempolicy failed" << endl;
>>>>>>> cout << numa_node << endl;
>>>>>>> }
>>>>>>>
>>>>>>> int main() {
>>>>>>> int64_t* x;
>>>>>>> int64_t n =3D 5000;
>>>>>>> //numa_set_preferred(1);
>>>>>>>
>>>>>>> numa_run_on_node(2);
>>>>>>> for(int i =3D 0; i < 20; i++) {
>>>>>>>    size_t s =3D n * sizeof(int64_t);
>>>>>>>    x =3D (int64_t*)numa_alloc_onnode(s, 1);
>>>>>>>    for(int j =3D 0; j < n; j++)
>>>>>>>       x[j] =3D j + i;
>>>>>>>    find_memory_node_for_addr(x);
>>>>>>> }
>>>>>>>
>>>>>>> return 0;
>>>>>>> }
>>>>>>>
>>>>>>> Output:
>>>>>>> 1
>>>>>>> 1
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>> 1
>>>>>>> 2
>>>>>>>
>>>>>>> When uncommenting the line "numa_set_preferred(1);=E2=80=9D, th=
e output is all 1s as expected. Am I doing something wrong? Have you se=
en similar issues?
>>>>>>>
>>>>>>> I am running Ubuntu 12.04.5 LTS:
>>>>>>> $ cat /proc/version
>>>>>>> Linux version 3.2.0-29-generic (buildd@allspice) (gcc version 4=
=2E6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #46-Ubuntu SMP Fri Jul 27 17:03=
:23 UTC 2012
>>>>>>>
>>>>>>> I am using libnuma 2.0.10 but I=E2=80=99ve had the same problem=
 with 2.0.8~rc3-1.
>>>>>>>
>>>>>>> Thank you very much,
>>>>>>> Andres
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux=
-numa" in
>>>>>>> the body of a message to [email protected]
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.h=
tml
>>>>>>
>>>>>> Hi Andres
>>>>>>
>>>>>> Can you try to use strict policy by calling numa_set_strict?
>>>>>>
>>>>>> If you comment out setting the preferred node, the default polic=
y is
>>>>>> in action (I assume you did no change it, not for the process, n=
ot
>>>>>> system wide) which is preferred also.
>>>>>> But here you set preferred to a specific node and manual says, t=
he
>>>>>> default for process is to allocate on the node it runs.
>>>>>> So I wonder what is the cpu affinity for this process looks like=
=2E..
>>>>>> Also maybe just to confirm you can check the policy from within =
your
>>>>>> running code?
>>>>>>
>>>>>> Can you also post the machine NUMA config?
>>>>>>
>>>>>> --
>>>>>> Elena
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Elena
>>>
>>>
>>
>>
>>
>> --
>> Elena
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-numa"=
 in
> the body of a message to [email protected]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html