Re: numa_alloc_onnode does not allocate on node passed as argument

Hello Andres

I looked at the example you gave, had multiple variations running and
have same strange results.
The default local policy should be in use when there is no other policy=
 defined.
The only thing what comes to my mind its the shared library libnuma
which has its data on different node then the node I try to run
the test process on.
Can you take a look and check what node is used by libnuma in
/proc/pid/numa_maps?

I will keep searching for an answer, its rather interesting topic.
Or maybe someone else will give more details on this.

Thank you!


On Thu, Oct 23, 2014 at 1:17 PM, Andres N=C3=B6tzli <[email protected]=
du> wrote:
> Hi Elena,
>
> That would be great! I created a gist with the kernel config (cat /bo=
ot/config-$(uname -r)): https://gist.github.com/4tXJ7f/408a562abe5d4f28=
656d
>
> Please let me know if you need anything else.
>
> Thank you very much,
> Andres
>
>> On 23 Oct 2014, at 06:15, Elena Ufimtseva <[email protected]> wrot=
e:
>>
>> Hi Andres
>>
>> I will poke around this on the weekend on my NUMA machine.
>> Can you also attach your kernel config please?
>>
>> Thank you.
>>
>> On Wed, Oct 22, 2014 at 12:40 PM, Andres N=C3=B6tzli <noetzli@stanfo=
rd.edu> wrote:
>>> Hi Elena,
>>>
>>> Thank you very much for your quick reply! numa_set_strict(1) and nu=
ma_set_strict(0) both result in the wrong output. I did not change the =
default policy.
>>>
>>> numa_get_membind returns 1 for all nodes before and after numa_run_=
on_node.
>>> numa_get_interleave_mask returns 0 for all nodes.
>>> numa_get_run_node_mask is all 1s before and 0010 after numa_run_on_=
node.
>>>
>>> The machine config (the CPUs are all Intel(R) Xeon(R) CPU E5-4657L =
v2 @ 2.40GHz):
>>>
>>> $ numactl --hardware
>>> available: 4 nodes (0-3)
>>> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 48 49 50 51 52 53 54 55 56 5=
7 58 59
>>> node 0 size: 262093 MB
>>> node 0 free: 966 MB
>>> node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 60 61 62 63 64 65 =
66 67 68 69 70 71
>>> node 1 size: 262144 MB
>>> node 1 free: 82 MB
>>> node 2 cpus: 24 25 26 27 28 29 30 31 32 33 34 35 72 73 74 75 76 77 =
78 79 80 81 82 83
>>> node 2 size: 262144 MB
>>> node 2 free: 102 MB
>>> node 3 cpus: 36 37 38 39 40 41 42 43 44 45 46 47 84 85 86 87 88 89 =
90 91 92 93 94 95
>>> node 3 size: 262144 MB
>>> node 3 free: 113 MB
>>> node distances:
>>> node   0   1   2   3
>>>  0:  10  20  30  20
>>>  1:  20  10  20  30
>>>  2:  30  20  10  20
>>>  3:  20  30  20  10
>>>
>>> Thanks again,
>>> Andres
>>>
>>>> On 22 Oct 2014, at 06:12, Elena Ufimtseva <[email protected]> wr=
ote:
>>>>
>>>> On Tue, Oct 21, 2014 at 11:47 PM, Andres N=C3=B6tzli <noetzli@stan=
ford.edu> wrote:
>>>>> Hi everyone,
>>>>>
>>>>> I am experiencing a weird problem. When using numa_alloc_onnode r=
epeatedly to allocate memory, it does not allocate memory on the node p=
assed as an argument.
>>>>>
>>>>> Sample code:
>>>>> #include <numa.h>
>>>>> #include <numaif.h>
>>>>> #include <iostream>
>>>>> using namespace std;
>>>>>
>>>>> void find_memory_node_for_addr(void* ptr) {
>>>>>  int numa_node =3D -1;
>>>>>  if(get_mempolicy(&numa_node, NULL, 0, ptr, MPOL_F_NODE | MPOL_F_=
ADDR) < 0)
>>>>>     cout << "WARNING: get_mempolicy failed" << endl;
>>>>>  cout << numa_node << endl;
>>>>> }
>>>>>
>>>>> int main() {
>>>>>  int64_t* x;
>>>>>  int64_t n =3D 5000;
>>>>>  //numa_set_preferred(1);
>>>>>
>>>>>  numa_run_on_node(2);
>>>>>  for(int i =3D 0; i < 20; i++) {
>>>>>     size_t s =3D n * sizeof(int64_t);
>>>>>     x =3D (int64_t*)numa_alloc_onnode(s, 1);
>>>>>     for(int j =3D 0; j < n; j++)
>>>>>        x[j] =3D j + i;
>>>>>     find_memory_node_for_addr(x);
>>>>>  }
>>>>>
>>>>>  return 0;
>>>>> }
>>>>>
>>>>> Output:
>>>>> 1
>>>>> 1
>>>>> 1
>>>>> 2
>>>>> 1
>>>>> 2
>>>>> 1
>>>>> 2
>>>>> 1
>>>>> 2
>>>>> 1
>>>>> 2
>>>>> 1
>>>>> 2
>>>>> 1
>>>>> 2
>>>>> 1
>>>>> 2
>>>>> 1
>>>>> 2
>>>>>
>>>>> When uncommenting the line "numa_set_preferred(1);=E2=80=9D, the =
output is all 1s as expected. Am I doing something wrong? Have you seen=
 similar issues?
>>>>>
>>>>> I am running Ubuntu 12.04.5 LTS:
>>>>> $ cat /proc/version
>>>>> Linux version 3.2.0-29-generic (buildd@allspice) (gcc version 4.6=
=2E3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #46-Ubuntu SMP Fri Jul 27 17:03:2=
3 UTC 2012
>>>>>
>>>>> I am using libnuma 2.0.10 but I=E2=80=99ve had the same problem w=
ith 2.0.8~rc3-1.
>>>>>
>>>>> Thank you very much,
>>>>> Andres
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-n=
uma" in
>>>>> the body of a message to [email protected]
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.htm=
l
>>>>
>>>> Hi Andres
>>>>
>>>> Can you try to use strict policy by calling numa_set_strict?
>>>>
>>>> If you comment out setting the preferred node, the default policy =
is
>>>> in action (I assume you did no change it, not for the process, not
>>>> system wide) which is preferred also.
>>>> But here you set preferred to a specific node and manual says, the
>>>> default for process is to allocate on the node it runs.
>>>> So I wonder what is the cpu affinity for this process looks like..=
=2E
>>>> Also maybe just to confirm you can check the policy from within yo=
ur
>>>> running code?
>>>>
>>>> Can you also post the machine NUMA config?
>>>>
>>>> --
>>>> Elena
>>>
>>>
>>
>>
>>
>> --
>> Elena
>
>



--=20
Elena