Re: numa_alloc_onnode does not allocate on node passed as argument

Hi Elena,

Thank you so much for looking into this issue. It is good to hear that =
you are getting the same strange result.

I posted the output of /proc/pid/numa_maps here: https://gist.github.co=
m/4tXJ7f/5e89f466e29cd1f7f1aa

I hope this helps.

Thanks again,
Andres

> On 29 Oct 2014, at 21:33, Elena Ufimtseva <[email protected]> wrote=
:
>=20
> Hello Andres
>=20
> I looked at the example you gave, had multiple variations running and
> have same strange results.
> The default local policy should be in use when there is no other poli=
cy defined.
> The only thing what comes to my mind its the shared library libnuma
> which has its data on different node then the node I try to run
> the test process on.
> Can you take a look and check what node is used by libnuma in
> /proc/pid/numa_maps?
>=20
> I will keep searching for an answer, its rather interesting topic.
> Or maybe someone else will give more details on this.
>=20
> Thank you!
>=20
>=20
> On Thu, Oct 23, 2014 at 1:17 PM, Andres N=C3=B6tzli <noetzli@stanford=
=2Eedu> wrote:
>> Hi Elena,
>>=20
>> That would be great! I created a gist with the kernel config (cat /b=
oot/config-$(uname -r)): https://gist.github.com/4tXJ7f/408a562abe5d4f2=
8656d
>>=20
>> Please let me know if you need anything else.
>>=20
>> Thank you very much,
>> Andres
>>=20
>>> On 23 Oct 2014, at 06:15, Elena Ufimtseva <[email protected]> wro=
te:
>>>=20
>>> Hi Andres
>>>=20
>>> I will poke around this on the weekend on my NUMA machine.
>>> Can you also attach your kernel config please?
>>>=20
>>> Thank you.
>>>=20
>>> On Wed, Oct 22, 2014 at 12:40 PM, Andres N=C3=B6tzli <noetzli@stanf=
ord.edu> wrote:
>>>> Hi Elena,
>>>>=20
>>>> Thank you very much for your quick reply! numa_set_strict(1) and n=
uma_set_strict(0) both result in the wrong output. I did not change the=
 default policy.
>>>>=20
>>>> numa_get_membind returns 1 for all nodes before and after numa_run=
_on_node.
>>>> numa_get_interleave_mask returns 0 for all nodes.
>>>> numa_get_run_node_mask is all 1s before and 0010 after numa_run_on=
_node.
>>>>=20
>>>> The machine config (the CPUs are all Intel(R) Xeon(R) CPU E5-4657L=
 v2 @ 2.40GHz):
>>>>=20
>>>> $ numactl --hardware
>>>> available: 4 nodes (0-3)
>>>> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 48 49 50 51 52 53 54 55 56 =
57 58 59
>>>> node 0 size: 262093 MB
>>>> node 0 free: 966 MB
>>>> node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 60 61 62 63 64 65=
 66 67 68 69 70 71
>>>> node 1 size: 262144 MB
>>>> node 1 free: 82 MB
>>>> node 2 cpus: 24 25 26 27 28 29 30 31 32 33 34 35 72 73 74 75 76 77=
 78 79 80 81 82 83
>>>> node 2 size: 262144 MB
>>>> node 2 free: 102 MB
>>>> node 3 cpus: 36 37 38 39 40 41 42 43 44 45 46 47 84 85 86 87 88 89=
 90 91 92 93 94 95
>>>> node 3 size: 262144 MB
>>>> node 3 free: 113 MB
>>>> node distances:
>>>> node   0   1   2   3
>>>> 0:  10  20  30  20
>>>> 1:  20  10  20  30
>>>> 2:  30  20  10  20
>>>> 3:  20  30  20  10
>>>>=20
>>>> Thanks again,
>>>> Andres
>>>>=20
>>>>> On 22 Oct 2014, at 06:12, Elena Ufimtseva <[email protected]> w=
rote:
>>>>>=20
>>>>> On Tue, Oct 21, 2014 at 11:47 PM, Andres N=C3=B6tzli <noetzli@sta=
nford.edu> wrote:
>>>>>> Hi everyone,
>>>>>>=20
>>>>>> I am experiencing a weird problem. When using numa_alloc_onnode =
repeatedly to allocate memory, it does not allocate memory on the node =
passed as an argument.
>>>>>>=20
>>>>>> Sample code:
>>>>>> #include <numa.h>
>>>>>> #include <numaif.h>
>>>>>> #include <iostream>
>>>>>> using namespace std;
>>>>>>=20
>>>>>> void find_memory_node_for_addr(void* ptr) {
>>>>>> int numa_node =3D -1;
>>>>>> if(get_mempolicy(&numa_node, NULL, 0, ptr, MPOL_F_NODE | MPOL_F_=
ADDR) < 0)
>>>>>>    cout << "WARNING: get_mempolicy failed" << endl;
>>>>>> cout << numa_node << endl;
>>>>>> }
>>>>>>=20
>>>>>> int main() {
>>>>>> int64_t* x;
>>>>>> int64_t n =3D 5000;
>>>>>> //numa_set_preferred(1);
>>>>>>=20
>>>>>> numa_run_on_node(2);
>>>>>> for(int i =3D 0; i < 20; i++) {
>>>>>>    size_t s =3D n * sizeof(int64_t);
>>>>>>    x =3D (int64_t*)numa_alloc_onnode(s, 1);
>>>>>>    for(int j =3D 0; j < n; j++)
>>>>>>       x[j] =3D j + i;
>>>>>>    find_memory_node_for_addr(x);
>>>>>> }
>>>>>>=20
>>>>>> return 0;
>>>>>> }
>>>>>>=20
>>>>>> Output:
>>>>>> 1
>>>>>> 1
>>>>>> 1
>>>>>> 2
>>>>>> 1
>>>>>> 2
>>>>>> 1
>>>>>> 2
>>>>>> 1
>>>>>> 2
>>>>>> 1
>>>>>> 2
>>>>>> 1
>>>>>> 2
>>>>>> 1
>>>>>> 2
>>>>>> 1
>>>>>> 2
>>>>>> 1
>>>>>> 2
>>>>>>=20
>>>>>> When uncommenting the line "numa_set_preferred(1);=E2=80=9D, the=
 output is all 1s as expected. Am I doing something wrong? Have you see=
n similar issues?
>>>>>>=20
>>>>>> I am running Ubuntu 12.04.5 LTS:
>>>>>> $ cat /proc/version
>>>>>> Linux version 3.2.0-29-generic (buildd@allspice) (gcc version 4.=
6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #46-Ubuntu SMP Fri Jul 27 17:03:23=
 UTC 2012
>>>>>>=20
>>>>>> I am using libnuma 2.0.10 but I=E2=80=99ve had the same problem =
with 2.0.8~rc3-1.
>>>>>>=20
>>>>>> Thank you very much,
>>>>>> Andres
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-=
numa" in
>>>>>> the body of a message to [email protected]
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.ht=
ml
>>>>>=20
>>>>> Hi Andres
>>>>>=20
>>>>> Can you try to use strict policy by calling numa_set_strict?
>>>>>=20
>>>>> If you comment out setting the preferred node, the default policy=
 is
>>>>> in action (I assume you did no change it, not for the process, no=
t
>>>>> system wide) which is preferred also.
>>>>> But here you set preferred to a specific node and manual says, th=
e
>>>>> default for process is to allocate on the node it runs.
>>>>> So I wonder what is the cpu affinity for this process looks like.=
=2E.
>>>>> Also maybe just to confirm you can check the policy from within y=
our
>>>>> running code?
>>>>>=20
>>>>> Can you also post the machine NUMA config?
>>>>>=20
>>>>> --
>>>>> Elena
>>>>=20
>>>>=20
>>>=20
>>>=20
>>>=20
>>> --
>>> Elena
>>=20
>>=20
>=20
>=20
>=20
> --=20
> Elena