Re: Contribute a RISC-V 64 JIT backend
Logan Chien <[email protected]> Thu, 15 Feb 2024 22:45:24 -0800
| Newsgroups | gmane.comp.python.pypy |
|---|---|
| Message-ID | <CALQyFuDbKishUMxBb0gPA7_SpFCRksXZ2jT5f9QkOOCk7=qwRA@mail.gmail.com> |
--===============4450406142231785322==
Content-Type: multipart/alternative; boundary="00000000000055809906117a17b8"
--00000000000055809906117a17b8
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Hi all,
I wonder if there are any tricks that can be used to debug memory
corruption?
I am debugging test_tokenize and test_zipfile64 (from lib_python_tests.py).
If I run test_zipfile64, I sometimes see this error backtrace:
```
testMoreThan64kFilesAppend (test.test_zipfile64.OtherTests) ... RPython
traceback:
File "rpython_jit_metainterp_8.c", line 35597, in CacheEntry_read
File "rpython_rtyper_lltypesystem.c", line 26925, in
ll_dict_getitem__dicttablePtr_objectPtr
memory corruption: bad size for object in the nursery
```
It is definitely related to the RISCV JIT backend which I am working on. I
tried to build a RISCV build with `-O2` and both test_tokenize and
test_zipfile64 passed without problem.
But, I can't further reduce to one of the following case:
1. Bad JIT opcode implementation that results in out-of-bound writes (thus
corrupting the heap data structure).
2. Bad gcmap calculation (thus object is being freed too early or reference
not being relocated properly)
3. Bad malloc* opcode implementation that corrupts the heap.
4. Something else.
For (3), I made two attempts:
a. I tried to skip all "fast paths" and only call the malloc_slowpath
(fixed size, str, unicode, array). But this attempt doesn't help.
b. I tried to build a JIT'ed PyPy targetstandalone.py with `--gc=3Dboehm`,
but, unfortunately, the generated C source code doesn't compile (with the
error message below).
```
pypy_module_cpyext.c: In function
'pypy_g_W_PyCTypeObject__cpyext_attach_pyobj':
pypy_module_cpyext.c:125333:9: warning: implicit declaration of
function 'OP_GC_RAWREFCOUNT_CREATE_LINK_PYOBJ'; did you mean
'OP_GC_RAWREFCOUNT_CREATE_LINK_PYPY'? [-Wimplicit-function-declaration]
125333 | OP_GC_RAWREFCOUNT_CREATE_LINK_PYOBJ(l_v451927,
l_v451928, /* nothing */);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| OP_GC_RAWREFCOUNT_CREATE_LINK_PYPY
pypy_module_cpyext.c:125333:80: error: expected expression before
')' token
125333 | OP_GC_RAWREFCOUNT_CREATE_LINK_PYOBJ(l_v451927,
l_v451928, /* nothing */);
|
^
make: *** [Makefile:762: pypy_module_cpyext.o] Error 1
make: *** Waiting for unfinished jobs....
```
So here comes my question: do we have some way to log the
allocation/marking/relocation/deallocation in the GC?
Or any other suggestions are much appreciated. Thank you.
Regards,
Logan
On Mon, Jan 29, 2024 at 6:51=E2=80=AFPM Logan Chien <tzuhsiang.chien@gmail.=
com>
wrote:
> Hi CF,
>
> Thank you for your reply.
>
> >> I also ran test_ll_random.py with `--repeat=3D20000 --random-seed=3D12=
34`
> >> and all test are passing.
> >
> > How long does that take, in wall clock time? I think for the other
> > backends we kept it running for a bunch of days after the last crash
> > occurred.
>
> It took only ~6 hrs (wall clock). If it takes a bunch of days on other
> architectures, I guess I must multiply `--repeat` by 40 times or run it o=
n
> real hardware. I'll try it again after I clear other bugs.
>
> Regards,
> Logan
>
> On Mon, Jan 29, 2024 at 12:37=E2=80=AFAM CF Bolz-Tereick via pypy-dev <
> [email protected]> wrote:
>
>> On 1/29/24 09:27, CF Bolz-Tereick via pypy-dev wrote:
>> > This test looks just wrong, in my opinion. Given that the variable nam=
e
>> > is `fff`, I think it was just meant as a check "does it roughly look
>> > like a pointer". So I think somebody just forgot that sys.maxint is no=
t
>> > a power of 2 (and then things failed on win32 and darwin and somebody
>> > fixed it with the extra if). You can change the test to always use
>> > sys.maxint*2+1.
>>
>> I went ahead and just did that change on the main branch.
>>
>> Cheers,
>>
>> CF
>> _______________________________________________
>> pypy-dev mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>> https://mail.python.org/mailman3/lists/pypy-dev.python.org/
>> Member address: [email protected]
>>
>
--00000000000055809906117a17b8
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div>Hi all,</div><div><br></div><div>I wonder if there ar=
e any tricks that can be used to debug memory corruption?</div><div><br></d=
iv><div>I am debugging test_tokenize and=C2=A0test_zipfile64 (from lib_pyth=
on_tests.py).</div><div><br></div><div>If I run test_zipfile64, I sometimes=
see this error backtrace:</div><div><br></div><div>```<br></div><div>testM=
oreThan64kFilesAppend (test.test_zipfile64.OtherTests) ... RPython tracebac=
k: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 <br>=C2=A0 File "rpython_jit_metainterp_8.c", line 355=
97, in CacheEntry_read =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0<br>=C2=A0 File "rpython_rtyper_lltypesystem.c", line 2=
6925, in ll_dict_getitem__dicttablePtr_objectPtr =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 <br>memory corruption: bad size for object in the nursery</di=
v><div>```<br></div><div><br></div><div>It is definitely related to the RIS=
CV JIT backend which I am working on.=C2=A0 I tried to build a RISCV build =
with `-O2` and both test_tokenize and test_zipfile64 passed without problem=
.</div><div><br></div><div>But, I can't further reduce to one of the fo=
llowing case:</div><div><br></div><div>1. Bad JIT opcode implementation tha=
t results in out-of-bound writes (thus corrupting the heap data structure).=
</div><div>2. Bad gcmap calculation (thus object is being freed too early o=
r reference not being relocated properly)</div><div>3. Bad malloc* opcode i=
mplementation that corrupts the heap.</div><div>4. Something else.</div><di=
v><br></div><div>For (3), I made two attempts:</div><div><br></div><div>a. =
I tried to skip all "fast paths" and only call the malloc_slowpat=
h (fixed size, str, unicode, array).=C2=A0 But this attempt doesn't hel=
p.</div><div>b. I tried to build a JIT'ed PyPy targetstandalone.py with=
`--gc=3Dboehm`, but, unfortunately, the generated C source code doesn'=
t compile (with the error message below).</div><div><br></div><div>```</div=
><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 pypy_module_cpyext.c: In function 'py=
py_g_W_PyCTypeObject__cpyext_attach_pyobj':<br>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 pypy_module_cpyext.c:125333:9: warning: implicit declaration of functio=
n 'OP_GC_RAWREFCOUNT_CREATE_LINK_PYOBJ'; did you mean 'OP_GC_RA=
WREFCOUNT_CREATE_LINK_PYPY'? [-Wimplicit-function-declaration]<br>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 125333 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 OP_GC_RAWREFC=
OUNT_CREATE_LINK_PYOBJ(l_v451927, l_v451928, /* nothing */);<br>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 OP_GC_RAWREFCOUNT_CREATE_=
LINK_PYPY<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 pypy_module_cpyext.c:125333:80: er=
ror: expected expression before ')' token<br>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 125333 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 OP_GC_RAWREFCOUNT_CREATE_LINK_P=
YOBJ(l_v451927, l_v451928, /* nothing */);<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0^<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 make: *** [Makefile:762: pypy=
_module_cpyext.o] Error 1<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 make: *** Waiting =
for unfinished jobs....<br></div><div>```</div><div><br></div><div>So here =
comes my question: do we have some way to log the allocation/marking/reloca=
tion/deallocation in the GC?</div><div><br></div><div>Or any other suggesti=
ons are much appreciated.=C2=A0 Thank you.</div><div><br></div><div>Regards=
,</div><div>Logan<br></div></div><br><div class=3D"gmail_quote"><div dir=3D=
"ltr" class=3D"gmail_attr">On Mon, Jan 29, 2024 at 6:51=E2=80=AFPM Logan Ch=
ien <<a href=3D"mailto:[email protected]">tzuhsiang.chien@gmail.=
com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e=
x"><div dir=3D"ltr"><div>Hi CF,</div><div><br></div><div>Thank you for your=
reply.</div><div><br></div><div>>> I also ran test_ll_random.py with=
`--repeat=3D20000 --random-seed=3D1234`<br>>> and all test are passi=
ng.<br>><br>> How long does that take, in wall clock time? I think fo=
r the other<br>> backends we kept it running for a bunch of days after t=
he last crash<br>> occurred.</div><div><br></div><div>It took only ~6 hr=
s (wall clock).=C2=A0 If it takes a bunch of days on other architectures, I=
guess I must multiply `--repeat` by 40 times or run it on real hardware.=
=C2=A0 I'll try it again after I clear other bugs.</div><div><br></div>=
<div>Regards,</div><div>Logan<br></div></div><br><div class=3D"gmail_quote"=
><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Jan 29, 2024 at 12:37=E2=80=
=AFAM CF Bolz-Tereick via pypy-dev <<a href=3D"mailto:[email protected]=
g" target=3D"_blank">[email protected]</a>> wrote:<br></div><blockquot=
e class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px s=
olid rgb(204,204,204);padding-left:1ex">On 1/29/24 09:27, CF Bolz-Tereick v=
ia pypy-dev wrote:<br>
> This test looks just wrong, in my opinion. Given that the variable nam=
e<br>
> is `fff`, I think it was just meant as a check "does it roughly l=
ook<br>
> like a pointer". So I think somebody just forgot that sys.maxint =
is not<br>
> a power of 2 (and then things failed on win32 and darwin and somebody<=
br>
> fixed it with the extra if). You can change the test to always use<br>
> sys.maxint*2+1.<br>
<br>
I went ahead and just did that change on the main branch.<br>
<br>
Cheers,<br>
<br>
CF<br>
_______________________________________________<br>
pypy-dev mailing list -- <a href=3D"mailto:[email protected]" target=3D"_=
blank">[email protected]</a><br>
To unsubscribe send an email to <a href=3D"mailto:[email protected]=
" target=3D"_blank">[email protected]</a><br>
<a href=3D"https://mail.python.org/mailman3/lists/pypy-dev.python.org/" rel=
=3D"noreferrer" target=3D"_blank">https://mail.python.org/mailman3/lists/py=
py-dev.python.org/</a><br>
Member address: <a href=3D"mailto:[email protected]" target=3D"_bla=
nk">[email protected]</a><br>
</blockquote></div>
</blockquote></div>
--00000000000055809906117a17b8--
--===============4450406142231785322==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
pypy-dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/pypy-dev.python.org/
Member address: [email protected]
--===============4450406142231785322==--