Re: Fwd: Python 3.11 performance with frame pointers
"Gregory P. Smith" <[email protected]> Wed, 4 Jan 2023 12:09:28 -0800
| Newsgroups | gmane.comp.python.devel |
|---|---|
| Message-ID | <CAGE7PN+KaiOS49bFm2_p=KAqJT4hYpJvfyw7k0Cq8_V7532vxQ@mail.gmail.com> |
--===============2705073542537817640== Content-Type: multipart/alternative; boundary="0000000000009bf23e05f175c3cb" --0000000000009bf23e05f175c3cb Content-Type: text/plain; charset="UTF-8" I suggest re-posting this on discuss.python.org as more engaged active core devs will pay attention to it there. On Wed, Jan 4, 2023 at 11:12 AM Daan De Meyer <[email protected]> wrote: > Hi, > > As part of the proposal to enable frame pointers by default in Fedora > (https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer), we > did some benchmarking to figure out the expected performance impact. > The performance impact was generally minimal, except for the > pyperformance benchmark suite where we noticed a more substantial > difference between a system built with frame pointers and a system > built without frame pointers. The results can be found here: > https://github.com/DaanDeMeyer/fpbench (look at the mean difference > column for the pyperformance results where the percentage is the > slowdown compared to a system built without frame pointers). One of > the biggest slowdowns was on the scimark_sparse_mat_mult benchmark > which slowed down 9.5% when the system (including python) was built > with frame pointers. Note that these benchmarks were run against > Python 3.11 on a Fedora 37 x86_64 system (one built with frame > pointers, another built without frame pointers). The system used to > run the benchmarks was an Amazon EC2 machine. > > We did look a bit into the reasons behind this slowdown. I'll quote > the investigation by Andrii on the Fesco issue thread here > (https://pagure.io/fesco/issue/2817): > > > So I did look a bit at Python with and without frame pointers trying to > > understand pyperformance > regressions. > > > First, perf data suggests that big chunk of CPU is spent in > _PyEval_EvalFrameDefault, > > so I looked specifically into it (also we had to use DWARF mode for > perf for apples-to-apples > > comparison, and a bunch of stack traces weren't symbolized properly, > which just again > > reminds why having frame pointers is important). > > > perf annotation of _PyEval_EvalFrameDefault didn't show any obvious hot > spots, the work > > seemed to be distributed pretty similarly with or without frame > pointers. Also scrolling through > > _PyEval_EvalFrameDefault disassembly also showed that instruction > patterns between fp > > and no-fp versions are very similar. > > > But just a few interesting observations. > > > The size of _PyEval_EvalFrameDefault function specifically (and all the > other functions didn't > > change much in that regard) increased very significantly from 46104 to > 53592 bytes, which is a > > considerable 15% increase. Looking deeper, I believe it's all due to > more stack spills and > > reloads due to one less register available to keep local variables in > registers instead of on the stack. > > > Looking at _PyEval_EvalFrameDefault C code, it is a humongous one > function with gigantic switch > > statement that implements Python instruction handling logic. So the > function itself is big and it has > > a lot of local state in different branches, which to me explained why > there is so much stack spill/load. > > > Grepping for instruction of the form mov -0xf0(%rbp),%rcx or mov > 0x50(%rsp),%r10 (and their reverse > > variants), I see that there is a substantial amount of stack spill/load > in _PyEval_EvalFrameDefault > > disassembly already in default no frame pointer variant (1870 out of > 11181 total instructions in that > > function, 16.7%), and it just increases further in frame pointer version > (2341 out of 11733 instructions, 20%). > > > One more interesting observation. With no frame pointers, GCC generates > stack accesses using %rsp > > with small positive offsets, which results in pretty compact binary > instruction representation, e.g.: > > > 0x00000000001cce40 <+44160>: 4c 8b 54 24 50 mov > 0x50(%rsp),%r10 > > > This uses 5 bytes. But if frame pointers are enabled, GCC switches to > using %rbp-relative offsets, > > which are all negative. And that seems to result in much bigger > instructions, taking now 7 bytes instead of 5: > > > 0x00000000001d3969 <+53065>: 48 8b 8d 10 ff ff ff mov > -0xf0(%rbp),%rcx > > > I found it pretty interesting. I'd imagine GCC should be capable to keep > using %rsp addressing just fine > > regardless of %rbp and save on instruction sizes, but apparently it > doesn't. Not sure why. But this instruction > > increase, coupled with increase of number of spills/reloads, actually > explains huge increase in byte size of > > _PyEval_EvalFrameDefault: (2341 - 1870) * 7 + 1870 * 2 = 7037 (2 extra > bytes for existing 1870 instructions > > that were switched from %rsp+positive offset to %rbp + negative offset, > plus 7 bytes for each of new 471 instructions). > > I'm no compiler expert, but it would be nice for someone from GCC > community to check this as well (please CC > > relevant folks, if you know them). > > > In summary, to put it bluntly, there is just more work to do for CPU > saving/restoring state to/from stack. But I don't > > think _PyEval_EvalFrameDefault example is typical of how application > code is written, nor is it, generally speaking, > > a good idea to do so much within single gigantic function. So I believe > it's more of an outlier than a typical case. > > We have a few questions: > - Is this slowdown when Python is built with frame pointers to be > expected? Has the Python community done any of their own experiments > with building Python with and without frame pointers? > - Is there anything we can do to fix the slowdown when Python is built > with frame pointers? > - Should we expect any change in benchmark results if we benchmark > against Python 3.12? Supposedly there are changes in Python 3.12 > related to frame pointers so we're wondering if those changes might > affect these results in any way. > > Cheers, > > Daan De Meyer > _______________________________________________ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/LVRUY7KAJ5I532NHMDWJIS5H4HXSGBWD/ > Code of Conduct: http://python.org/psf/codeofconduct/ > --0000000000009bf23e05f175c3cb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">I suggest re-posting this on <a href=3D"http://discuss.pyt= hon.org">discuss.python.org</a> as more engaged active core devs will pay a= ttention to it there.</div><br><div class=3D"gmail_quote"><div dir=3D"ltr" = class=3D"gmail_attr">On Wed, Jan 4, 2023 at 11:12 AM Daan De Meyer <<a h= ref=3D"mailto:[email protected]">[email protected]</a>> wr= ote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px= 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br> <br> As part of the proposal to enable frame pointers by default in Fedora<br> (<a href=3D"https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer" = rel=3D"noreferrer" target=3D"_blank">https://fedoraproject.org/wiki/Changes= /fno-omit-frame-pointer</a>), we<br> did some benchmarking to figure out the expected performance impact.<br> The performance impact was generally minimal, except for the<br> pyperformance benchmark suite where we noticed a more substantial<br> difference between a system built with frame pointers and a system<br> built without frame pointers. The results can be found here:<br> <a href=3D"https://github.com/DaanDeMeyer/fpbench" rel=3D"noreferrer" targe= t=3D"_blank">https://github.com/DaanDeMeyer/fpbench</a> (look at the mean d= ifference<br> column for the pyperformance results where the percentage is the<br> slowdown compared to a system built without frame pointers). One of<br> the biggest slowdowns was on the scimark_sparse_mat_mult benchmark<br> which slowed down 9.5% when the system (including python) was built<br> with frame pointers. Note that these benchmarks were run against<br> Python 3.11 on a Fedora 37 x86_64 system (one built with frame<br> pointers, another built without frame pointers). The system used to<br> run the benchmarks was an Amazon EC2 machine.<br> <br> We did look a bit into the reasons behind this slowdown. I'll quote<br> the investigation by Andrii on the Fesco issue thread here<br> (<a href=3D"https://pagure.io/fesco/issue/2817" rel=3D"noreferrer" target= =3D"_blank">https://pagure.io/fesco/issue/2817</a>):<br> <br> > So I did look a bit at Python with and without frame pointers trying t= o<br> > understand pyperformance > regressions.<br> <br> > First, perf data suggests that big chunk of CPU is spent in _PyEval_Ev= alFrameDefault,<br> > so I looked specifically=C2=A0 into it (also we had to use DWARF mode = for perf for apples-to-apples<br> > comparison, and a bunch of stack traces weren't symbolized properl= y, which just again<br> > reminds why having frame pointers is important).<br> <br> > perf annotation of _PyEval_EvalFrameDefault didn't show any obviou= s hot spots, the work<br> > seemed to be distributed pretty similarly with or without frame pointe= rs. Also scrolling through<br> > _PyEval_EvalFrameDefault disassembly also showed that instruction patt= erns between fp<br> > and no-fp versions are very similar.<br> <br> > But just a few interesting observations.<br> <br> > The size of _PyEval_EvalFrameDefault function specifically (and all th= e other functions didn't<br> > change much in that regard) increased very significantly from 46104 to= 53592 bytes, which is a<br> > considerable 15% increase. Looking deeper, I believe it's all due = to more stack spills and<br> > reloads due to one less register available to keep local variables in = registers instead of on the stack.<br> <br> > Looking at _PyEval_EvalFrameDefault C code, it is a humongous one func= tion with gigantic switch<br> > statement that implements Python instruction handling logic. So the fu= nction itself is big and it has<br> > a lot of local state in different branches, which to me explained why = there is so much stack spill/load.<br> <br> > Grepping for instruction of the form mov -0xf0(%rbp),%rcx or mov 0x50(= %rsp),%r10 (and their reverse<br> > variants), I see that there is a substantial amount of stack spill/loa= d in _PyEval_EvalFrameDefault<br> > disassembly already in default no frame pointer variant (1870 out of 1= 1181 total instructions in that<br> > function, 16.7%), and it just increases further in frame pointer versi= on (2341 out of 11733 instructions, 20%).<br> <br> > One more interesting observation. With no frame pointers, GCC generate= s stack accesses using %rsp<br> > with small positive offsets, which results in pretty compact binary in= struction representation, e.g.:<br> <br> > 0x00000000001cce40 <+44160>: 4c 8b 54 24 50=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 mov=C2=A0 =C2=A0 0x50(%rsp),%r10<br> <br> > This uses 5 bytes. But if frame pointers are enabled, GCC switches to = using %rbp-relative offsets,<br> > which are all negative. And that seems to result in much bigger instru= ctions, taking now 7 bytes instead of 5:<br> <br> > 0x00000000001d3969 <+53065>: 48 8b 8d 10 ff ff ff=C2=A0 =C2=A0 m= ov=C2=A0 =C2=A0 -0xf0(%rbp),%rcx<br> <br> > I found it pretty interesting. I'd imagine GCC should be capable t= o keep using %rsp addressing just fine<br> > regardless of %rbp and save on instruction sizes, but apparently it do= esn't. Not sure why. But this instruction<br> > increase, coupled with increase of number of spills/reloads, actually = explains huge increase in byte size of<br> > _PyEval_EvalFrameDefault: (2341 - 1870) * 7 + 1870 * 2 =3D 7037 (2 ext= ra bytes for existing 1870 instructions<br> > that were switched from %rsp+positive offset to %rbp + negative offset= , plus 7 bytes for each of new 471 instructions).<br> > I'm no compiler expert, but it would be nice for someone from GCC = community to check this as well (please CC<br> > relevant folks, if you know them).<br> <br> > In summary, to put it bluntly, there is just more work to do for CPU s= aving/restoring state to/from stack. But I don't<br> > think _PyEval_EvalFrameDefault example is typical of how application c= ode is written, nor is it, generally speaking,<br> > a good idea to do so much within single gigantic function. So I believ= e it's more of an outlier than a typical case.<br> <br> We have a few questions:<br> - Is this slowdown when Python is built with frame pointers to be<br> expected? Has the Python community done any of their own experiments<br> with building Python with and without frame pointers?<br> - Is there anything we can do to fix the slowdown when Python is built<br> with frame pointers?<br> - Should we expect any change in benchmark results if we benchmark<br> against Python 3.12? Supposedly there are changes in Python 3.12<br> related to frame pointers so we're wondering if those changes might<br> affect these results in any way.<br> <br> Cheers,<br> <br> Daan De Meyer<br> _______________________________________________<br> Python-Dev mailing list -- <a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= rg" target=3D"_blank">[email protected]</a><br> <a href=3D"https://mail.python.org/mailman3/lists/python-dev.python.org/" r= el=3D"noreferrer" target=3D"_blank">https://mail.python.org/mailman3/lists/= python-dev.python.org/</a><br> Message archived at <a href=3D"https://mail.python.org/archives/list/python= [email protected]/message/LVRUY7KAJ5I532NHMDWJIS5H4HXSGBWD/" rel=3D"noreferre= r" target=3D"_blank">https://mail.python.org/archives/list/python-dev@pytho= n.org/message/LVRUY7KAJ5I532NHMDWJIS5H4HXSGBWD/</a><br> Code of Conduct: <a href=3D"http://python.org/psf/codeofconduct/" rel=3D"no= referrer" target=3D"_blank">http://python.org/psf/codeofconduct/</a><br> </blockquote></div> --0000000000009bf23e05f175c3cb-- --===============2705073542537817640== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline