Re: Performance tuning and parallelisation

Marco Sirabella <[email protected]> Thu, 26 Aug 2021 09:38:15 -0700

Newsgroups	gmane.linux.gentoo.portage.devel
Message-ID	<20210826163815.vr7jtxkwuqz37wyz@Ridl3y>

Hi Ed,

I've taken a dabble at trying to track down portage's bottlenecks (and have
stopped for the time being at solving them :/ )

> Can anyone give me a leg up on how I could benchmark this further and look
> for the hotspot? Perhaps someone understand the architecture of this point
> more intimately and could point at whether there are opportunities to do some
> of the processing on mass, rather than per file?

From my notes at the timem, it looks like
[yappi](https://pypi.org/project/yappi/) worked a bit better than python's
built in cProfile for me because it properly dove into async calls. I used
[snakeviz](https://jiffyclub.github.io/snakeviz/) for visualizing the profile
results.

I was taking a look at depclean, but I found similarly that a lot of duplicate
process was being done due to encapsulated abstractions not being able to
communicate that the same thing was being done multiple times eg removing each
package processes a massive json structure for each package removed, although I
opted to work on the more-understandable unicode conversions.

My stalled progress can be found here: [#700](https://github.com/gentoo/portage/pull/700). 
Lost the drive to continue for now unfortunately :<

Good luck! Looking forward to your optimizations

-- 
Marco Sirabella

signature.asc (application/pgp-signature, 833 B)