Re: Building with many cores without OOM

Helmut Grohne:
> Hi Guillem and other developers,
> 
> I am one of those who builds a lot of different packages with different
> requirements and found that picking a good parallel=... value in
> DEB_BUILD_OPTIONS is hard. Go too low and your build takes very long. Go
> too high and you swap until the OOM killer terminates your build. (Usage
> of choom recommended in any case.)
> 
> [...]
> 
> I think this demonstrates that we probably have something between 10 and
> 50 packages in unstable that would benefit from a generic parallelism
> limit based on available RAM. Do others agree that this is a problem
> worth solving in a more general way?
> 
> For one thing, I propose extending debhelper to provide
> --min-ram-per-parallel-core as that seems to be the most common way to
> do it. I've proposed
> https://salsa.debian.org/debian/debhelper/-/merge_requests/128
> to this end.
> 
> Unfortunately, a the affected packages tend to not just be big, but also
> so special that they cannot use dh_auto_*. As a result, I also looked at
> another layer to support this and found /usr/share/dpkg/buildopts.mk,
> which sets DEB_BUILD_OPTION_PARALLEL by parsing DEB_BUILD_OPTIONS. How
> about extending this file with a mechanism to reduce parallelity? I am
> attaching a possible extension to it to this mail to see what you think.
> Guillem, is that something you consider including in dpkg?
> 

My suggestion would be to have `dpkg` have a "RAM restrained" 
parallelization limit next to the default one. This is similar to how 
the `debhelper` one works (the option only applies to the dh_auto_ 
steps). This might be what you are proposing here (was unsure).

Generally, the RAM limit only applies to the upstream build side of 
things and this can be relevant for some packages. As an example, 
`firefox-esr` builds 20+ packages, so there is value in having "post 
processing" (dh_install ... dh_builddeb) happening with a higher degree 
of parallelization than the upstream build part to keep the build time 
as low as possible.

I think 2 degrees of parallelization limits would be sufficient for most 
cases (at least from a cost/benefit PoV).

> Are there other layers that could reasonably be used to implement a more
> general form of parallelism limiting based on system RAM? Ideally, we'd
> consolidate these implementations into fewer places.
> 
> [...]
> 
> Helmut

We do have some custom implementations of debhelper build systems around 
in the archive[1], that in theory could do with this (though they are 
probably not worth hunting out - more of "update if they become a 
problem"). Then there is `debputy`, but I can have a look at that later 
after I have reviewed the patch for `debhelper`.

Best regards,
Niels

[1] Any `dh-sequence-X` sequences that replace `dh_auto_*` commands 
would likely fall into this category.

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEE9ecZmu9eXGflVYc/dA1oiINl0okFAmdJyeIACgkQdA1oiINl
0olVrggArjhwjfYELAes00+fi/QZY0w5esCv2GY3Y70bD63iR/w99UQRtw2iYs9R
lkkLDWDzL4RN75AM1Ip+Q9wdAoFLWp14szr1xj86C1QFE1NUgCAz08Pkn/5lIu7G
yuqRyv3nBE0tKLOGjwSdNwMhRCDKDJ8F9+TzB9NqDG/pbdtspQwleJCnQvQnPVL4
Ajmg1vQwYajAPwbs3m4bHM3tzwMUF3OskEMSl0jMrF+GyEbB3PG0D4aEekRtBSjG
BggekmFiadonJzsnmPfkq3hwVOSJH1cPeUd/UZrscexA9junVkRn/opxFpllrwQW
L1CZnbtf+Nt2Bve7PsQ8A8PujrAFzg==
=7rkq
-----END PGP SIGNATURE-----