Re: Building with many cores without OOM

Hi,

Let me plop in right into this discussion with no general solution and 
more things to think about. For the context I'm packaging Java things, 
and Java has historically been notoriously bad at guessing how much 
memory it could actually use on a given system. I'm not sure things are 
much better these days. This is just to remind that the issue is nowhere 
near as easy as it looks and that many attempts to generalize an 
approach that works well in some cases have failed.

Le 2024-12-09 14:42, Guillem Jover a écrit :
> 
> My thinking here was also about the general case too, say a system
> that has many cores relative to its available memory, where each core
> would get what we'd consider not enough memory per core

This is actually a common situation on most systems but a few privileged 
developers configurations. This is especially true in cloud-like, 
VM/containerized environments where it is much easier (i.e. with less 
immediate consequences) to overcommit CPU cores than RAM. Just look at 
the price list of any cloud computing provider to get an idea of the 
ratios you could start with. And then the provider may well lie about 
the actual availability of the cores they will readily bill you for, and 
you will only notice that when your application will grind to a halt at 
the worst possible time (e.g. on a Black Friday if your business is to 
sell stuff), but at least it won't get OOM-killed.

There are a few packages that are worrying me about how I'm going to 
make them build and run their test suites on Salsa without either timing 
out on one side, and getting immediately OOM killed at the other end of 
the slider. One of them wants to allocate 17GiB of RAM per test worker, 
and wants at least 3 of them. Another (Gradle) needs approximately 4 GiB 
of RAM (JVM processes alone, adding OS cache + overhead to that probably 
makes the total around 6-7 GiB) per additional worker for its build, and 
I don't know yet how much is needed for its tests suites as my current 
setup lacks the storage space necessary to run them. On my current 
low-end laptop (4 threads, 16 GiB RAM) dpkg guesses [1] are wrong , I 
can only run a single worker if I want to keep an IDE and a web browser 
running on the side. Two if I close the IDE and kill all browser tabs 
and other memory hogs. I would expect FTBFS bug reports if a 
run-of-the-mill dpkg-buildpackage command failed to build the package on 
such a system.

> (assuming for
> example a baseline for what dpkg-deb might require, plus build helpers
> and their interpreters, and what a compiler with say an empty C, C++
> or similar file might need, etc).

+1 for taking a baseline into consideration, as the first worker is 
usually significantly more expensive than additional workers. In my 
experience with Java build processes the first worker penalty is in the 
vicinity of +35% and can be much higher for lighter build processes (but 
then they are lighter and less likely to hit a limit excepted on very 
constrained environments).

Another thing I would like to add is that the requirements may change 
depending on the phase of the build, especially between building and 
testing. For larger projects, building requires usually more memory but 
less parallelism than testing. You could always throw more workers at 
building, but at some point additional workers will just sit mostly idle 
consuming RAM and resources as there is a limited number of tasks that 
the critical path will allow at any given point. Testing, especially 
with larger test suites, usually allows for (and sometimes needs) much 
more parallelism.

Also worth noting, on some projects the time spent testing can be orders 
of magnitude greater than the time spent building.

> This could also imply alternatively or in addition, providing a tool
> or adding some querying logic in an existing tools (in the dpkg 
> toolset)
> to gather that information which the packaging could use, or…

Additional tooling may help a bit, but I think what would really help at 
that point would be to write and publish guidelines relevant to the 
technology being packaged, based on empirical evidence collected while 
fine tuning the build or packaging, and kept reasonably up-to-date (i.e. 
never more than 2-3 years old) with the current state of technologies 
and projects. Salsa (or other CI) pipelines could be instrumented to 
provide some data and once the guidelines cover a majority of packages 
you will have a better insight of what, if anything, needs to be done 
with the tooling.


[1]: 
https://salsa.debian.org/jpd/gradle/-/blob/upgrade-to-8.11.1-wip/debian/rules#L49