[bug #42288] limit parallelism based on available memory

--8323329-1350490027-1778993392=:1696394
Content-Type: TEXT/plain; CHARSET=utf-8
Content-Transfer-Encoding: QUOTED-PRINTABLE
Content-Disposition: inline

Follow-up Comment #5, bug #42288 (group make):=0A=0AI like the -l memory_li=
mit idea.  If less than memory_limit RAM is available,=0Adon't start new jo=
bs.=0A=0AI think we should focus on how much RAM is free on the system.  It=
 doesn't=0Amatter how much jobs use, nor how much memory other processes on=
 the system=0Ause.  What matters is whether there's enough space available =
now to start a=0Anew job.=0A=0AI can confirm that when I talk about memory,=
 I am not talking about swap at=0Aall.  I'm talking about RAM.=0A=0AI don't=
 know about you, but as long as there is memory for all the processes,=0AI =
don't care if my CPU load is at 1.00 or 100.00.  I already have niceness,=
=0Aand io-niceness, with which to control that pile of work.  Also, that pi=
le of=0Awork is work that must be done.  Any CPU, memory, or disk required =
by that=0Awork is part of the job.=0A=0AWhat is not required is virtual mem=
ory swapping.  Any time swapping occurs the=0Asystem is doing way more work=
 than is necessary, and is using the slowest=0Ahardware to do it: disk.=0A=
=0ATherefore the primary goal of this feature is to avoid swap if possible,=
 and=0Ato stop thrashing if not.=0A=0AIf we take an example Linux system wi=
th no swap, then the OOM killer will stop=0Aprocessing completely.  A smart=
 make could restart any jobs that were killed=0Athat way, but then we run i=
nto the same problem: doing work that is=0Aunnecessary, by doing it twice.=
=0A=0A=0AWays to limit memory usage:=0A=0A1. Some systems have ulimits, bot=
h soft and hard.  I've never managed to make=0Athis work practically.  If a=
 job needs the resources, it just won't complete=0Awithout them, so it's a =
matter of juggling or priority rather than setting=0Ahard limits.=0A2. The =
-l limit feature: don't start a process unless the current amount of=0Afree=
 RAM is at an expected threshold.=0A=0A=0ABoth of these techniques still le=
ave open the possibility of thrashing in the=0Acase where N jobs are starte=
d and they begin using so much memory that=0Aswapping begins.=0A=0AWhat is =
thrashing?  I define it as switching back and forth between two or=0Amore p=
rocesses that are both stuck in swap.  The way to stop thrashing once it=0A=
starts is to stop all such processes but one, so that swapping actually hel=
ps=0Aone process to finish.=0A=0AThere are two ways to do this:=0A=0A1. Rel=
y on the OOM killer, or have make itself kill and restart the job.=0A2. Put=
 all but one job (the one using the most memory) to sleep with SIGSTOP,=0Ar=
estoring them slowly one at a time with SIGCONT until the system has -l lim=
it=0Amemory free again.=0A=0AOption 2 prevents thrashing, allowing the syst=
em to shift the entire excess=0Ajob load into swap if necessary in order to=
 finish the largest one.  Recovery=0Aof that overloaded situation is done g=
radually by waking them up one by one=0Auntil load and memory usage are bac=
k in normal range.=0A=0ABoth options give a clear, finite path to job compl=
etion, while thrashing, if=0Aleft unattended, may not finish in any reasona=
ble timeframe.=0A=0AI have done this manually, running top in memory sorted=
 mode, and stopping=0Acompilers as needed in order to finish, and then rest=
arting them when=0Aresources free up.  If this were automated, I think it c=
ould work.=0A=0AIn the end, the Make user would set -l to the approximate m=
emory size of the=0Aaverage job, perhaps slightly larger.  Make would then =
take care of the rest.=0AAs long as there was enough swap for (N-1)*memory_=
limit bytes, then -jN is=0Asomething that would be safe to walk away from a=
nd allow Make to monitor and=0Afinish.=0A=0A=0A=0A    _____________________=
__________________________________=0A=0AReply to this item at:=0A=0A  <http=
s://savannah.gnu.org/bugs/?42288>=0A=0A____________________________________=
___________=0AMessage sent via Savannah=0Ahttps://savannah.gnu.org/=0A
--8323329-1350490027-1778993392=:1696394
Content-Type: APPLICATION/pgp-signature; name=signature.asc

-----BEGIN PGP SIGNATURE-----

iHUEABYIAB0WIQQk97aszIMMAvLLwm6qLAuaBUf3TgUCaglI8AAKCRCqLAuaBUf3
TgN+AP4uTUl9PHtkkIsEsDpR+Y8hMo4Sao9UBRd94NMs+X9fDwD/ZjJlOwIlmaFO
uUDWFJl0KFaWD21CKfC5py/JSI4fbgw=
=1Lu/
-----END PGP SIGNATURE-----

--8323329-1350490027-1778993392=:1696394--