Re: continuing excessive parallelism building go

Jonathan Perkin <[email protected]> writes:

> * On 2026-05-08 at 14:43 BST, Greg Troxel wrote:
>
>>Are other people seeing this?  If you build go126, with MAKE_JOBS=2, on
>>a computer with a highish nw.ncpu, do you find that the load average
>>impact is only 2, or is it even more than MAKE_JOBS*hw.ncpu?
>
> I haven't noticed it with go, but I see it with various python builds
> still, where each will spawn ncpu build threads regardless of
> MAKE_JOBS, and as py*-foo are often built simultaneously this
> completely swamps my 32-core host for an extended period and ends up
> with something like 150 concurrent compile processes.

It sounds like python has grown unreasonably aggressive concurrency and
needs a similar fix.

>>I see that this MAKE_JOBS_SAFE=no/GOMAXPROCS pattern exists in
>>lang/go/go-module.mk.  That seems to work; I get one compile process
>>that seems to max out at 200% cpu when building e.g. hugo.  (It seems
>>that it belongs in go-package.mk also.)
>>
>>+# Respect pkgsrc concurrency limits.
>>+MAKE_JOBS_SAFE=		no
>
> Why does it need MAKE_JOBS_SAFE=no?  Notably this will cause bob
> builds that use the dynamic scheduler to be done at -j1.  I realise
> this is a "I don't care" for most people, but it still seems wrong to
> be setting that variable yet still expecting to use parallel jobs, and
> I don't think it's necessary or even correct.

I'm not really sure what you mean "done at -j1".  Do you mean bob's
scheduler will expect concurrency of 1, but it will actually be
MAKE_JOBS, and thus be higher than expectations, which is wrong?  If so,
I see your point.

I will use N for the value of MAKE_JOBS.

The basic issue is that MAKE_JOBS is supposed to control the number of
concurrent build threads.  The standard approach is that each build job
is single-threaded and make uses -jN and will allow N jobs to run.

With go, a build job will use GOMAXPROCS threads, defaulting it seems to
2*hw.cpu or certainly way too much.  Because the go build is invoked
with 'go build' instead of make, the -jN that's added with
make/ninja/meson(?) isn't added.  But if it were, with a program that
has a build system that does make-style job concurrency, you'd need to
either set the variable passed to -j1 and let GOMAXPROCS be N, or
set -jN and GOMAXPROCS=1 (or factor the number :-).

I agree that MAKE_JOBS_SAFE=no isn't quite right.  We need instead
MAKE_JOBS_DIFFERENT=yes that declines to add -j, but doesn't mark the
package for not having concurrency.

Alternatively and avoiding the work, we could

  - just set GOMAXPROCS=1 always, and let the -j mechanism work if the
    package uses make, and leave it for future work to implement -jN if
    the package doesn't use a tool that is coded for -jN type behavior.

  - don't set MAKE_JOBS_SAFE=no, after verifying by code reading that
    there won't actually be any use of -jN.  Or if there is, set
    GOMAXPROCS=1 for that package (always, because the -j mechanism has
    the token).  Consider that passing GOMAXPROCS=1 to 'go build' is how
    you spell -jN on the command line, and call it right.

and having written that I think the 2nd option is the right answer for
now.

If you don't like that, how would you do it instead?  I think it's far
more important to get this 80% right than to leave it, but obviously if
we can get to a much higher right% soon, that's better.

not tested yet, but by code reading it should be ok....


Index: Makefile
===================================================================
RCS file: /cvsroot/pkgsrc/lang/go126/Makefile,v
retrieving revision 1.1
diff -u -p -r1.1 Makefile
--- Makefile	22 Feb 2026 15:20:08 -0000	1.1
+++ Makefile	8 May 2026 14:43:45 -0000
@@ -101,9 +101,14 @@ post-extract:
 post-patch:
 	${RM} -f ${WRKSRC}/src/cmd/link/internal/x86/*.orig_dist
 
+# By default, go build invokes unreasonable parallelism.  Rather than
+# ${MAKE_JOBS} of 1 thread as with make, we run just "go build" (1
+# job) and tell it to not exceed ${MAKE_JOBS} threads.  This is sound
+# because there is no make with -jN args also.
 do-build:
 	cd ${WRKSRC}/src && \
 		env \
+		GOMAXPROCS=${MAKE_JOBS:U1} \
 		GOROOT_BOOTSTRAP=${GOROOT_BOOTSTRAP:Q} \
 		${GOOPT} \
 		GOCACHE=${WRKDIR}/.cache/go-build \