Re: A recent time problem for the ports-package builders for main-amd64 [beefy18] and main-* generally

On Jul 8, 2025, at 11:18, Mark Millard <[email protected]> wrote:

> [Hopefully my classification of the below as a
> freebsd-arch issue is reasonable.]
> 
> My notes below are also on discord in the project
> infrastruture area --but with a plot that has
> essential information. The message sequence starts
> with:
> 
> https://discord.com/channels/727023752348434432/1250652091702050826/1392189829839847495
> 
> 
> The text without the plot:
> 
> For poudriere bulk runs that rebuild most everything,
> pkg 2.2.1's involvement is a small time improvement
> for the likes of main-amd64 (and likely all th
> main-*) compared to pkg 2.1.2 --but is vastly worse
> compared to pkg 2.0.6 or before. It is a much bigger
> improvement for the release-based build machines.
> My guess is that the debug vs. non-debug kernel
> usage is heavily involved in the distinctions:
> pkg 2.1+ is doing something that is rather time
> expensive in the debug kernels is my guess.
> 
> PLOT OF beefy18 main-aamd64 EXAMPLES GOES HERE:
> Elapsed Hrs vs. pkgs built
> 
> I would claim that the concave-up structure can not
> survive long term as more and more package builds
> are made in future years.
> 
> A basic problem is that packages that have any
> build/lib/run dependencies and were built toward
> the right of that graph have a
> build-depends/lib-depends/run-depends stage effort
> that involves all the previously built packages (not
> just the ones it is actually dependent on). The
> thousands of not large packages spend lots of (new)
> time in the depend stages that install packages,
> especially ones that have lots of dependencies to
> install in the builder. On beefy18 for the above,
> the last package to finish, gitlab-ce-18.1.1,
> spent over 3 hrs in the build-depends stage.
> 
> When beefy17 stops being used for main-i386, I
> wonder if the debug-kernel based testing of
> building main-amd64 packages and the
> for-distribution building of main-amd64's packages
> (with a non-debug kernel in use) could be done
> separately on separate machines.
> 
> This would not help for the slower environment:
> ampere2 for main-arm64 and main-armv7. My guess
> is that ampere2 will take about a month to cycle
> through both main-arm64 and main-armv7 to start
> over (when both are similar in scale to full
> builds of all the buildable packages). That is so
> long between updates for, say, main-arm64, that
> each update is likely rather large and may take a
> similar sort of time frame. (pkg 2.1.* being
> involved lead to over a month between, say,
> main-arm64 updates.)

Turns out that plotting:

Pkgs-Built-so-far/Elapsed-Hrs-so-far
vs. Pkgs-Built-so-far

produces a much simpler to see relationship
for when using pkg 2.2.1 and 2.1.2:

After about 15000 packages have been built,
the ratio decreases at an approximately
constant rate as more packages are built.

(That can not stay true for a sufficiently
large number of packages being built.)

See:

https://discord.com/channels/727023752348434432/1250652091702050826/1392889401834475664


===
Mark Millard
marklmi at yahoo.com