Re: Possible regression in main causing poor performance

On Aug 18, 2023, at 19:09, Mark Millard <[email protected]> wrote:

> Glen Barber <gjb_at_freebsd.org> wrote on
> Date: Sat, 19 Aug 2023 00:10:59 UTC :
> 
>> I am somewhat inclined to look in the direction of ZFS here, as two
>> things changed:
>> 
>> 1) the build machine in question was recently (as in a week and a half
>> ago) upgraded to the tip of main in order to ease the transition from
>> this machine from building 14.x to building 15.x;
>> 2) there is the recent addition of building ZFS-backed virtual machine
>> and cloud images.
>> 
>> . . .
>> The first machine runs:
>> # uname -a
>> FreeBSD releng1.nyi.freebsd.org 14.0-CURRENT FreeBSD 14.0-CURRENT \
>> amd64 1400093 #5 main-n264224-c84617e87a70: Wed Jul 19 19:10:38 UTC 2023
> 
> I'm confused:
> 
> "the build machine in question was recently (as in a week and a half
> ago) upgraded to the tip of main in order to ease the transition from
> this machine from building 14.x to building 15.x"? But the above
> kernel is from mid July? (-aKU was not used to also get some clue
> about world from the pair of 140009? that would show.)
> 
>> Last week's snapshot builds were completed in a reasonable amount of
>> time:
>> 
>> [email protected]:/releng/scripts-snapshot/scripts # ./thermite.sh -c ./builds-14.conf ; echo ^G
>> 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/logs
>> 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/chroots
>> 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/release
>> 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/ports
>> 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/doc
>> 20230811-00:03:13 INFO: Checking out https://git.FreeBSD.org//src.git (main) to /releng/scripts-snapshot/release
>> [...]
>> 20230811-15:11:13 INFO: Staging for ftp: 14-i386-GENERIC-snap
>> 20230811-16:27:28 INFO: Staging for ftp: 14-amd64-GENERIC-snap
>> 20230811-16:33:43 INFO: Staging for ftp: 14-aarch64-GENERIC-snap
>> 
>> Overall, 17 hours, including the time to upload EC2, Vagrant, and GCE.
>> 
>> With no changes to the system, no stale ZFS datasets laying around from
>> last week (everything is a pristine environment, etc.), this week's
>> builds are taking forever:
> 
> My confusion may extend to this "no changes" status vs. the uname
> output identifying the kernel is from mid July.
> 
>> [email protected]:/releng/scripts-snapshot/scripts # ./thermite.sh -c ./builds-14.conf ; echo ^G
>> 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/logs
>> 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/chroots
>> 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/release
>> 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/ports
>> 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/doc
>> 20230818-00:15:46 INFO: Checking out https://git.FreeBSD.org//src.git (main) to /releng/scripts-snapshot/release
>> [...]
>> 20230818-18:46:22 INFO: Staging for ftp: 14-aarch64-ROCKPRO64-snap
>> 20230818-20:41:02 INFO: Staging for ftp: 14-riscv64-GENERIC-snap
>> 20230818-22:54:49 INFO: Staging for ftp: 14-amd64-GENERIC-snap
>> 
>> Note, it is just about 4 minutes past 00:00 UTC as of this writing, so
>> we are about to cross well over the 24-hour mark, and cloud provider
>> images have not yet even started.
>> 
>> . . .
> 
> In:
> 
> https://lists.freebsd.org/archives/freebsd-current/2023-August/004314.html
> ("HEADS UP: $FreeBSD$ Removed from main", Wed, 16 Aug 2023)
> 
> Warner wrote:
> 
> QUOTE
> . . . , but there's no incremental building
> with this change, . . . Also: expect long build times, git fetch times, etc
> after this.
> END QUOTE
> 
> Might this be contributing? How long did those two
> "Checking out . . ." take? Similar time frames?
> 

The build process and information is not available. So
I looked at something I thought might have a chance
of being somewhat invariant and have a limited range of
types of (parallel) activity: time differences for the
CHECKSUM files taht have timestamps after the last
*.img* timestamp, as seen via:

http://ftp3.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/14.0/?C=M&O=D
(so: most recent to oldest as displayed)

First today's:

CHECKSUM.SHA256-FreeBSD-14.0-ALPHA2-arm64-aarch64-20230819-77013f29d048-264841 1232 2023-Aug-19 00:26
CHECKSUM.SHA512-FreeBSD-14.0-ALPHA2-arm64-aarch64-20230819-77013f29d048-264841 1744 2023-Aug-19 00:25
CHECKSUM.SHA256-FreeBSD-14.0-ALPHA2-amd64-20230818-77013f29d048-264841 1168 2023-Aug-18 22:59
CHECKSUM.SHA512-FreeBSD-14.0-ALPHA2-amd64-20230818-77013f29d048-264841 1680 2023-Aug-18 22:58
CHECKSUM.SHA256-FreeBSD-14.0-ALPHA2-riscv-riscv64-20230818-77013f29d048-264841 1232 2023-Aug-18 20:49
CHECKSUM.SHA512-FreeBSD-14.0-ALPHA2-riscv-riscv64-20230818-77013f29d048-264841 1744 2023-Aug-18 20:48
CHECKSUM.SHA256-FreeBSD-14.0-ALPHA2-arm64-aarch64-ROCKPRO64-20230818-77013f29d048-264841 156 2023-Aug-18 18:47
CHECKSUM.SHA512-FreeBSD-14.0-ALPHA2-arm64-aarch64-ROCKPRO64-20230818-77013f29d048-264841 220 2023-Aug-18 18:46
FreeBSD-14.0-ALPHA2-arm64-aarch64-ROCKPRO64-20230818-77013f29d048-264841.img.xz 738514252 2023-Aug-18 18:43

So, in rough terms: RockPro64->2hrs->riscv64->2hrs->amd64->1.5hrs->aarch64


Last week's:

CHECKSUM.SHA256-FreeBSD-14.0-ALPHA1-arm64-aarch64-20230811-136fc495615f-264678 1232 2023-Aug-11 16:34
CHECKSUM.SHA512-FreeBSD-14.0-ALPHA1-arm64-aarch64-20230811-136fc495615f-264678 1744 2023-Aug-11 16:34
CHECKSUM.SHA256-FreeBSD-14.0-ALPHA1-amd64-20230811-136fc495615f-264678 1168 2023-Aug-11 16:28
CHECKSUM.SHA512-FreeBSD-14.0-ALPHA1-amd64-20230811-136fc495615f-264678 1680 2023-Aug-11 16:28
CHECKSUM.SHA256-FreeBSD-14.0-ALPHA1-i386-20230811-136fc495615f-264678 1160 2023-Aug-11 15:13
CHECKSUM.SHA512-FreeBSD-14.0-ALPHA1-i386-20230811-136fc495615f-264678 1672 2023-Aug-11 15:12
CHECKSUM.SHA256-FreeBSD-14.0-ALPHA1-riscv-riscv64-20230811-136fc495615f-264678 1232 2023-Aug-11 14:46
CHECKSUM.SHA512-FreeBSD-14.0-ALPHA1-riscv-riscv64-20230811-136fc495615f-264678 1744 2023-Aug-11 14:46
CHECKSUM.SHA256-FreeBSD-14.0-ALPHA1-arm64-aarch64-ROCKPRO64-20230811-136fc495615f-264678 156 2023-Aug-11 13:54
CHECKSUM.SHA512-FreeBSD-14.0-ALPHA1-arm64-aarch64-ROCKPRO64-20230811-136fc495615f-264678 220 2023-Aug-11 13:54
FreeBSD-14.0-ALPHA1-arm64-aarch64-ROCKPRO64-20230811-136fc495615f-264678.img.xz 738475300 2023-Aug-11 13:50

So, in rough terms: RockPro64->1hr->riscv->0.5hr->i386->1hr->amd64->6min->aarch64


Thus, 2hr vs. 1hr in crude terms.


I do not see any 2hr like time frames for 13.2 via:

http://ftp3.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/13.2/?C=M&O=D



===
Mark Millard
marklmi at yahoo.com