Re: Very long-running buildworld process

On Tue, Jun 10, 2025 at 10:24:26PM -0700, Mark Millard wrote:
> On Jun 10, 2025, at 20:59, bob prohaska <[email protected]> wrote:
> 
> > Buildworld for -current on a Pi2B is strangely slow. The machine is responsive,
> > but seems to make no (or very slow) progress:
> > 
> > 
> > login: Jun  7 22:51:31 www su[3032]: bob to root on /dev/pts/0
> > 
> > 
> > FreeBSD/arm (www.zefox.org) (ttyu0)
> > 
> > login: bob
> > Password:
> > Last login: Sat Jun  7 22:51:25 from gateway.zefox.net
> > FreeBSD 15.0-CURRENT #85 main-d8773fdcbfa3: Fri Jun  6 14:17:58 PDT 2025     [email protected]:/usr/obj/usr/src/arm.armv7/sys/GENERIC
> > 
> > Welcome to FreeBSD!
> > 
> > Release Notes, Errata: https://www.FreeBSD.org/releases/
> > Security Advisories:   https://www.FreeBSD.org/security/
> > last pid: 34625;  load averages:    1.01,    1.01,    1.00                                                             up 2+22:03:17  20:45:47
> > last pid: 34628;  load averages:    1.07,    1.02,    1.01                                                             up 2+22:05:21  20:47:51
> > 45 processes:  2 running, 43 sleeping
> > CPU: 25.0% user,  0.0% nice,  0.0% system,  0.0% interrupt, 75.0% idle
> > Mem: 6196K Active, 297M Inact, 174M Wired, 98M Buf, 443M Free
> > Swap: 1770M Total, 422M Used, 1348M Free, 23% Inuse
> 
> Free  443M
> Inact 297M
> Wired 174M
> 
> Yet:
> 
> c++ RES is just 17M and Active is just 6196K.
> 
> Also: 422M SWAP used.
> 
> I suggest sorting the top output by (decreasing) RES and
> seeing if that shows what is contributing to Inact 297M
> as part of RESident memory use.
> 

Changing order to res yields in part:
last pid: 37994;  load averages:    1.01,    1.01,    1.00                                                             up 3+07:55:55  06:38:25
45 processes:  2 running, 43 sleeping
CPU: 24.6% user,  0.0% nice,  0.3% system,  0.0% interrupt, 75.2% idle
Mem: 5672K Active, 307M Inact, 150M Wired, 73M Buf, 459M Free
Swap: 1770M Total, 422M Used, 1348M Free, 23% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
21654 root          1 135    0   561M    17M CPU2     2  49.6H 100.12% c++
37994 bob           1  20    0  6664K  3088K CPU3     3   0:00   0.19% top
 2951 root          1  20    0    11M  1840K select   0   0:28   0.02% sendmail
 2843 root          1  20    0  4684K   756K select   0   2:14   0.02% powerd
 3044 root          1  20    0  5920K   848K select   0   0:08   0.00% make

 

> Any chance of tmpfs use or other significant competition
> for RAM+SWAP?

I don't see how; /etc/fstab contains:
# Custom /etc/fstab for FreeBSD embedded images
#/dev/ufs/rootfs   /       ufs     rw      1       1
/dev/da0s2a     /       ufs     rw      1       1
#/dev/da0s2d    /usr    ufs     rw      1       1
#/dev/msdosfs/MSDOSBOOT /boot/msdos msdosfs rw,noatime 0 0
/dev/da0s1      /boot/msdos msdosfs rw,noatime 0 0
#tmpfs /tmp tmpfs rw,mode=1777,size=50m 0 0
/dev/da0s2b     none    swap    sw
#/dev/sdda0s2b  none    swap    sw
#/dev/mmcsd0s2b none    swap    sw

and mount reports:
root@www:/usr/src # mount
/dev/da0s2a on / (ufs, local, soft-updates)
devfs on /dev (devfs)
/dev/da0s1 on /boot/msdos (msdosfs, local, noatime

> 
> Also, figuring out what is using the SWAP space could be
> important. (It might not the same.)
> 
> Also, top can display more of the command lines. With such
> one can more about what the c++ command involved is.

Is this the output you're referring to?
last pid: 38057;  load averages:    1.08,    1.02,    1.01                                                             up 3+08:07:33  06:50:03
67 processes:  3 running, 62 sleeping, 2 waiting
CPU: 24.8% user,  0.0% nice,  0.1% system,  0.0% interrupt, 75.1% idle
Mem: 5744K Active, 307M Inact, 150M Wired, 73M Buf, 459M Free
Swap: 1770M Total, 422M Used, 1348M Free, 23% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
21654 root          1 135    0   561M    17M CPU2     2  49.8H 100.07% c++ -target armv7-gnueabihf-freebsd15.0 --sysroot=/usr/obj/usr/src/arm.

> > Buildworld output ends with:
> > ...
> > Building /usr/obj/usr/src/arm.armv7/lib/clang/libclang/Tooling/JSONCompilationDatabase.pico
> > Building /usr/obj/usr/src/arm.armv7/lib/clang/libclang/Tooling/Refactoring.pico
> > Building /usr/obj/usr/src/arm.armv7/lib/clang/libclang/Tooling/RefactoringCallbacks.pico
> > Building /usr/obj/usr/src/arm.armv7/lib/clang/libclang/Tooling/Tooling.pico
> > root@www:/usr/src #
> 
> Note all those ".pico" references. The build environment has had
> a significant change. Its consequences for small RAM armv7 or
> aarch64 boards need not be obvious:
> 
> QUOTE
> The branch main has been updated by dim:
> 
> URL: https://cgit.FreeBSD.org/src/commit/?id=2e47f35be5dc61945afdbd1a70e8fd505c032c94
> 

author	Dimitry Andric <[email protected]>	2025-04-25 17:56:39 +0000
The system has been updated several times since then without this occurring. 
It has, for a long time, exhibited a tendency to silently hang (no debug response)
in the past, usually long after swap use has peaked. I wonder if this loopish
behavior might be related. Rebooting always cleared the problem.
> 
> Avoiding multithreaded linking on 32 bit systems is normal because
> of bigger problems fitting in the 32 bit address space otherwise.

After running overnight, with swap still at 422 MB, it looks like a loop to me.

The obvious thing is to kill the PID and see what happens, but that will lose
this seemingly special opportunity...
I'll let it run in hopes more informative experiments come to mind. 

Thanks very much for writing! 

bob prohaska