Re: Very long-running buildworld process

On Jun 12, 2025, at 09:50, bob prohaska <[email protected]> wrote:

> On Wed, Jun 11, 2025 at 09:31:54PM -0700, Mark Millard wrote:
>> On Jun 11, 2025, at 07:08, bob prohaska <[email protected]> wrote:
>> 
>>> On Tue, Jun 10, 2025 at 10:24:26PM -0700, Mark Millard wrote:
>>>> On Jun 10, 2025, at 20:59, bob prohaska <[email protected]> wrote:
>>>> 
>>>>> Buildworld for -current on a Pi2B is strangely slow. The machine is responsive,
>>>>> but seems to make no (or very slow) progress:
>>>>> 
>>>>> 
>>>>> login: Jun  7 22:51:31 www su[3032]: bob to root on /dev/pts/0
>>>>> 
>>>>> 
>>>>> FreeBSD/arm (www.zefox.org) (ttyu0)
>>>>> 
>>>>> login: bob
>>>>> Password:
>>>>> Last login: Sat Jun  7 22:51:25 from gateway.zefox.net
>>>>> FreeBSD 15.0-CURRENT #85 main-d8773fdcbfa3: Fri Jun  6 14:17:58 PDT 2025     [email protected]:/usr/obj/usr/src/arm.armv7/sys/GENERIC
>>>>> 
>>>>> Welcome to FreeBSD!
>>>>> 
>>>>> Release Notes, Errata: https://www.FreeBSD.org/releases/
>>>>> Security Advisories:   https://www.FreeBSD.org/security/
>>>>> last pid: 34625;  load averages:    1.01,    1.01,    1.00                                                             up 2+22:03:17  20:45:47
>>>>> last pid: 34628;  load averages:    1.07,    1.02,    1.01                                                             up 2+22:05:21  20:47:51
>>>>> 45 processes:  2 running, 43 sleeping
>>>>> CPU: 25.0% user,  0.0% nice,  0.0% system,  0.0% interrupt, 75.0% idle
>>>>> Mem: 6196K Active, 297M Inact, 174M Wired, 98M Buf, 443M Free
>>>>> Swap: 1770M Total, 422M Used, 1348M Free, 23% Inuse
>>>> 
>>>> Free  443M
>>>> Inact 297M
>>>> Wired 174M
>>>> 
>>>> Yet:
>>>> 
>>>> c++ RES is just 17M and Active is just 6196K.
>>>> 
>>>> Also: 422M SWAP used.
>>>> 
>>>> I suggest sorting the top output by (decreasing) RES and
>>>> seeing if that shows what is contributing to Inact 297M
>>>> as part of RESident memory use.
>>>> 
>>> 
>>> Changing order to res yields in part:
>>> last pid: 37994;  load averages:    1.01,    1.01,    1.00                                                             up 3+07:55:55  06:38:25
>>> 45 processes:  2 running, 43 sleeping
>>> CPU: 24.6% user,  0.0% nice,  0.3% system,  0.0% interrupt, 75.2% idle
>>> Mem: 5672K Active, 307M Inact, 150M Wired, 73M Buf, 459M Free
>>> Swap: 1770M Total, 422M Used, 1348M Free, 23% Inuse
>>> 
>>> PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
>>> 21654 root          1 135    0   561M    17M CPU2     2  49.6H 100.12% c++
>>> 37994 bob           1  20    0  6664K  3088K CPU3     3   0:00   0.19% top
>>> 2951 root          1  20    0    11M  1840K select   0   0:28   0.02% sendmail
>>> 2843 root          1  20    0  4684K   756K select   0   2:14   0.02% powerd
>>> 3044 root          1  20    0  5920K   848K select   0   0:08   0.00% make
>> 
>> Hmm. May be:
>> 
>> # ps -axldww
>> 
>> might show something that would prove interesting?
> I've placed the output at
> http://www.zefox.net/~fbsd/rpi2/20250612/ps-axldww.log
> It's too wide to view on the list. It shows the complete
> (very long) command line for the offending PID.
> 
>> 
>> As for watching the specific c++ process, may be
>> you might temporarily run (output goes to stderr):
>> 
>> # truss -fae -p 21654
>> 
>> then ^C it. From this you would learn if the kernel is
>> in use via system calls and what kinds of system calls.
>> 
>> (I picked truss for simpliicty vs. ktrace and kdump use.)
>> 
> 
> Near as I can tell, truss produces zero output after running
> for a couple of minutes.

That indicates internal looping with no IO or other
kernel-based activity.

In the ps output, it is the only process showing more
than 15000 (574536) for VSZ. Thus it seems likely to be
the major source for the Inact and Used SWAP showing in
your top runs.

I wonder if your c++ has enough symbol information (or
possibly debug information) to make attachment to the
process with gdb or lldb and a backtrace useful (routine
names, for example, not just addresses)? (An attached
debuger can also quit/exit the program being debugged.)

Other than that sort of information gathering, it looks
like the process is stuck looping strictly internal to
itself, not asking for kernel services. Ultimately,
killing or quitting it some way.

If you could figure the right directory to execute the
long c++ command from, you might be able to retry the
command from the same file context to see if it again
gets stuck. If it does, then there might be some hope
of tracking down what is going on if the context is
preserved. (That is not the same as saying doing so
would be reasonable to try.)

===
Mark Millard
marklmi at yahoo.com