Re: Very long-running buildworld process

On Jun 12, 2025, at 11:51, bob prohaska <[email protected]> wrote:

> On Thu, Jun 12, 2025 at 10:33:24AM -0700, Mark Millard wrote:
>> On Jun 12, 2025, at 09:50, bob prohaska <[email protected]> wrote:
>> 
>>> On Wed, Jun 11, 2025 at 09:31:54PM -0700, Mark Millard wrote:
>>>> Hmm. May be:
>>>> 
>>>> # ps -axldww
>>>> 
>>>> might show something that would prove interesting?
>>> I've placed the output at
>>> http://www.zefox.net/~fbsd/rpi2/20250612/ps-axldww.log
>>> It's too wide to view on the list. It shows the complete
>>> (very long) command line for the offending PID.
>>> 
>>>> 
>>>> As for watching the specific c++ process, may be
>>>> you might temporarily run (output goes to stderr):
>>>> 
>>>> # truss -fae -p 21654
>>>> 
>>>> then ^C it. From this you would learn if the kernel is
>>>> in use via system calls and what kinds of system calls.
>>>> 
>>>> (I picked truss for simpliicty vs. ktrace and kdump use.)
>>>> 
>>> 
>>> Near as I can tell, truss produces zero output after running
>>> for a couple of minutes.
>> 
>> That indicates internal looping with no IO or other
>> kernel-based activity.
>> 
>> In the ps output, it is the only process showing more
>> than 15000 (574536) for VSZ. Thus it seems likely to be
>> the major source for the Inact and Used SWAP showing in
>> your top runs.
>> 
>> I wonder if your c++ has enough symbol information (or
>> possibly debug information) to make attachment to the
>> process with gdb or lldb and a backtrace useful (routine
>> names, for example, not just addresses)? (An attached
>> debuger can also quit/exit the program being debugged.)
>> 
> The machine is a vanilla -current install, with whatever
> comes "out of the box". There's a gdb man page and binary,
> but neither for lldb.

gdb these days is normally via installing a pre-built
package or building/installing from the ports tree
( devel/gdb ):

# which gdb
/usr/local/bin/gdb
# find -s / -name gdb.1.gz -print
/usr/local/share/man/man1/gdb.1.gz

That shows a package/ports style of install paths. What
does your context show?

# find -s / -name gdb -print
/usr/local/bin/gdb
/usr/local/share/gdb
/usr/local/share/gdb/python/gdb
/usr/share/bhyve/gdb
/usr/src/contrib/netbsd-tests/usr.bin/gdb
/usr/src/sys/gdb
/usr/src/usr.sbin/bhyve/gdb

# pkg info gdb
gdb-15.1
Name           : gdb
Version        : 15.1
Installed on   : Wed Dec  4 16:15:50 2024 UTC
Origin         : devel/gdb
Architecture   : FreeBSD:14:armv7
Prefix         : /usr/local
Categories     : devel
Licenses       : GPLv3
Maintainer     : [email protected]
WWW            : https://www.gnu.org/software/gdb/
Comment        : GNU Project Debugger
Options        :
	BUNDLED_READLINE: off
	BUNDLED_ZLIB   : off
	DEBUGINFOD     : off
	GDB_LINK       : on
	GUILE          : off
	KGDB           : on
	NLS            : on
	PORT_ICONV     : on
	PORT_READLINE  : on
	PYTHON         : on
	SOURCE_HIGHLIGHT: on
	SYSTEM_ICONV   : off
	SYSTEM_ZLIB    : on
	TUI            : on
	XXHASH         : on
Shared Libs required:
	libzstd.so.1
	libxxhash.so.0
	libsource-highlight.so.4
	libreadline.so.8
	libpython3.11.so.1.0
	libmpfr.so.6
	libintl.so.8
	libiconv.so.2
	libgmp.so.10
	libexpat.so.1
	libboost_regex.so.1.85.0
Annotations    :
	FreeBSD_version: 1401000
	build_timestamp: 2024-11-24T15:09:41+0000
	built_by       : poudriere-git-3.4.2
	cpe            : cpe:2.3:a:gnu:gdb:15.1:::::freebsd14:armv7
	flavor         : py311
	port_checkout_unclean: no
	port_git_hash  : 82beca9e630
	ports_top_checkout_unclean: no
	ports_top_git_hash: 203aa1081a2
	repo_type      : binary
	repository     : FreeBSD
Flat size      : 52.4MiB
Description    :
GDB is a source-level debugger for Ada, C, C++, Objective-C, Pascal and
many other languages.  GDB can target (i.e., debug programs running on)
more than a dozen different processor architectures, and GDB itself can
run on most popular GNU/Linux, Unix and Microsoft Windows variants.


As for lldb:

# which lldb
/usr/bin/lldb
# find -s / -name lldb.1.gz -print
/usr/share/man/man1/lldb.1.gz

That shows a FreeBSD style of system paths, not a
package/port installation path.


(Note: I have devel/llvm19 also installed.)

# find -s / -name lldb -print
/usr/bin/lldb
/usr/local/llvm19/bin/lldb
/usr/local/llvm19/include/lldb
/usr/local/llvm19/lib/python3.11/site-packages/lldb
/usr/src/contrib/llvm-project/lldb
/usr/src/contrib/llvm-project/lldb/include/lldb
/usr/src/lib/clang/include/lldb
/usr/src/usr.bin/clang/lldb

So I'm confused by what you report.

This area probably needs to be figured out
first and possibly have devel/gdb installed.

Are you more familiar with lldb? gdb? I've
used gdb more and not frequently.

>> Other than that sort of information gathering, it looks
>> like the process is stuck looping strictly internal to
>> itself, not asking for kernel services. Ultimately,
>> killing or quitting it some way.
>> 
> It's possible to get into (a) debugger using ~^B, so:
> bob@www:~ % KDB: enter: Break to debugger
> [ thread pid 10 tid 100002 ]
> Stopped at      kdb_alt_break_internal+0x1b8:   ldrb    r15, [r15, r15, ror r15]!
> db> 
> but what to do next is unclear. I know how to type bt and
> copy the console output, but that's all at this stage.

That is the built-in kernel debugger, not a
user-process debugger. The process is not
making system calls.

> 
>> If you could figure the right directory to execute the
>> long c++ command from, you might be able to retry the
>> command from the same file context to see if it again
>> gets stuck. If it does, then there might be some hope
>> of tracking down what is going on if the context is
>> preserved. (That is not the same as saying doing so
>> would be reasonable to try.)
>> 
> 
> ISTR it's possible to place a job under the "supervision"
> of a debugger and then poke around in the process at will.
> The details are unclear, might that be instructive in this 
> case? I can always pull the plug and then restart buildworld.
> Or, maybe just kill the originating job, though I'm not sure 
> that will kill all the child processes.

For now I suggest figuring out the lldb and gdb context
and then attaching one of those to the existing process.
After that, figuring out how to look around.

attaching will likely report on what symbols or debug
information is found for the process attached to --vs.
if no such is found.

> If history is any guide starting the buildworld over will
> lead to the same loop, though of course one can't be sure
> without trying 8-)




===
Mark Millard
marklmi at yahoo.com