Suggestion: a function or helper to acquire a controlling terminal

Hello,
I see this is the preferred venue for suggestions and feature requests. Right now, writing a program that acquires a particular terminal as a controlling terminal is not easy and I think Gnulib is the right place for improvement. POSIX says that the means by which a process acquires a controlling terminal is implementation-defined. The standard permits this to happen for an eligible session as soon as a TTY is open()ed, if the O_NOCTTY flag isn't given. Here's a short round-up of system behaviors I know of:
 • On GNU/Linux, this is done implicitly in the call to open(), but an ioctl(), TIOCSCTTY, can be used in an application to do this separately if desired.
 • On OpenBSD, it is an intentional design choice that this isn't done implicitly, but the TIOCSCTTY ioctl() is necessary.
 • NetBSD used to behave like GNU/Linux, but now works like OpenBSD.
	https://man.netbsd.org/tty.4
 • On macOS, it seems they've gone the opposite direction and allowed for the Linux-like implicit behavior when they didn't before. However the manual pages technically conflict with each other.
 ◦ https://man.freebsd.org/cgi/man.cgi?query=open&sektion=2&manpath=macOS#COMPATIBILITY (updated in 2021)
	> open()  on a terminal device (i.e., /dev/console) will now make that device a controlling terminal for the process. Use the O_NOCTTY flag to open a terminal device without changing your controlling terminal.
 ◦ https://man.freebsd.org/cgi/man.cgi?query=tty&sektion=4&apropos=0&manpath=macOS (last changed in 1992 BSD)
	> The current system does not allocate a controlling terminal to a process on an open() call: there is a specific ioctl called TIOSCTTY [sic] to make a terminal the controlling terminal.
 • MINIX 3 has the implicit behavior
	https://man.freebsd.org/cgi/man.cgi?query=tty&sektion=4&manpath=Minix
 • BlackBerry's QNX supports the ioctl(), but also has a <termios.h>-style tcsetsid() function. Despite that OS being proprietary, this deserves an honorable mention for being a clever interface: it takes a pid_t argument to identify a process (which need not be a session leader), and associates the terminal with the session that process is in. For the common case of wanting to associate the terminal with the session of the current process, getsid(0) or getpid() could be passed.
	https://www.qnx.com/developers/docs/8.0/com.qnx.doc.neutrino.lib_ref/topic/t/tcsetsid.html
 • SCO OpenServer 6 has the implicit open behavior at the very least http://osr600doc.xinuos.com/en/SDK_sysprog/_The_Controlling-Terminal_and_Pr.html
 • UnixWare 7 also has implicit behavior http://uw714doc.xinuos.com/en/SDK_sysprog/_The_Controlling-Terminal_and_Pr.html
 • illumos is implicit https://www.illumos.org/man/4I/termio 

My need for this is to write programs that interface with serial devices, such as the NeuG libre random number generator, GNSS receivers, and embedded GNU/Linux boards. Even though these aren't human interface devices, some of the semantics of a controlling terminal are still useful. In particular I'd like SIGHUP semantics for when a sensor is disconnected or when the device on the other end of a USB serial adapter is dead. Also, a terminal can only be the controlling terminal for one session at a time, so this has the perk of being a mutual exclusion mechanism too.

Here are some of the interfaces that are most close to meeting my needs, but not quite:
 ‣ POSIX.1-2024 has totally removed the specification of ioctl(), replacing it with posix_devctl(), and even the header it was formerly declared in is no longer required to exist. The Linux Standard Base does specify ioctl(), and although it specifies many arguments related to terminals, it does not include TIOCSCTTY. Thus even an LSB-conformant system doesn't need to have TIOCSCTTY.
	⁃ Gnulib has an ioctl module, but it's not clear when or how it can be relied on to define TIOSCTTY. It appears to try pulling in constants with #include_next. I suppose an application could see if a plain open() suffices, and if it doesn't, do #ifdef TIOSCTTY to see if Gnulib's ioctl is worth trying.
 ‣ The login_tty() function is in the LSB and present on most systems mentioned above, but it has a couple inappropriate quirks. It calls setsid() on behalf of the caller, but if doing the implicit open() then the caller needs to do that itself first, and it also clobbers the standard file descriptors needlessly. If my goal is to interface with a hardware device, my application may still want to kick out warning or error messages to stderr which should go to the service manager or be displayed to the user, and not sent to the GNSS chip.

It would be nice if Gnulib had some ctrlterm(int fd) function or the like, that would save me some boilerplate code by letting me do the following:
{
	if(setsid() == -1) abort();
	int fd = open("/dev/ttyXXX", O_RDWR);
	if(fd == -1) abort();
	if(!ctrlterm(fd)) {
		perror("Failed to obtain controlling terminal");
	}
}

In particular, this new function could return a success condition and be a no-op if the terminal is *already* the controlling terminal for the current session (perhaps because it became so during the open() call), and otherwise do whatever needs to be done behind-the-scenes like the ioctl() to try to make that happen. If one is on a system without the ioctl() or another appropriate method, this'll not be able to try anything, and the caller should get ENOSYS or the like. Otherwise, it should return success or failure to indicate if obtaining the controlling terminal succeeded.

So in conclusion, this is hard to get right in an application, but all systems I know of require an ioctl() if the controlling terminal isn't set on open(). Even trying that is increasingly more challenging in an application due to namespace issues. It's also not semantic: if some third mechanism for obtaining a controlling terminal is needed someday, it'd be nice to centralize that code in Gnulib so an application can pick it up later.

Thanks for your consideration.

-----BEGIN PGP SIGNATURE-----

iPsEABYKAKMWIQSiPzylvTnZ6xisfzWz9N0oYfTNugUCaoXuWnIYaHR0cHM6Ly9q
b2huc2NvdHQubWUvLndlbGwta25vd24vbmkvc2hhLTI1Ni9zWUF3OTN6QUVrRkIy
RDREM1hOemRSeHEyMFBjNnByZGdtbEVWeXo0QUZRP2N0PWFwcGxpY2F0aW9uJTJG
cGdwLWtleXMSHGpzY290dEBwb3N0ZW8ubmV0AAoJELP03Shh9M26pa0BAOAzXkp/
464BBDryWGdeK1ha9HM++QJYdv92hFCp8nFFAQDbrA0wRjH6MA37kzzzBxun67SE
WJFijFZ6vbaCafvSBQ==
=/3Ml
-----END PGP SIGNATURE-----