RFC: ENOSYS vs SIGSYS for FreeBSD ABIs

FreeBSD has a longstanding behavior of sending a SIGSYS signal to a thread
that invokes an invalid system call.  This behavior was inherited from BSD
from a commit from Mike Karels whose log message is "POSIX changes"
(https://svnweb.freebsd.org/csrg/sys/kern/kern_sig.c?revision=39513&view=markup).

 From what I can tell though, while POSIX documents both ENOSYS and SIGSYS,
it does not document the semantics of these (and the one reference I could
find to SIGSYS specifically noted that the semantics aren't documented since
there aren't any consistent semantics across systems).

Linux does not send SIGSYS for an invalid system call, but instead returns
ENOSYS which motivated several changes from kib@ a few years ago which,
amony other things, makes FreeBSD's behavior as there is now a sysctl to
determine if the FreeBSD ABIs raise SIGSYS or just return ENOSYS.

In my experience to date, I have generally found SIGSYS a pain.  While we
don't generally support forwards compatibility (newer world on older
kernel), it sometimes happens, and it's really annoying to have init, etc.
all die on boot (resulting in an instant panic in the case of init).
This is especially true if one is adding a new "advisory" system call for
which the program can still run fine if the syscall fails.  However, there
is not a really easy way to deal with this, particularly in library code.
Installing a handler for SIGSYS (or just ignoring it), means a library
has to stomp on process state that typically should be managed by the
main binary.  The other alternative is to try to do extra checks to determine
if a syscall might work before you invoke it, whether that's
kern.feature.foo sysctls (but that does mean invoking yet-another-syscall
before the one you actually care about), or AT_BSDFLAGS, etc.

On the other side, I have _never_ had SIGSYS be an aid.  It's only gotten
in the way.

Over in CheriBSD land we just got bitten by this (again) for a system call
that is purely advisory (permits adding a label to memory regions that you
can see in procstat -v, etc., but isn't needed for functionality) due to
some of the cross-threading we have between kernels and worlds which is why
this is top of my mind again.

Both OpenBSD and NetBSD still send SIGSYS.  To deal with SIGSYS in /sbin/init,
all of the BSDs imported a change from BSDI after 4.4BSD that ignored the
first 25 errors, but both FreeBSD and OpenBSD have removed that (NetBSD still
has it).  The log message for our removal claims that relying on ENOSYS vs
SIGSYS is harmful, but I'm less convinced that that is true.  You have to
have code invoking a system call that doesn't check for errors for SIGSYS
to actually be useful.

My (long-winded) proposal is to change the default setting of the sysctl kib@
added a few years ago such that we only return ENOSYS by default.  Folks who
really want SIGSYS could still opt-in via the sysctl/loader tunable, but I
feel that ENOSYS really is the better default behavior instead of SIGSYS.

-- 
John Baldwin