Re: Leveraging process descriptors for process creation without fork

On Mon, Jan 10, 2022, at 3:55 AM, John Ericson wrote:
> Hi, I have long been interested in seeing an alternative to fork+exec
> implemented in a real-world Unix OS. What I have in mind is:
> 
> 1. Load an executable into a fresh new unscheduled process (we might
>    call this an "embryonic" process)
> 2. Set file descriptors and other state on that process
> 3. Submit it to the scheduler
> 
> I am far from the first person to think of this interface, of course.
> Since becoming interested in this, I have been referred to this
> paper[1], which very nicely describes the concept in detail, and also
> evaluates an all-userspace implementation of it.
> 
> I must admit I first emailed the Linux mailing list about this.[2] The
> reception was positive, but the multitude of other features[3]
> supported in the relevant code makes refactoring it to elegantly
> implement both the existing and new interfaces a rather large task.
> 
> I checked, and FreeBSD's fork and exec code is, surprise surprise, a
> great deal simpler, and therefore a better venue for demonstrating
> this feature's viability. Also, as I am interested in the feature in
> the context of efforts like Capsicum and CloudABI, FreeBSD is a
> natural starting point for cultural and historical reasons.
> 
> I am under no illusion that, even with FreeBSD's comparative
> simplicity, I will have time to finish this project in the near
> future, but I hope it is still OK to discuss the merits of the idea
> itself.
> 
> Thanks,
> 
> John
> 
> [1]: http://catern.com/rsys21.pdf
> 
> [2]: https://lore.kernel.org/lkml/[email protected]/T/#m6be1994668e6f34837496c86f37f9fe52bfae990
> 
> [3]: Especially binfmt_misc, if anyone was curious

Hello, years later I am happy to say that I finally have a draft patch
series for this: https://reviews.freebsd.org/D58688. That is the main
patch in the series, which has a bunch of introductory text that instead
could go in this email in its own right. (Maybe the most interesting
part of the introductory text is the similar work going on in Linux land
right now, which I have also had a small hand in discussing.) Before it
are preparatory refactors, and after it is a testing commit.

For the actual change itself, see that patch and the man page changes it
contains. Insofar as I don't have a *specific* architectural question at
this time, I don't think it is useful for me just to restate those
things here.

What I am more concerned with is broad feedback and "unknown unknowns".
This is only my second patch series after
https://reviews.freebsd.org/D58405, which I am happy to say has been a
delightful experience writing and getting reviewed. (As I say in the
aside in each commit, I am completely fine if this series lies dormant
until that series is fully dealt with.)

Unfortunately, this series is a lot more involved than that one was. To
implement this sort of functionality, one has to finely "slice and dice"
`fork` and `exec` in order to allow putting the pieces back together
either in `fork`/`exec` form, or in "new process from scratch" (this
functionality) form. This means making many little functions that deal
with numerous subsystems (indeed, one can think of process spawning in
general sort of being a "grand tour of the subsystems"). So these are
hard refactors to review for anyone, let alone me who is barely familiar
with any of it.

I want to make sure I am not dumping a bunch of slop in someone else's
lap and saying "you go figure it out".

That said, I very much like the broad look of things. The details that
surfaced during the implementation process which made it into the man
page I am proud of. I also have long believed that this is a very
natural way of creating a process, different as it may be from the
status quo, so that if something like this does land, it should be
surprisingly maintainable.

The myriad little functions factored out of `fork` and `exec` might be
daunting, but they do ensure that we share as much code as possible,
which I think is crucially important for something as involved as
process spawning. Every new syscall starts less widely used and tested,
and if we had two completely separate process spawning code paths, it
would be very hard to not end up in a permanent state of whack-a-mole,
fixing bugs in one only to neglect them in the other. Maximal code reuse
limits that, doing its best to ensure that the correctness and
performance of one can "piggy-back" on the other.

Even though this will take a number of rounds of review and revision
before it can land, I hope other people would be excited about this too.
I think it will be a huge boon for "file descriptor-oriented
programming" (i.e. being in capability mode, or for portable software,
pretending/wishing one is in capability mode), because *the* central
task one does with capabilities is selectively share/withhold them from
other processes, and this makes that so much easier in the parent-child
case.

Also, it ought to be plain faster than `fork`, too, and less crazy to
reason about than `vfork`. (And it might also be faster than `vfork`,
too.)

The next step, I realize as I write this, is to reimplement
`posix_spawn` on top of this new way of process spawning. This will end
up providing far more test coverage in live systems than my test suite,
and in the <hour I devoted to this so far, some interesting things have
already come up (a bug where we inherited process state that we
shouldn't have). As I should soon go to sleep, I will not finish
implementing it right now; but as I already submitted these patches, I
will not delay sending this email either. Therefore, I suppose my
conclusion is "sit tight, actually". Once I make my revisions and submit
new patch(es) extending the series implementing `posix_spawn`, then I
think this stuff will be in a more extensively featureful and
battle-tested state.

Cheers,

John