Concurrent native callbacks performance

Dear mailing list,

recently I have asked about ways how to call an embedded CHICKEN library from
multiple native threads. The problem is that since objects are allocated in the
stack of the thread that initializes the runtime, using other threads to call
functions (eg, defined with `define-external`) causes the system to hang or
crash. Felix pointed me to CRUNCH (which I tried and is very promising and 
easy to get started) and to the `concurrent-native-callbacks` (cncb) egg.

I am not ready just yet to go for CRUNCH, so I decided to try cncb again. In my
micro benchmark, however, cncb seems quite slow. So I'd like to share with you
the results and a potential improvement and I'd be happy with feedback on
whether I am doing something silly or not.

The goal of the benchmark is to measure the C-to-Scheme call overhead. It
consists of a native thread calling one function `(foo)` 10M times. The time is
measured in the C program, around the loop calling the function. The time per
call is reported. Here is the loop code:

```
void bench_run(void) {
  size_t count = 1000000;
  nanosec_t ts = now();
  while (count--) {
    (void)foo(count);
  }
  nanosec_t then = now();
  printf("tput: %.2fop/s\tlatency: %0.0fns/op\n",
	  SAMPLES/in_sec(then-ts), 1.0*(then-ts)/SAMPLES);
  exit(0);
}
```

The baseline (`plain`) has only one native thread. It initializes the CHICKEN
runtime and then enters `bench_run`. Here is the implementation of `foo`.

```
(define-external (foo (integer x)) int
 (+ x 100))
```

Of course, the baseline cannot serve multiple threads, but it serves as ideal
performance.

The second variant is `cncb`. One native thread enters `bench_run`, a second
native thread is captured in the `(dispatch)` to serve the requests. `foo`
is simply:

```
(define-concurrent-native-callback (foo (integer x))
 (+ x 100))
```

Here are the results. All C code compiled with -O3 and Scheme with -O3 -d0,
run on macOS, M1 processor. 

```
       plain    tput: 10741173.17op/s   latency: 93ns/op
        cncb    tput: 278135.99op/s     latency: 3595ns/op
    ucontext    tput: 673281.25op/s     latency: 1485ns/op
    assembly    tput: 8065283.63op/s    latency: 124ns/op
```

The results include the variants I'll explain below, but the first observation
is cncb introduces a significant overhead. Am I measuring something incorrectly
or is this expected? I know that cncb uses a UNIX pipe. The caller transforms
its calls in requests, pushes them into the pipe. Another native thread (the
one stuck in dispatch) serves the requests from the caller.

I haven't experimented in other OSes, maybe macOS isn't very clever when
scheduling the thread serving the request. Anybody has experience here?

Now to the other variants. `ucontext` and `assembly` are pretty much the same
code except that in the `assembly` I reimplemented a custom `ucontext` in
AArch64 assembly. How it works is as follows: When the C program starts it
allocates a memory area to serve as stack when calling CHICKEN. Let's call this
area "S". Then using ucontext, the main thread temporarily replaces its stack
with S and calls `CHICKEN_run(C_toplevel)`. Then it swaps the stacks back.
After that, whenever a native thread wants to call the CHICKEN runtime, it
takes a lock (because only one thread can call CHICKEN at a time), swaps its
stack with S, calls the external function, swaps the stacks back. So in the
benchmark above, this swaping will happen 10M times.

The call to CHICKEN is not super straight-forward, I had to "massage" the
arguments and return value to make it work (probably in a similar fashion as
cncb does). One difference to cncb is that there is no native thread serving
as dispatcher, but rather a reserved stack S which is used by any thread that
needs to call CHICKEN.

Unfortunately, `ucontext` seems to go to the kernel and I guess that is causing
some overhead, so I wrote a simpler (20 lines) assembly version that simply
saves the registers and replace the stack pointer. That is what the `assembly`
version does.

Is the approach above a reasonable alternative to cncb, or am I missing some
major point?

Happy with any feedback.

Regards,
-Diogo