Re: Coroutines, goroutines

Hi Jan,

I was saving some of this for when I was ready to present my
implementation, but I leaked
it early, so here are some answers to your thoughtful questions:

The programming model is Java Project Loom's "virtual threads" adapted to
Common Lisp.
The core idea: fibers are lightweight cooperative threads with their own
stacks that
multiplex onto carrier OS threads. From the programmer's perspective, a
fiber looks and
acts like a thread -- you write ordinary sequential code, call ordinary
blocking I/O,
and the runtime handles the rest. No function coloring, no async/await, no
special
monadic style. This is the key property that motivated the design.

Coroutines vs. continuations: These are stackful coroutines, not first-class
continuations. Each fiber gets its own mmap'd control stack (default 256KB)
and binding
stack (16KB). There are no heap-allocated continuation objects. The stack
frames
themselves are the continuity mechanism -- on yield, the stack is preserved
in place; on
resume, execution continues from the exact point of suspension. This was a
deliberate
choice: first-class delimited continuations are a different (and
substantially harder)
feature. Fibers solve the concurrent-I/O-without-coloring problem directly.
Someone who
wants delimited continuations for backtracking solvers would need a
different mechanism,
and I don't think fibers should try to be that.

Lexical variables: No special handling needed. Lexical variables live on
the fiber's
control stack and are naturally preserved across yields. Closures captured
within a
fiber work exactly as expected.

Dynamic variables: Each fiber maintains its own dynamic bindings. On yield,
the fiber's
current TLS values are saved to an overlay array and the carrier thread's
original
values are restored. On resume, the fiber's values are written back. The
binding stack
entries are never modified by the scheduler -- they remain intact for
normal Lisp
unbinding semantics. So (let ((*x* 1)) (fiber-yield) *x*) sees 1 after
resuming, and the
carrier thread's binding of *x* is undisturbed while the fiber is
suspended. This is
analogous to how threads don't inherit dynamic bindings from their parent
-- fibers are
independent.

UNWIND-PROTECT: Works correctly. The fiber's catch block and unwind-protect
block chain
pointers are saved on yield and restored on resume. Cleanup forms do NOT
run when a
fiber suspends -- they run when the fiber eventually exits the protected
form, either
normally or via error. This matches the expected semantics: suspension is
not unwinding.

Pinning (your DYNAMIC-WIND question): A fiber can be "pinned" via
(with-fiber-pinned () ...), which prevents yielding inside the dynamic
extent of the body.
If a pinned fiber encounters a blocking primitive (mutex, condition-wait,
I/O), it falls through
to the OS blocking path instead of yielding. This is for holding resources
that can't survive a
yield (ffi). The *pinned-blocking-action* variable controls whether this
situation warns, errors,
or silently falls through.

Blocking I/O, locks, SLEEP: SBCL's standard blocking primitives (grab-mutex,
condition-wait, wait-until-fd-usable) are fiber-aware. When called from
within a fiber,
they transparently yield with a wake-condition predicate instead of
blocking the carrier
thread. The scheduler polls these predicates and resumes fibers when their
conditions
are met. For I/O specifically, the scheduler uses epoll (Linux) or kqueue
(BSD) for
efficient multiplexing. fiber-sleep yields with a time-based deadline
predicate.
Existing code using standard CL I/O and SBCL threading primitives becomes
fiber-aware
automatically when called within a fiber context -- no code changes
required.

Multi-carrier scheduling: Fibers aren't limited to a single OS thread. The
run-fibers
API accepts a :carrier-count parameter that spawns multiple carrier threads
with a
work-stealing scheduler (Chase-Lev deques). Each carrier owns a local deque
-- push/pop
from the bottom for locality, thieves steal from the top for fairness. This
means fibers
get both the lightweight concurrency model and actual parallelism across
cores, which
is again directly analogous to how Loom's virtual threads are scheduled
onto a
ForkJoinPool. A fiber may migrate between carriers across yields, which the
runtime
handles transparently (updating thread pointers, TLS overlays, GC roots,
etc.).

I welcome comments and questions, and I'm open to advice or suggestions.

Thanks!
AG

On Thu, Feb 26, 2026 at 7:12 AM Jan Moringen <[email protected]> wrote:

> I don't think it is my place anymore to comment on whether or with
> which tools such a feature should be implemented in SBCL but I would
> like to add one consideration regarding the process: Except for the
> first message, the discussion has focused on tools used for the
> implementation and details like stack size or abstraction for machine
> code generation.
>
> The original message (and the quoted message) was about requirements of
> potential users. Those requirements mentioned a programming model for
> asynchronous I/O without "coloring" functions and also delimited
> continuations with backtracking in solvers as a possible application.
>
> I feel the discussion should start with the envisioned programming
> model, (concurrent) evaluation semantics and user interface of the
> continuation/fiber/coroutine feature rather than the details of one
> particular implementation that happens to be available and close to
> finished. In particular as the original messages were talking about CL
> in general not SBCL. I have the impression that a lot of decisions have
> to be made (implicitly or, better, explicitly):
>  * coroutines vs. continuations
>  * Closing over lexical variables (only applies to the continuation
>    model, i guess)
>  * Interaction with dynamic variables (are dynamic binding stacks
>    concatenated when invoking a continuation in a context that has its
>    own bindings?)
>  * Interaction with UNWIND-PROTECT when coroutines/continuations are
>    suspended
>  * DYNAMIC-WIND whether pinning
>  * Interaction with blocking I/O, locks, waiting for processes and
>    threads, SLEEP, etc.
>
> If this discussion has already happened in a space I'm not aware of, I
> apologize for the noise.
>
> Kind regards,
> Jan
>

_______________________________________________
Sbcl-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/sbcl-devel