Re: Threads.... the long story, and a proposal

On 9/4/26 11:29 AM, Leon Timmermans wrote:
> On Fri, Sep 4, 2026 at 5:13 PM Laurent <[email protected]> wrote:
>
>     In Python, the
>     numeric stack (NumPy, and friends) *releases the GIL* around its C
>     kernels: Python
>     stays single-threaded and safe, while the heavy C runs on OS threads
>     and uses all
>     the cores. The Perl analogue is: keep one interpreter, let green
>     threads schedule
>     the Perl side, and let blocking or CPU-bound XS run on OS threads.
>     Threading *the
>     XS part* — not the Perl part — is what most people are really after.
>
>     ### PDL: the pattern already shipped — and where it stops
>
>     PDL is the closest thing Perl has to NumPy, and it is worth looking at
>     because it
>     demonstrates both halves of this split *and* the seam between them.
>
>     It demonstrates the orchestration half so thoroughly that it is
>     packaged that way:
>     alongside its plain 32- and 64-bit editions, Strawberry Perl publishes
>     a dedicated
>     **PDL edition** — an entire Perl distribution assembled around "Perl
>     orchestrates,
>     XS executes", with the numeric stack and its supporting libraries
>     prebuilt. That
>     is about as direct an endorsement of the model as an ecosystem can
>     give.
>
>     It also already does the *parallel* half, without any help from
>     perl. Built with
>     POSIX threads, `set_autopthread_targ` / `set_autopthread_size`
>     split an
>     operation's implicit-loop dimensions across a pool of pthreads
>     that execute only
>     generated C and never touch the interpreter. (Confusingly, PDL
>     historically called
>     that implicit looping "threading"; it is now called
>     *broadcasting*, precisely
>     because it is a loop-shape feature and not a CPU-count one. The
>     pthread support is
>     the separate, genuinely parallel thing.)
>
>     What PDL does **not** do is release the interpreter.
>
>
> Sorry but I don't think people are going to read a 16k word proposal 
> from a stranger.
>
> Maybe you have some good ideas here, but we're not going to find out 
> if it's presented like this.
>
> Leon

TL;DR - Perl lacks the ability to treat a long-running XS function call 
as an asynchronous event waiting for completion.  If perl had this 
ability, the event loop could get back to other perl logic while e.g. 
PDL does a massive distributed computation. While a general solution can 
be hacked together in Coro and/or Future::AsyncAwait with enough effort 
and tinkering with private internals, it sure would be nice if the 
official core API exposed the pieces that Coro and FAA depend on so that 
it's less of a hack.

Followed by a large number of technical details.

Laurent: correct me if I missed the gist.

-Mike C