Re: RFC on ${^RNG} callback API for overriding rand()

On Mon, 17 Aug 2026 at 15:34, demerphq <[email protected]> wrote:

> Please see
>
> https://github.com/Perl/perl5/pull/24722
>
> And consider it a POC/RFC for hooking rand() sanely.
>

I feel like I have failed to express my intent, and my perspective of what
I see as the problem that I am trying to solve with this. I had assumed
that between the commit message and the content of the change that it would
be clear. But I've seen enough comments which suggest to me that I
didn't explain myself very well. So before I engage with them in the github
discussion thread I wanted to post this here. The pod form is attached.

  NAME
    Making Perl's Random Number Generator Pluggable

BACKGROUND AND FACTS
    Perl exposes pseudorandom number generation through the traditional
    C-like "rand()" and "srand()" functions.

    Perl exposes randomness through:

        srand($seed);
        my $value = rand();

    The generator itself is implicit. Code does not normally hold an RNG
    object or identify which generator it wishes to use. Instead, calls to
    "rand()" operate on shared interpreter state.

    Consequently, code throughout a Perl interpreter which calls "rand()"
    consumes values from the same pseudorandom stream.

    Since Perl 5.20, "rand()" has used the "drand48" pseudorandom number
    generator. "drand48" is a linear congruential generator (LCG) with 48
    bits of state, and it is fast, widely understood, and of adequate
    statistical quality for many ordinary non-cryptographic/non-mathematical
    uses. There is no way to change the internal RNG.

    THe only real leverage a developer has over rand() and srand() is that
    it is possible to install alternatives in the global symbol table.

    This combination of a global fixed implementation and implicit shared
    state creates a number of problems.

PROBLEMS WITH THE CURRENT MODEL
  1. The implementation has become part of the compatibility surface
    One useful property of a pseudorandom generator is reproducibility.
    Given the same generator and initial state, it produces the same
    sequence.

    Perl programmers have naturally made use of this in tests:

        srand(12345); # randomized test follows

    The problem is that such tests are dependent not merely on the seed, but
    on the exact sequence generated by Perl's current implementation.

    Changing the underlying generator in any way would cause the same seed
    to produce a different sequence. Tests -- and potentially other programs
    -- which have come to depend on that sequence could consequently change
    behaviour or fail.

    Thus drand48 has therefore become baked into Perl's de facto
    compatibility surface, we cant change it in nearly anyway without
    breaking a lot of code.

    Whatever theoretical benefits might result from replacing "drand48", the
    practical compatibility cost makes doing so unattractive. So if we want
    the benefits of a better RNG then we need to find an alternative.

  2. Unrelated consumers interfere with one another
    Because "rand()" operates on shared state, otherwise unrelated pieces of
    software affect each other's randomized behaviour.

    Consider:

        srand(12345);

        do_something_random();
        do_something_else_random();

    If some dependency begins making an additional call to "rand()", every
    subsequent consumer observes a different part of the sequence.

    A minor modification to one component can therefore change randomized
    behaviour elsewhere in the program.

    This is particularly awkward when reproducing failures. Adding debugging
    code, changing an implementation detail, or upgrading a dependency can
    perturb the random stream and cause the behaviour under investigation to
    change.

    The source of this coupling is not randomness itself. It is the single
    implicit random stream shared by otherwise independent components.

  3. Random streams are difficult to isolate
    There is no straightforward standard Perl mechanism by which one
    component can say:

        Continue using ordinary rand(), but use a random stream independent
        of the rest of the application.

    A component can instead use an explicit RNG object or another
    random-number API. But that requires the component to have been written
    specifically to do so.

    Isolation therefore has to be designed into each consumer rather than
    being something the surrounding application can control.

  4. "rand()" is poorly composable
    Suppose a library correctly and conventionally uses Perl's standard
    "rand()" function.

    An application using that library may later want to:

    *   use a higher-quality PRNG;

    *   use a particular deterministic generator during testing;

    *   give one subsystem its own random stream;

    *   reproduce a failure from a known PRNG state; or

    *   use a generator with properties appropriate to a particular
        application.

    There is no clean general mechanism by which the caller can make the
    library's existing calls to "rand()" use that generator.

    The usual solution is to modify the library so that it accepts an RNG
    object, callback, or some other source of random values.

    That is often good API design for an individual component, but it does
    not solve the problem at the language level.

    Indeed, when every component invents its own mechanism, application-wide
    determinism becomes harder:

        seed Perl's rand()
        seed subsystem A's RNG
        seed subsystem B's RNG
        seed library C's RNG
        ...

    One can easily imagine a sufficiently complicated test having to
    initialize several unrelated RNGs correctly just to reproduce one
    execution.

    That way lies madness.

    The application should be able to establish a randomness policy without
    having to know the implementation details of every component which
    happens to need random numbers.

  5. Repeated calls to "srand()" are not a good isolation mechanism
    It might be tempting to approximate separate streams by repeatedly
    reseeding Perl's global generator.

    That is not equivalent to creating independent random streams.

    "drand48" has only 48 bits of state. Reseeding merely selects another
    state within that relatively small state space. In an LCG, different
    seeds correspond to positions within the deterministic structure of the
    generator; they do not create intrinsically independent generators.

    With a sufficiently small state space, careless reseeding also makes it
    possible to move into state which has already been traversed.

    Repeatedly changing shared state introduces another coordination problem
    as well: every component must know when it is safe to call "srand()",
    because doing so changes the sequence observed by every other consumer.

    "srand()" is useful for initializing the generator. It is not a good
    abstraction for partitioning an application into independent random
    streams.

  6. There is no common protocol
    Individual libraries can solve parts of these problems themselves.

    A module can:

    *   accept an RNG object;

    *   accept a callback;

    *   expose its own seed;

    *   use a separate CPAN random-number package; or

    *   define its own global hook.

    Each approach may be perfectly reasonable in isolation.

    What is missing is a standard mechanism connecting such a choice to
    Perl's ordinary "rand()" interface.

    Without one, independently written components cannot easily cooperate on
    questions such as:

    *   which PRNG should this application use?

    *   how should deterministic execution be requested?

    *   how should independent random streams be created?

    *   how should child streams relate deterministically to a parent
        stream?

    *   how should Perl and XS code obey the same randomness policy?

    The missing piece is therefore not another random-number generator.

    It is an abstraction boundary and a common protocol.

  7. Users cannot meaningfully choose the PRNG
    A 48-bit LCG is no longer a particularly strong general-purpose source
    of pseudorandom numbers.

    There are many newer designs with substantially better statistical
    properties, and different generators offer different trade-offs in
    speed, state size, reproducibility, stream management, and statistical
    behaviour.

    Perl does not need to decide which of these should universally replace
    "drand48".

    Indeed, because of the compatibility problem described above, it
    probably should not.

    What users need instead is the ability to make that choice themselves.

    An application which wants a modern generator should be able to use one.
    An application which requires historical Perl behaviour should be able
    to continue using "drand48". A test harness may want a deliberately
    simple deterministic generator. Another application may have completely
    different requirements.

    At present, using a different generator generally means rewriting the
    code which consumes randomness, intercepting Perl's core functions, or
    introducing a separate API.

    The interface gives the user almost no control over what lies behind
    "rand()".

EXISTING SOLUTIONS
    There are already mechanisms which solve portions of this problem, but
    none provides a language-wide solution.

  Override "CORE::rand" and "CORE::srand"
    It is possible to arrange for calls to Perl's core functions to be
    replaced before consuming code is compiled.

    This addresses some use cases, but it is an invasive mechanism. It is
    sensitive to compilation and load order, is awkward to compose, and is
    not a natural protocol through which independently developed software
    can cooperate.

    When the practical answer to selecting an implementation is "override
    the core function", that is a good indication that an abstraction is
    missing beneath the core function.

  Change Perl's internal implementation
    Perl could simply replace "drand48".

    This is technically possible, but it would break deterministic behaviour
    which has accumulated over many years while solving almost none of the
    architectural problems described above.

    There would still be a single implicit shared generator. Components
    would still perturb one another. Users would still have no standard way
    to choose another generator.

    For that reason, this proposal does not depend on replacing Perl's
    existing generator. "drand48" can remain the default indefinitely for
    compatibility.

  Use an explicit RNG object
    Applications and libraries can use RNG objects rather than "rand()".

    This is often a good local design because the state becomes explicit and
    multiple generators can coexist.

    The problem is that it only helps software written to participate in
    that particular interface.

    Existing code:

        my $x = rand();

    remains attached to Perl's normal random stream.

    Furthermore, two independently developed libraries may accept completely
    different RNG abstractions. The application's ability to make its
    overall behaviour deterministic then depends on knowing about each one
    separately.

    Explicit RNG objects are useful, but without a common bridge to "rand()"
    they cannot solve the ecosystem-wide problem.

PRIOR ART
  Perl: $List::Util::RAND
    There is already particularly relevant prior art in Perl itself.

    Since List::Util 1.54, the module has exposed:

        local $List::Util::RAND = sub {
            ...
        };

    Functions which require random values, including "shuffle" and "sample",
    use this callback when it is installed. Otherwise they fall back to
    Perl's built-in "rand()".

    Conceptually, List::Util already implements:

        Need a random value
                |
                v
        $List::Util::RAND installed?
               / \
             yes  no
              |    |
              v    v
           callback rand()

    This is very close to the model proposed here.

    The important difference is scope.

    $List::Util::RAND solves the problem for List::Util. "${^RNG}"
    generalizes the same pattern to Perl's language-level random-number
    interface.

    Instead of individual libraries inventing:

        $Some::Module::RAND
        $Another::Module::RNG
        $Something::Else::random_source

    there is one common point underneath ordinary "rand()".

    This is also useful precedent for concerns about allowing a caller to
    provide a "bad" RNG.

    $List::Util::RAND deliberately allows arbitrary caller-provided code
    satisfying its callback contract. A caller could provide a statistically
    terrible generator, or even a constant function. That possibility has
    not required List::Util to prohibit the abstraction.

    The interface defines the contract; the caller is responsible for
    choosing an implementation appropriate to its needs.

    "${^RNG}" generalizes an established Perl design rather than introducing
    an entirely novel concept.

  Similar concerns in other languages
    Perl is not unique in having encountered these problems. Random-number
    APIs in a variety of other languages and numerical environments have
    evolved to address similar concerns, although the exact solutions are
    necessarily shaped by each language's execution model and compatibility
    constraints.

    Despite those differences, several common themes recur: treating RNG
    state or providers as first-class entities; allowing explicit generators
    to coexist with a convenient default; providing some form of contextual
    or isolated random stream; preserving deterministic behaviour where
    compatibility requires it; and supporting deterministic derivation of
    independent streams for testing or parallel execution.

    These designs are not direct precedents for "${^RNG}", and none should
    be taken as a model Perl ought simply to copy. They are nevertheless
    evidence that the underlying problems -- hidden shared state,
    reproducibility, isolation, composability, and selection of the
    generator -- are general API design concerns rather than peculiarities
    of Perl.

    The especially relevant precedent remains $List::Util::RAND, because it
    already applies essentially the same delegation pattern within Perl
    itself.

WHAT THE INTERFACE SHOULD ALLOW
    In an ideal Perl, the randomness interface should provide several
    properties.

  1. Replace the generator without rewriting consumers
    I should be able to write ordinary Perl code using:

        rand()

    and allow the application containing that code to choose which PRNG
    supplies the values.

    A library should not need to know whether the application has selected
    "drand48", PCG, a deterministic test generator, or something else.

  2. Make an entire application deterministic
    There should be a standard way to say:

        For this execution, make randomness deterministic.

    That decision should be capable of propagating through cooperating
    components without requiring the caller to discover and seed several
    unrelated RNG implementations.

  3. Create isolated random streams
    It should be possible to give separate components independent random
    streams so that consumption by one does not perturb the others.

    At the same time, stream isolation should not prevent deterministic
    execution of the application as a whole.

    Ideally, a deterministic parent configuration should be capable of
    producing deterministic child streams.

    The current proposal provides the necessary interception point for
    establishing such a convention, although it does not yet define the
    complete protocol for deriving and managing those streams.

  4. Change RNG policy in a particular context
    An application should be able to use a chosen generator in a particular
    context without changing the source code of every component involved.

    This allows code with different requirements to coexist.

    Code which depends on historical "drand48" behaviour can retain it,
    while another context can select a different generator.

  5. Work consistently from Perl and XS
    The abstraction should live beneath Perl's normal random-number
    operation rather than merely being another Perl utility function.

    Perl and XS code using Perl's normal RNG machinery should therefore
    observe the same selected provider.

    A language-wide solution is considerably more useful than one which only
    intercepts calls written directly in Perl.

PROPOSAL: MAKE "rand()" PLUGGABLE
    The proposal is to introduce a standard indirection between Perl's
    "rand()" operation and the PRNG which implements it.

    A new special variable:

        ${^RNG}

    is used to install the RNG object or callback for the current context.
    The variable is forced into the "main" namespace in the same manner as
    other appropriate special variables.

    Perl's internal random-number path -- currently reaching "Drand01()" --
    is changed so that it checks "${^RNG}" and delegates to the installed
    provider when one is present.

    The exact internal calling convention, caching, and fast paths are
    implementation details and are deliberately omitted here.

    Conceptually the mechanism is:

                          rand()
                            |
                            v
                     Perl RNG interface
                        /       \
                       /         \
                ${^RNG} set?      no
                     |             |
                    yes            v
                     |          drand48
                     v
               installed RNG

    If "${^RNG}" is unset, nothing changes.

    Perl continues to use its existing "drand48" implementation and existing
    programs retain their historical behaviour.

    If "${^RNG}" is set, existing code which calls "rand()" automatically
    obtains its values from the installed provider.

    This is essentially the $List::Util::RAND model moved one layer
    downward:

        List::Util today:

            shuffle()
               |
               +-- $List::Util::RAND --> custom RNG
               |
               +-- otherwise ---------> rand()


        Proposed Perl model:

            shuffle()
               |
               v
             rand()
               |
               +-- ${^RNG} -----------> custom RNG
               |
               +-- otherwise ---------> drand48

    With the language-level hook, List::Util and every other well-behaved
    consumer of "rand()" can participate in the same randomness policy
    automatically.

WHY THIS APPROACH
    The central goal of this proposal is not to replace "drand48".

    The goal is to separate two questions which Perl currently couples
    together:

    1.  What interface does Perl code use when it needs a random number?

    2.  Which generator and state supply that random number?

    "rand()" is already Perl's universal and familiar answer to the first
    question.

    There is considerable value in retaining it.

    What is missing is a standard answer to the second.

  It preserves compatibility completely by default
    If "${^RNG}" is not used, Perl behaves exactly as it does today.

    "rand()" continues to use "drand48". Existing deterministic sequences do
    not need to change. Programs which depend on historical behaviour can
    continue doing so.

    There is no requirement that Perl ever change its default generator.

  It gives control to the application
    The code which consumes random numbers does not necessarily know the
    requirements of the application in which it will eventually run.

    A reusable library should be able to say:

        rand()

    and leave the choice of RNG policy to the environment which composes
    that library with the rest of the application.

    This is the same inversion of control already demonstrated on a smaller
    scale by $List::Util::RAND.

  It retrofits existing software
    Perhaps the most important property is that software does not need to
    have anticipated the requirement.

    A module containing:

        my $n = rand();

    can participate without modification.

    That makes the proposal fundamentally different from introducing another
    RNG class which is useful only to new or rewritten software.

  It improves composability
    Instead of every component introducing its own RNG hook, applications
    gain a common facility.

    A component which wants its own isolated stream can still have one. But
    that stream can be created according to a shared convention derived from
    the application's overall RNG policy rather than through an unrelated
    collection of seeds and APIs.

    The proposed "${^RNG}" variable does not, by itself, define that
    complete convention.

    It creates the common abstraction point required to make such a
    convention possible.

  It generalizes an existing Perl pattern
    The proposal does not require Perl to adopt a foreign abstraction.

    $List::Util::RAND already implements essentially the same policy:

        Use the caller-provided RNG if one exists; otherwise use Perl's
        normal rand().

    The proposed language-level policy is:

        Use the caller-provided Perl RNG if one exists; otherwise use
        Perl's normal drand48 implementation.

    The important change is simply where the hook lives.

    Putting it beneath "rand()" means that the convention applies uniformly
    to all cooperating Perl and XS code rather than only to one module.

  It does not need to protect users from deliberately bad generators
    A pluggable interface inevitably permits a caller to install a poor
    generator.

    That is a feature of giving the caller control.

    The language should define the contract which an RNG provider must
    satisfy. It does not need to guarantee that every provider chosen by an
    application has good statistical properties.

    Again, $List::Util::RAND already establishes this principle.

    For example, nothing conceptually prevents a caller from supplying:

        local $List::Util::RAND = sub { 0.5 };

    That does not make the hook itself defective. It means the caller has
    selected an inappropriate implementation for most purposes.

    "${^RNG}" should follow the same philosophy.

    The existing default remains unchanged. Choosing an alternative is an
    explicit action, and responsibility for the properties of that
    alternative belongs to the code which chooses it.

  It fixes the abstraction rather than the algorithm
    Replacing one PRNG with another would leave Perl with the same
    architectural problem.

    Adding a standard indirection solves a different and more fundamental
    issue.

    It allows Perl to retain "drand48" indefinitely for backwards
    compatibility while giving applications which need something else a
    supported way to obtain it.

    The stable interface remains:

        rand()

    The implementation becomes policy.

SUMMARY
    Perl's problem with random numbers is not primarily that "drand48" is
    old.

    The deeper problem is that the current "rand()" interface combines three
    independent concepts:

    *   requesting a random value;

    *   selecting a PRNG implementation; and

    *   selecting the random stream from which that value is drawn.

    Today these are collapsed into one implicit interpreter-global
    generator.

    That causes otherwise unrelated components to perturb one another, makes
    random streams difficult to isolate, complicates deterministic testing,
    and gives applications very little control over the PRNG used by their
    dependencies.

    These are not uniquely Perl problems. Other languages and numerical
    environments have developed a range of context-specific mechanisms for
    explicit RNG state, isolated streams, deterministic execution, and
    selectable providers. The details vary considerably, but the recurring
    concerns are much the same.

    More importantly, Perl itself already contains a smaller-scale version
    of the proposed solution. List::Util's $RAND hook means, in effect, "use
    my supplied random provider if present; otherwise use "rand()"."

    The proposed "${^RNG}" mechanism generalizes that idea.

    Instead of making every random-consuming library provide its own hook,
    Perl provides the hook once, beneath "rand()" itself.

    Existing code remains unchanged. Existing "drand48" behaviour remains
    unchanged. Code which cares about the generator gains control over it.
    Code which does not care continues to call "rand()" exactly as before.

    The objective is not to choose a better random number generator for
    Perl.

    It is to let Perl programs choose their random number generator.


-- 
perl -Mre=debug -e "/just|another|perl|hacker/"