Re: RFC on ${^RNG} callback API for overriding rand()
[email protected] (demerphq)
| Newsgroups | perl.perl5.porters |
|---|---|
| Message-ID | <CANgJU+UiNEkXUOqGvp+cFdZ5yK6Scs5R+WPWZKgs6DTE_dgcCQ@mail.gmail.com> |
On Mon, 17 Aug 2026 at 15:34, demerphq <[email protected]> wrote: > Please see > > https://github.com/Perl/perl5/pull/24722 > > And consider it a POC/RFC for hooking rand() sanely. > I feel like I have failed to express my intent, and my perspective of what I see as the problem that I am trying to solve with this. I had assumed that between the commit message and the content of the change that it would be clear. But I've seen enough comments which suggest to me that I didn't explain myself very well. So before I engage with them in the github discussion thread I wanted to post this here. The pod form is attached. NAME Making Perl's Random Number Generator Pluggable BACKGROUND AND FACTS Perl exposes pseudorandom number generation through the traditional C-like "rand()" and "srand()" functions. Perl exposes randomness through: srand($seed); my $value = rand(); The generator itself is implicit. Code does not normally hold an RNG object or identify which generator it wishes to use. Instead, calls to "rand()" operate on shared interpreter state. Consequently, code throughout a Perl interpreter which calls "rand()" consumes values from the same pseudorandom stream. Since Perl 5.20, "rand()" has used the "drand48" pseudorandom number generator. "drand48" is a linear congruential generator (LCG) with 48 bits of state, and it is fast, widely understood, and of adequate statistical quality for many ordinary non-cryptographic/non-mathematical uses. There is no way to change the internal RNG. THe only real leverage a developer has over rand() and srand() is that it is possible to install alternatives in the global symbol table. This combination of a global fixed implementation and implicit shared state creates a number of problems. PROBLEMS WITH THE CURRENT MODEL 1. The implementation has become part of the compatibility surface One useful property of a pseudorandom generator is reproducibility. Given the same generator and initial state, it produces the same sequence. Perl programmers have naturally made use of this in tests: srand(12345); # randomized test follows The problem is that such tests are dependent not merely on the seed, but on the exact sequence generated by Perl's current implementation. Changing the underlying generator in any way would cause the same seed to produce a different sequence. Tests -- and potentially other programs -- which have come to depend on that sequence could consequently change behaviour or fail. Thus drand48 has therefore become baked into Perl's de facto compatibility surface, we cant change it in nearly anyway without breaking a lot of code. Whatever theoretical benefits might result from replacing "drand48", the practical compatibility cost makes doing so unattractive. So if we want the benefits of a better RNG then we need to find an alternative. 2. Unrelated consumers interfere with one another Because "rand()" operates on shared state, otherwise unrelated pieces of software affect each other's randomized behaviour. Consider: srand(12345); do_something_random(); do_something_else_random(); If some dependency begins making an additional call to "rand()", every subsequent consumer observes a different part of the sequence. A minor modification to one component can therefore change randomized behaviour elsewhere in the program. This is particularly awkward when reproducing failures. Adding debugging code, changing an implementation detail, or upgrading a dependency can perturb the random stream and cause the behaviour under investigation to change. The source of this coupling is not randomness itself. It is the single implicit random stream shared by otherwise independent components. 3. Random streams are difficult to isolate There is no straightforward standard Perl mechanism by which one component can say: Continue using ordinary rand(), but use a random stream independent of the rest of the application. A component can instead use an explicit RNG object or another random-number API. But that requires the component to have been written specifically to do so. Isolation therefore has to be designed into each consumer rather than being something the surrounding application can control. 4. "rand()" is poorly composable Suppose a library correctly and conventionally uses Perl's standard "rand()" function. An application using that library may later want to: * use a higher-quality PRNG; * use a particular deterministic generator during testing; * give one subsystem its own random stream; * reproduce a failure from a known PRNG state; or * use a generator with properties appropriate to a particular application. There is no clean general mechanism by which the caller can make the library's existing calls to "rand()" use that generator. The usual solution is to modify the library so that it accepts an RNG object, callback, or some other source of random values. That is often good API design for an individual component, but it does not solve the problem at the language level. Indeed, when every component invents its own mechanism, application-wide determinism becomes harder: seed Perl's rand() seed subsystem A's RNG seed subsystem B's RNG seed library C's RNG ... One can easily imagine a sufficiently complicated test having to initialize several unrelated RNGs correctly just to reproduce one execution. That way lies madness. The application should be able to establish a randomness policy without having to know the implementation details of every component which happens to need random numbers. 5. Repeated calls to "srand()" are not a good isolation mechanism It might be tempting to approximate separate streams by repeatedly reseeding Perl's global generator. That is not equivalent to creating independent random streams. "drand48" has only 48 bits of state. Reseeding merely selects another state within that relatively small state space. In an LCG, different seeds correspond to positions within the deterministic structure of the generator; they do not create intrinsically independent generators. With a sufficiently small state space, careless reseeding also makes it possible to move into state which has already been traversed. Repeatedly changing shared state introduces another coordination problem as well: every component must know when it is safe to call "srand()", because doing so changes the sequence observed by every other consumer. "srand()" is useful for initializing the generator. It is not a good abstraction for partitioning an application into independent random streams. 6. There is no common protocol Individual libraries can solve parts of these problems themselves. A module can: * accept an RNG object; * accept a callback; * expose its own seed; * use a separate CPAN random-number package; or * define its own global hook. Each approach may be perfectly reasonable in isolation. What is missing is a standard mechanism connecting such a choice to Perl's ordinary "rand()" interface. Without one, independently written components cannot easily cooperate on questions such as: * which PRNG should this application use? * how should deterministic execution be requested? * how should independent random streams be created? * how should child streams relate deterministically to a parent stream? * how should Perl and XS code obey the same randomness policy? The missing piece is therefore not another random-number generator. It is an abstraction boundary and a common protocol. 7. Users cannot meaningfully choose the PRNG A 48-bit LCG is no longer a particularly strong general-purpose source of pseudorandom numbers. There are many newer designs with substantially better statistical properties, and different generators offer different trade-offs in speed, state size, reproducibility, stream management, and statistical behaviour. Perl does not need to decide which of these should universally replace "drand48". Indeed, because of the compatibility problem described above, it probably should not. What users need instead is the ability to make that choice themselves. An application which wants a modern generator should be able to use one. An application which requires historical Perl behaviour should be able to continue using "drand48". A test harness may want a deliberately simple deterministic generator. Another application may have completely different requirements. At present, using a different generator generally means rewriting the code which consumes randomness, intercepting Perl's core functions, or introducing a separate API. The interface gives the user almost no control over what lies behind "rand()". EXISTING SOLUTIONS There are already mechanisms which solve portions of this problem, but none provides a language-wide solution. Override "CORE::rand" and "CORE::srand" It is possible to arrange for calls to Perl's core functions to be replaced before consuming code is compiled. This addresses some use cases, but it is an invasive mechanism. It is sensitive to compilation and load order, is awkward to compose, and is not a natural protocol through which independently developed software can cooperate. When the practical answer to selecting an implementation is "override the core function", that is a good indication that an abstraction is missing beneath the core function. Change Perl's internal implementation Perl could simply replace "drand48". This is technically possible, but it would break deterministic behaviour which has accumulated over many years while solving almost none of the architectural problems described above. There would still be a single implicit shared generator. Components would still perturb one another. Users would still have no standard way to choose another generator. For that reason, this proposal does not depend on replacing Perl's existing generator. "drand48" can remain the default indefinitely for compatibility. Use an explicit RNG object Applications and libraries can use RNG objects rather than "rand()". This is often a good local design because the state becomes explicit and multiple generators can coexist. The problem is that it only helps software written to participate in that particular interface. Existing code: my $x = rand(); remains attached to Perl's normal random stream. Furthermore, two independently developed libraries may accept completely different RNG abstractions. The application's ability to make its overall behaviour deterministic then depends on knowing about each one separately. Explicit RNG objects are useful, but without a common bridge to "rand()" they cannot solve the ecosystem-wide problem. PRIOR ART Perl: $List::Util::RAND There is already particularly relevant prior art in Perl itself. Since List::Util 1.54, the module has exposed: local $List::Util::RAND = sub { ... }; Functions which require random values, including "shuffle" and "sample", use this callback when it is installed. Otherwise they fall back to Perl's built-in "rand()". Conceptually, List::Util already implements: Need a random value | v $List::Util::RAND installed? / \ yes no | | v v callback rand() This is very close to the model proposed here. The important difference is scope. $List::Util::RAND solves the problem for List::Util. "${^RNG}" generalizes the same pattern to Perl's language-level random-number interface. Instead of individual libraries inventing: $Some::Module::RAND $Another::Module::RNG $Something::Else::random_source there is one common point underneath ordinary "rand()". This is also useful precedent for concerns about allowing a caller to provide a "bad" RNG. $List::Util::RAND deliberately allows arbitrary caller-provided code satisfying its callback contract. A caller could provide a statistically terrible generator, or even a constant function. That possibility has not required List::Util to prohibit the abstraction. The interface defines the contract; the caller is responsible for choosing an implementation appropriate to its needs. "${^RNG}" generalizes an established Perl design rather than introducing an entirely novel concept. Similar concerns in other languages Perl is not unique in having encountered these problems. Random-number APIs in a variety of other languages and numerical environments have evolved to address similar concerns, although the exact solutions are necessarily shaped by each language's execution model and compatibility constraints. Despite those differences, several common themes recur: treating RNG state or providers as first-class entities; allowing explicit generators to coexist with a convenient default; providing some form of contextual or isolated random stream; preserving deterministic behaviour where compatibility requires it; and supporting deterministic derivation of independent streams for testing or parallel execution. These designs are not direct precedents for "${^RNG}", and none should be taken as a model Perl ought simply to copy. They are nevertheless evidence that the underlying problems -- hidden shared state, reproducibility, isolation, composability, and selection of the generator -- are general API design concerns rather than peculiarities of Perl. The especially relevant precedent remains $List::Util::RAND, because it already applies essentially the same delegation pattern within Perl itself. WHAT THE INTERFACE SHOULD ALLOW In an ideal Perl, the randomness interface should provide several properties. 1. Replace the generator without rewriting consumers I should be able to write ordinary Perl code using: rand() and allow the application containing that code to choose which PRNG supplies the values. A library should not need to know whether the application has selected "drand48", PCG, a deterministic test generator, or something else. 2. Make an entire application deterministic There should be a standard way to say: For this execution, make randomness deterministic. That decision should be capable of propagating through cooperating components without requiring the caller to discover and seed several unrelated RNG implementations. 3. Create isolated random streams It should be possible to give separate components independent random streams so that consumption by one does not perturb the others. At the same time, stream isolation should not prevent deterministic execution of the application as a whole. Ideally, a deterministic parent configuration should be capable of producing deterministic child streams. The current proposal provides the necessary interception point for establishing such a convention, although it does not yet define the complete protocol for deriving and managing those streams. 4. Change RNG policy in a particular context An application should be able to use a chosen generator in a particular context without changing the source code of every component involved. This allows code with different requirements to coexist. Code which depends on historical "drand48" behaviour can retain it, while another context can select a different generator. 5. Work consistently from Perl and XS The abstraction should live beneath Perl's normal random-number operation rather than merely being another Perl utility function. Perl and XS code using Perl's normal RNG machinery should therefore observe the same selected provider. A language-wide solution is considerably more useful than one which only intercepts calls written directly in Perl. PROPOSAL: MAKE "rand()" PLUGGABLE The proposal is to introduce a standard indirection between Perl's "rand()" operation and the PRNG which implements it. A new special variable: ${^RNG} is used to install the RNG object or callback for the current context. The variable is forced into the "main" namespace in the same manner as other appropriate special variables. Perl's internal random-number path -- currently reaching "Drand01()" -- is changed so that it checks "${^RNG}" and delegates to the installed provider when one is present. The exact internal calling convention, caching, and fast paths are implementation details and are deliberately omitted here. Conceptually the mechanism is: rand() | v Perl RNG interface / \ / \ ${^RNG} set? no | | yes v | drand48 v installed RNG If "${^RNG}" is unset, nothing changes. Perl continues to use its existing "drand48" implementation and existing programs retain their historical behaviour. If "${^RNG}" is set, existing code which calls "rand()" automatically obtains its values from the installed provider. This is essentially the $List::Util::RAND model moved one layer downward: List::Util today: shuffle() | +-- $List::Util::RAND --> custom RNG | +-- otherwise ---------> rand() Proposed Perl model: shuffle() | v rand() | +-- ${^RNG} -----------> custom RNG | +-- otherwise ---------> drand48 With the language-level hook, List::Util and every other well-behaved consumer of "rand()" can participate in the same randomness policy automatically. WHY THIS APPROACH The central goal of this proposal is not to replace "drand48". The goal is to separate two questions which Perl currently couples together: 1. What interface does Perl code use when it needs a random number? 2. Which generator and state supply that random number? "rand()" is already Perl's universal and familiar answer to the first question. There is considerable value in retaining it. What is missing is a standard answer to the second. It preserves compatibility completely by default If "${^RNG}" is not used, Perl behaves exactly as it does today. "rand()" continues to use "drand48". Existing deterministic sequences do not need to change. Programs which depend on historical behaviour can continue doing so. There is no requirement that Perl ever change its default generator. It gives control to the application The code which consumes random numbers does not necessarily know the requirements of the application in which it will eventually run. A reusable library should be able to say: rand() and leave the choice of RNG policy to the environment which composes that library with the rest of the application. This is the same inversion of control already demonstrated on a smaller scale by $List::Util::RAND. It retrofits existing software Perhaps the most important property is that software does not need to have anticipated the requirement. A module containing: my $n = rand(); can participate without modification. That makes the proposal fundamentally different from introducing another RNG class which is useful only to new or rewritten software. It improves composability Instead of every component introducing its own RNG hook, applications gain a common facility. A component which wants its own isolated stream can still have one. But that stream can be created according to a shared convention derived from the application's overall RNG policy rather than through an unrelated collection of seeds and APIs. The proposed "${^RNG}" variable does not, by itself, define that complete convention. It creates the common abstraction point required to make such a convention possible. It generalizes an existing Perl pattern The proposal does not require Perl to adopt a foreign abstraction. $List::Util::RAND already implements essentially the same policy: Use the caller-provided RNG if one exists; otherwise use Perl's normal rand(). The proposed language-level policy is: Use the caller-provided Perl RNG if one exists; otherwise use Perl's normal drand48 implementation. The important change is simply where the hook lives. Putting it beneath "rand()" means that the convention applies uniformly to all cooperating Perl and XS code rather than only to one module. It does not need to protect users from deliberately bad generators A pluggable interface inevitably permits a caller to install a poor generator. That is a feature of giving the caller control. The language should define the contract which an RNG provider must satisfy. It does not need to guarantee that every provider chosen by an application has good statistical properties. Again, $List::Util::RAND already establishes this principle. For example, nothing conceptually prevents a caller from supplying: local $List::Util::RAND = sub { 0.5 }; That does not make the hook itself defective. It means the caller has selected an inappropriate implementation for most purposes. "${^RNG}" should follow the same philosophy. The existing default remains unchanged. Choosing an alternative is an explicit action, and responsibility for the properties of that alternative belongs to the code which chooses it. It fixes the abstraction rather than the algorithm Replacing one PRNG with another would leave Perl with the same architectural problem. Adding a standard indirection solves a different and more fundamental issue. It allows Perl to retain "drand48" indefinitely for backwards compatibility while giving applications which need something else a supported way to obtain it. The stable interface remains: rand() The implementation becomes policy. SUMMARY Perl's problem with random numbers is not primarily that "drand48" is old. The deeper problem is that the current "rand()" interface combines three independent concepts: * requesting a random value; * selecting a PRNG implementation; and * selecting the random stream from which that value is drawn. Today these are collapsed into one implicit interpreter-global generator. That causes otherwise unrelated components to perturb one another, makes random streams difficult to isolate, complicates deterministic testing, and gives applications very little control over the PRNG used by their dependencies. These are not uniquely Perl problems. Other languages and numerical environments have developed a range of context-specific mechanisms for explicit RNG state, isolated streams, deterministic execution, and selectable providers. The details vary considerably, but the recurring concerns are much the same. More importantly, Perl itself already contains a smaller-scale version of the proposed solution. List::Util's $RAND hook means, in effect, "use my supplied random provider if present; otherwise use "rand()"." The proposed "${^RNG}" mechanism generalizes that idea. Instead of making every random-consuming library provide its own hook, Perl provides the hook once, beneath "rand()" itself. Existing code remains unchanged. Existing "drand48" behaviour remains unchanged. Code which cares about the generator gains control over it. Code which does not care continues to call "rand()" exactly as before. The objective is not to choose a better random number generator for Perl. It is to let Perl programs choose their random number generator. -- perl -Mre=debug -e "/just|another|perl|hacker/"
rng.pod
(application/x-perl, 22.8 KB) - not displayed