RFC on ${^RNG} callback API for overriding rand()

Please see

https://github.com/Perl/perl5/pull/24722

And consider it a POC/RFC for hooking rand() sanely.

This one is for you Scott.

BTW, currently in theory we allow people to use the Configure process to
override the random number we use, but I am quite sure that would cause
chaos. We should remove all of it from configure, on plan-9 it is just
wrong, and overriding stuff at that level would just break a lot of stuff.
It is entirely vestigial from the early days.

Commit message:

Add support for an ${^RNG} hook, along with support PCG randomization


Historically the random number generator used by perl has evolved over
time, in the early days it was the system RNG, then later we rolled our
own version of drand48 for cross platform stability. But these days

drand48 has a bunch of flaws so it would be nice to be able to use
something better.


However this is not so easy. Most code just calls rand() because it is
the easiest thing to do. Most people dont bother adding support for
their own hook, or burdening their user with providing an RNG,
especially as there is no standard for doing so. Some code maybe ends up
with support for an RNG hook, like List::Util did, but IMO such cases
are a rarity, and a proliferation of slightly different hooks for this
would not be helpful.


In theory one can override CORE::rand() and CORE::srand() before a
module is loaded, and thus change the RNG, but the reality is there is
no good way to overload srand()/rand() in another packages namespace.
(So when people say "just put your pet RNG on CPAN" it is kind of a cop
out, that RNG is likely to never really be useful as it just wont get
used by most code.)


At the same time we dont really want to change Perls internal code to
                    use something other than drand48(). There is lots of
code that expects                       that doing srand(1234) will produce
test results that look a specific                                 way, so
changing cores RNG would cause all kinds of unfortunate drama
        through the ecosystem.


This patch takes a different approach: ${^RNG} is used as a hook for
                      rand() and srand(). If ${^RNG} contains an object
then it uses the                                   objects rand_uv() and
srand() methods to do its thing. If it
          contains an unblessed subref then it calls it with no arguments
for
rand() and with arguments for srand() (srand(undef) is defined to be
the same as srand()).

This is wired into the macro Drand01() which despite its name and
appearance is actually a pTHX style macro, but which actually hides the
fact by access what looks like a global var, but is actually an
interpreter var under threads, thus hiding that it does operate on aTHX
just not directly. This allows us to redirect so that modules like
List::Util will still respect the hook even though they bypass pp_rand()
for most of their random calls.

I thought about using ${^HOOKS}{RNG} instead of ${^RNG}, but for the
time I decided to just use ${^RNG} as it feels better to do
something like:

        local ${^RNG} = RNG::PCG->new();

than it does to do:

        local ${^HOOKS}{RNG}

and it allows the rng internals to avoid the hash lookup. Although
currently it is not optimizing the ${^RNG} package var fetch, and the
var is not magic in any way so the ${^RNG} hook is not being stored in
an interpreter var so the current implementation could be sped up we
wished. IMO it is preferable to keep this as simple as possible.

As part of this patch I have added a new dist/RNG which contains XS code
to implement the two dimensional PCG-XSH-RR 64/128, which allows for 128
bit state with only 64 bit or 32 operations. This seems to be the state
of the art RNG these days.

I also implemented an RNG::SHA which uses SHA256 to produce the stream
of random values.

I chose to make RNG use a UV as its base currency, so rand() calls
${^RNG}->rand_uv() which returns a UV, which is then converted into
whatever the user requested. I did it this way so that any floating
point operations or conversion to floating point operations could be
performed by the C code. It feels like a more flexible foundation
than using a float.

Modules like List::Util which bind to Drand01() at the C level still
call the callback, so even without using List::Util's hook you can
override it just like any other perl code.

cheers,
Yves


-- 
perl -Mre=debug -e "/just|another|perl|hacker/"