MAGIC on PADNAMEs; and PADNAMEs as SVs

Another question that's deep in the inner details of the interpreter,
which is partly just me "thinking out loud" before I try to do
something.

I have two large long-running branches at the moment, one that adds
"Magic v2", and one that adds "Attributes v2". I've mentioned these
both at length before, but both are now basically at the point where
the really interesting things start to become possible as good
justification use-cases if you combine the two. To briefly summarise so
far:

  *Magic v2* is all about creating more kinds of magic vtable, to have
    more trigger functions that can be invoked at more interesting times
    on the things the magic is attached to

  *Attributes v2* is all about allowing modules to define new
    lexically-scoped named attributes, whose behaviour allows them to
    interrupt the parser and run more code early at compiletime

90% of the good use-cases for Attributes v2 will simply be to attach
some kind of magic on the target of the attribute, and 90% of the good
use-cases for Magic v2 will be to be the interesting behaviours that
attributes attach. This is how they work together. While each is kindof
interesting on its own, it is the combination of both that really
demonstrates useful things and why we want them.

One key thing mentioned in the introduction docs for both of these
ideas was to be able to attach magic and attributes in more places,
and one of those was to put them on lexical variables, such as plain
`my` variables wherever they appear, but also `field` variables of
classes, or signature parameters (which are really just lexical
variables with implicitly-provided initialisation).

What all these use-cases have in common, and what makes them distinct
from applying magic or attributes onto package variables or
subroutines, is that at compiletime when the attribute is actually
being applied there does not exist one unique SV that will become "the
thing" that is being targeted. Most lexical variables, as well as all
fields and parameter signature variables, come into existence as actual
SV storage many times over during the runtime of the program.

The thing that exists at compiletime in each case is the name of the
variable, in the form of a PADNAME structure that's in the PADNAMELIST
of the currently-compiling scope. This is mentioned as a note in the
Attributes v2 PPC document[1] as a potential way to attach magic, and
hence attributes, onto the abstract compiletime concept of these kinds
of variables.

It would at first glance appear to be an easy task here then to pass
PADNAMEs as the targets for attribute v2 declarations, and to attach
magic v2 structures onto them.

However, the latter isn't currently possible, due to the fact that
PADNAME structures are not SVs, and thus cannot currently be used with
magic.

At this point, I come to one of my classic "what do we do?" questions:

  * We could create an entire parallel "magic on PADNAMEs"
    infrastructure with duplicate copies of all the data structures,
    functions, etc copied from "magic on SVs"

  * We could convert PADNAMEs and associated structures into being
    actual SVs, thus allowing us to reüse all of the existing "apply
    magic to an SV" infrastructure we already have.

That second option looks to me to be far more tempting, for two big
reasons. Firstly, as compared to the first plan it would likely involve
a lot less code addition. The first option of creating this entire new
parallel magic system would be a surprisingly large collection of data
types and code, because it needs a whole "magic chain" maintaining on
each PADNAME structure, a whole new set of management functions, virtual
function tables, etc etc... I don't really like that option as it
involves a lot of near-duplication of code and data, and begins a
slippery slope of suggesting that yet other things that aren't SVs
could also have magic concepts applied to them, leading to much further
duplication. It feels largely messy.

But the second reason that this second idea looks very tempting is that
in early versions of Perl, the entire pad infrastructure *was* created
out of SVs. Before 2012, the padlist for a CV was just an AV, all of
whose elements were themselves directly AVs (with no interposed RVs).
The topmost AV in the pad list stored the pad names list, all of whose
elements were plain stringy SVs whose PV part was just the names of each
variable. The remaining AVs then stored the individual pads, all of
whose elements were the corresponding actual SV data structures storing
the values of those variables. It was in many ways a nice neat use of
existing perl internals.

All of that changed in 2012 in a series of work performed by Father
Chrysostomos, which rewrote all of that structure I just explained
above, and made the PADLIST, the PADNAMELIST and PADNAME now be their
own new C-level struct type, rather than reusing perl's AVs and SVs.
I've been digging into the specific commits to try to understand this
surrounding context, and came upon two commits in particular, whose logs
I shall paste part of here:

  Stop padlists from being AVs

  In order to fix a bug, I need to add new fields to padlists.  But I
  cannot easily do that as long as they are AVs.

    -- https://github.com/Perl/perl5/commit/7261499db89d7afd6c64079406dc32f10acfe512

  Make PADNAME a separate type distinct from SV.

  This should fix the CPAN modules that were failing when the
  PadnameLVALUE flag was added, because it shared the same bit as
  SVs_OBJECT and pad names were going through code paths not designed
  to handle pad names.

    -- https://github.com/Perl/perl5/commit/0f94cb1fe27e58a59d3391214dab34037ab184db

I've also had a quick read around the perl5-porters mailing list in the
few years leading up to these changes, but I didn't really see anything
of interest which added further context, so it seems all the history is
contained in just the commit messages.

It seems that the overall theme of these changes was to allow padlists
and padname structures to have new distinct behaviour via new flags and
fields that previously being based on AVs and SVt_PV-style SVs did not
allow them to have.

In the 14 years since those changes were made, we've rearranged the SV
types structure a bit more, most notably by adding the new SVt_PVOBJ
which required expanding the bit field to store the type in various
places, which has allowed us space for a further more 14 types. I feel
it might be feasible to add two more SV types in here for SVt_PADLIST
and SVt_PADNAME to keep them properly distinct. Already for Attributes
v2 we're suggesting we might need a dedicated new SV type to store an
"lexical attribute definition" in anyway, so while we're adding types
that could be part of it.

Other comments in some of the commit messages around the padlist rework
mention that these new structures are now smaller, using less memory
than the previous SV types. That was true initially, but over time a
lot of these structures have gained more fields anyway (such as perl
5.38's adding of the fieldinfo pointer to store details about the class
features's `field` keyword), so these days I think it's less of a
consideration. Plus also, the whole lot of these special types still
uses refcount fields with dedicated refcount-management code, which
could all be removed and worked back into the regular SV refcount
management if we convert them back.

Plus I think there's an overall neatness in having as many things as
possible just reusing the SV infrastructure, because as well as "you
can attach magic", there's also various other side-benefits to do with
memory allocation, heap-walking, debugging for memory leaks, all kinds
of things.

I think I've gone on long enough here, so I'll end with a quick outline
of a suggested plan of action, and see if anyone disagrees with my
thinking:

  1) Create two new dedicated SV types for PADNAME and PADLIST

  2) Convert existing PADNAME and PADLIST custom-structure code into
     using those. This *may* require adding also a PADNAMELIST
     structure, but I am hoping that since PADNAMEs will now be normal
     SVs, it should be possible to return a PADNAMELIST back to being a
     regular AV and not its own weird thing

  3) Ensure that the Magic v2 infrastructure is happy to operate on
     these new PADNAME-as-SV types

  4) Create new Magic v2 functions tables for PADNAMEs. These are
     likely to require a few specific subtypes to use for object fields
     and subroutine signature variables, as each of those use-cases of
     variables have their own particular lifecycle triggering times
     that would be of interest to such magic


[1]: https://github.com/Perl/PPCs/blob/main/ppcs/ppc0029-attributes-v2.md

-- 
Paul "LeoNerd" Evans

[email protected]
http://www.leonerd.org.uk/  |  https://metacpan.org/author/PEVANS