Re: AV API thoughts: AVs as proxies for other things

On 6/11/26 8:40 PM, Tony Cook wrote:
> On Thu, Jun 11, 2026 at 11:31:21AM +0100, Paul "LeoNerd" Evans wrote:
>>    https://github.com/Perl/perl5/pull/24451
>>
>> The currently-last comment in that PR I liked above, is some thinking
>> out loud on the complicated problem with the C-level API it proposes.
>> It observes that if at the C level you call `av_splice(...)` on a given
>> AV that is tied, there's no guarantee that the Perl `SPLICE` method that
>> gets invoked returns the right number of deleted items. In particular,
>> it could return more, which would be too many items for the caller to
>> account for in the C-level array it passed in for the purpose. Much
>> trouble. My comment ends with the observation that perhaps a workable
>> API would be to have three variants of the function, which differ in
>> the way they pass in or out a list (sequence?) of SVs for the splice
>> operation. The three variants could use C-level arrays of SV pointers,
>> perl's AV structures, or the perl data stack.
>>
> ...
>> As I said at the top, this isn't a fully-formed plan, or a set of
>> questions. I just wanted to write it out while it was in my head, for
>> future reference in case someone else came to similar ideas before, or
>> in future looking back at this.
> My initial thought thinking over this was that for inputs the (SV**,
> size_t) pair is sufficient, if the caller has a (non-tied) AV they can
> pass (AvARRAY(av), av_count(av)), if they have a local C array they
> can populate that, if the SVs are on the stack they can pass (MARK,
> (PL_stack_sp-MARK)).
>
> But Michael's comment on exceptions vs warnings reminded me that we
> need to deal with exceptions - if we support tied targets how do we
> deal with cleanup of these objects on an exception?
>
> For the stack it's no big deal, the objects can be mortal
> (non-RC_STACK) or owned by the stack (RC_STACK) and are cleaned up on
> rewind, but the others are more messy.  So perhaps your proxies are
> the right way to do it.
>
> Returning stuff is harder as we found.
>
> One option might be the caller supplies an AV and an offset, the API
> extends the AV as needed and adds the elements to the AV, which should
> cover returning on the stack too. (though we'd need to check something
> like (av == PL_curstack && rpp_stack_is_rc()) to decide on reference
> counting handling).
After trying to reply to this a few times with contradictory 
conclusions, I think I've finally arrived at an answer I'm comfortable 
with, and which doesn't require a pseudo-AV concept and doesn't have any 
unnecessary refcount changes to the array elements.

If the caller supplies SV to be copied (sv_setsv) into the array, the 
references must already be mortal or otherwise-owned; the normal XS 
conditions when calling any api that could croak.  Likewise, if the 
caller supplies SV to be added as-is with an additional reference added, 
they must already be mortal or otherwise-owned.  If the caller supplies 
SV that are "live references", the av_splice function must be 
responsible for cleaning them up.

Non-tied AV, Caller supplies SV* to be copied:

   - Create a temporary mortal AV, and populate it with SV copies of the 
supplied SVs.  sv_setsv can die, but this is ok, because the mortal AV 
cleans up the progress so far.
   - The temporary AV is no worse efficiency than if the caller had done 
all the sv_setsv, because they would also need something to own the 
copies until it was known that all copying succeeded.
   - If caller wants mortal references of removed elements, swap 
pointers between AVs.  The removed elements are mortal by virtue of 
being owned by a mortal AV.
   - If caller wants to claim references of removed elements, move 
pointers from target AV to return buffer and move pointers from temp AV 
to target AV

Non-tied AV, Caller supplies SV* to be referenced:
(never dies)

   - If caller wants mortal references, copy out with sv_2mortal
   - If caller wants to claim references, move pointers from AV to 
return buffer
   - Copy SV* into AV, increasing refcount of each

Non-tied AV, Caller supplies SV* to be claimed:
(never dies)

   - If caller wants mortal references, copy out with sv_2mortal
   - If caller wants to claim references, move pointers from AV to 
return buffer
   - Copy SV* into AV

Tied AV, Caller supplies SV* to be copied or referenced:

   - SV* get added to the stack, and SPLICE is called.  On RC_STACK, 
refcounts get incremented.
   - No need to trap exceptions, let stack cleanup handle it
   - If caller wants removed items returned, transfer from the stack

Tied AV, Caller supplies SV* to be claimed:

   - SV* get added to the stack, and SPLICE is called.  On RC_STACK, 
stack is new owner of supplied SVs.  On non-RC stack, sv_2mortal gets 
called on each element.
   - No need to trap exceptions, let stack cleanup handle it
   - If caller wants removed items returned, transfer from the stack

The only real inefficiency here is if the intended return buffer is the 
stack.  The API I originally proposed for av_splice requires the caller 
to have pre-allocated SV** storage for the removed elements, but that 
storage could end up occupied by elements in the return value of tied 
SPLICE, and ideally could just be left on the stack instead of popping 
them off and then pushing them back on.  I have no idea whether "return 
values in an AV which can be a virtual AV backed by the stack" would be 
more or less effort than another flag to av_splice that says "leave 
results on the stack".

-Mike C.