Re: [PATCH] ipa-inline: credit aggregate copies into the return slot as eliminated by inlining

Ping.
Thanks,
Kyrill

> On 14 Jul 2026, at 15:44, Kyrylo Tkachov <[email protected]> wrote:
> 
> Ping.
> Thanks,
> Kyrill
> 
>> On 6 Jul 2026, at 20:32, Kyrylo Tkachov <[email protected]> wrote:
>> 
>> Ping.
>> https://gcc.gnu.org/pipermail/gcc-patches/2026-June/719927.html
>> Thanks,
>> Kyrill
>> 
>>> On 10 Jun 2026, at 13:37, Kyrylo Tkachov <[email protected]> wrote:
>>> 
>>> From: Kyrylo Tkachov <[email protected]>
>>> 
>>> eliminated_by_inlining_prob only credited a write to the return value as
>>> free-after-inlining when the RHS was a register or invariant (a scalar
>>> "return x;").  A function that returns an aggregate by value
>>> ("V t = a; ...; return t;") therefore had the copy of the result into the
>>> return slot charged in full to the inliner's growth estimate, even though
>>> return-slot optimization / SRA eliminates that copy once the call is inlined
>>> (the inlined callee writes the caller's destination in place).
>>> 
>>> For a small value-type wrapper -- a SIMD/array lane type, a std::array-like
>>> POD, std::complex, a short Tensor/Matrix -- whose overloaded operators are
>>> written "V op(const V &a, const V &b) { V t = a; t op= b; return t; }", the
>>> mispriced return-slot copy pushes the wrapper over --param
>>> early-inlining-insns, so the early inliner leaves it out of line.  Such
>>> operators are called pervasively, and keeping them out of line also forces
>>> their operands and result through memory in the caller.
>>> 
>>> Credit aggregate copies whose destination is the return value (a RESULT_DECL,
>>> or a store through the invisible-reference return pointer) as eliminated by
>>> inlining, mirroring the existing treatment of reads of by-reference
>>> parameters.  Statements with volatile operands are excluded: a volatile
>>> access is never elided, so such a copy is not free after inlining.
>>> 
>>> This is a size-model refinement only; no IR is changed and no semantics are
>>> affected.
>>> 
>>> This gives about 8.5% on an aarch64 machine on the 766.femflow_r
>>> benchmark from SPEC2026 where these routines are critical to performance
>>> and their vectorised form needs to be inlined for good performance.
>>> 
>>> Bootstrapped and tested on aarch64-none-linux-gnu.
>>> 
>>> Signed-off-by: Kyrylo Tkachov <[email protected]>
>>> 
>>> gcc/ChangeLog:
>>> 
>>> * ipa-fnsummary.cc (eliminated_by_inlining_prob): Credit aggregate
>>> copies into the return slot (a RESULT_DECL or invisible-reference
>>> return) as eliminated by inlining.
>>> 
>>> gcc/testsuite/ChangeLog:
>>> 
>>> * g++.dg/ipa/inline-aggregate-retval.C: New test.
>>> * gcc.dg/ipa/inline-aggregate-retval.c: New test.
>>> ---
>>> gcc/ipa-fnsummary.cc                          | 27 ++++++++++++++++
>>> .../g++.dg/ipa/inline-aggregate-retval.C      | 21 +++++++++++++
>>> .../gcc.dg/ipa/inline-aggregate-retval.c      | 31 +++++++++++++++++++
>>> 3 files changed, 79 insertions(+)
>>> create mode 100644 gcc/testsuite/g++.dg/ipa/inline-aggregate-retval.C
>>> create mode 100644 gcc/testsuite/gcc.dg/ipa/inline-aggregate-retval.c
>>> 
>>> diff --git a/gcc/ipa-fnsummary.cc b/gcc/ipa-fnsummary.cc
>>> index a6ccdb1852a..c05ffc313ed 100644
>>> --- a/gcc/ipa-fnsummary.cc
>>> +++ b/gcc/ipa-fnsummary.cc
>>> @@ -1476,6 +1476,33 @@ eliminated_by_inlining_prob (ipa_func_body_info *fbi, gimple *stmt)
>>> if (lhs_free
>>>    && (is_gimple_reg (rhs) || is_gimple_min_invariant (rhs)))
>>>  rhs_free = true;
>>> +  /* A by-value aggregate return ("T t = ...; return t;") stores the
>>> +     result into the return slot.  Under the return-slot optimization the
>>> +     inlined callee writes the caller's destination in place and SRA then
>>> +     scalarises the small aggregate, so the store is eliminated by inlining
>>> +     -- the return-value analogue of the by-reference parameter reads
>>> +     credited above.  The is_gimple_reg/is_gimple_min_invariant test only
>>> +     credits scalar returns; also credit an aggregate return whose source is
>>> +     an ordinary memory value.  This is the value-type wrapper idiom
>>> +     (a SIMD/array lane type, std::array, std::complex, a short Tensor):
>>> +     "T operator OP (const T &a, const T &b) { T t = a; t OP= b; return t; }".
>>> +     This is the whole-aggregate-copy case.  */
>>> +  if (!rhs_free
>>> +      && !gimple_has_volatile_ops (stmt)
>>> +      && AGGREGATE_TYPE_P (TREE_TYPE (lhs))
>>> +      && gimple_assign_rhs_class (stmt) == GIMPLE_SINGLE_RHS
>>> +      && (TREE_CODE (inner_lhs) == RESULT_DECL
>>> +  || (TREE_CODE (inner_lhs) == MEM_REF
>>> +      && TREE_CODE (TREE_OPERAND (inner_lhs, 0)) == SSA_NAME
>>> +      && SSA_NAME_VAR (TREE_OPERAND (inner_lhs, 0))
>>> +      && TREE_CODE (SSA_NAME_VAR (TREE_OPERAND (inner_lhs, 0)))
>>> + == RESULT_DECL))
>>> +      && (TREE_CODE (inner_rhs) == MEM_REF
>>> +  || TREE_CODE (inner_rhs) == COMPONENT_REF
>>> +  || VAR_P (inner_rhs)
>>> +  || TREE_CODE (inner_rhs) == PARM_DECL
>>> +  || TREE_CODE (inner_rhs) == RESULT_DECL))
>>> +    return 2;
>>> if (lhs_free && rhs_free)
>>>  return 1;
>>> }
>>> diff --git a/gcc/testsuite/g++.dg/ipa/inline-aggregate-retval.C b/gcc/testsuite/g++.dg/ipa/inline-aggregate-retval.C
>>> new file mode 100644
>>> index 00000000000..a60bad00b66
>>> --- /dev/null
>>> +++ b/gcc/testsuite/g++.dg/ipa/inline-aggregate-retval.C
>>> @@ -0,0 +1,21 @@
>>> +// The value-type operator-overload idiom: a by-value 'operator*' returns its
>>> +// result aggregate, and the copy of that result into the return slot is
>>> +// eliminated by inlining (return-slot optimization / SRA).  It used to be
>>> +// charged to the early-inliner growth, pushing the operator over
>>> +// --param early-inlining-insns.  -fno-inline-functions isolates the early
>>> +// inliner so the operator is inlined only when the copy is credited.
>>> +// { dg-do compile }
>>> +// { dg-options "-O3 --param early-inlining-insns=14 -fno-inline-functions -fno-partial-inlining -fdump-tree-einline-optimized" }
>>> +
>>> +struct V {
>>> +  double d[8];
>>> +  V &operator*= (const V &o) { for (int i = 0; i < 8; i++) d[i] *= o.d[i]; return *this; }
>>> +};
>>> +
>>> +static V operator* (const V &a, const V &b) { V t = a; return t *= b; }
>>> +
>>> +V ga, gb, gc, r;
>>> +
>>> +void use () { r = ga * gb * gc; }
>>> +
>>> +// { dg-final { scan-tree-dump "Inlining.*operator.*into.*use" "einline" } }
>>> diff --git a/gcc/testsuite/gcc.dg/ipa/inline-aggregate-retval.c b/gcc/testsuite/gcc.dg/ipa/inline-aggregate-retval.c
>>> new file mode 100644
>>> index 00000000000..e21cff3438f
>>> --- /dev/null
>>> +++ b/gcc/testsuite/gcc.dg/ipa/inline-aggregate-retval.c
>>> @@ -0,0 +1,31 @@
>>> +/* A small by-value wrapper that returns its aggregate result by value.  The
>>> +   copy of the result into the return slot is eliminated by inlining
>>> +   (return-slot optimization / SRA combines it with the caller's destination),
>>> +   but it used to be charged in full to the early-inliner growth estimate,
>>> +   pushing the wrapper over --param early-inlining-insns and leaving it out of
>>> +   line.  -fno-inline-functions isolates the early inliner, so 'mul' can only
>>> +   be inlined when the return-slot copy is correctly credited as eliminable.  */
>>> +/* { dg-do compile } */
>>> +/* { dg-options "-O3 --param early-inlining-insns=14 -fno-inline-functions -fno-partial-inlining -fdump-tree-einline-optimized" } */
>>> +
>>> +struct V { double d[8]; };
>>> +
>>> +static struct V
>>> +mul (const struct V *a, const struct V *b)
>>> +{
>>> +  struct V t = *a;
>>> +  for (int i = 0; i < 8; i++)
>>> +    t.d[i] *= b->d[i];
>>> +  return t;
>>> +}
>>> +
>>> +struct V ga, gb, gc, r;
>>> +
>>> +void
>>> +use (void)
>>> +{
>>> +  struct V x = mul (&ga, &gb);
>>> +  r = mul (&x, &gc);
>>> +}
>>> +
>>> +/* { dg-final { scan-tree-dump "Inlining mul.* into use" "einline" } } */
>>> -- 
>>> 2.50.1 (Apple Git-155)
>>> 
>> 
>> <favicon.ico>
>