Re: some things about Closure

Hi Jochen,

I merged GROOVY-12165 reworking it to I think the direction you were
outlining and fixing GROOVY-12164 as part of that. It is definitely
still CTR, so feedback certainly welcome.

Then I reworked GROOVY-12151 (still marked draft) and updated GEP-27:

https://github.com/apache/groovy/pull/2709
https://groovy.apache.org/wiki/GEP-27.html

Re: "I would like to have this as standard feature, not as opt-in", I
have included the AI read on the opt-in decision below.

Cheers, Paul.

----

Subject: GEP-27 closure packing: factors for the opt-in vs default-on decis=
ion

Here is a summary of the factors bearing on whether
`groovy.target.closure.pack` should eventually default on, to help
that discussion happen against facts rather than impressions. To scope
it first: the question only concerns the @CompileStatic PROVEN path
(the type checker's delegate-independence proof). Dynamic packing
stays annotation-only (@PackedClosures) regardless =E2=80=94 it cannot be
proven sound without types =E2=80=94 so the blast radius of any default is:
statically compiled closures the compiler has proved safe, with the
escape analysis already declining the risky shapes (field-assigned DSL
blocks like Grails constraints never pack, by construction).

In favour of default-on
-----------------------
* Class count and bytes: -93% classes / -73% bytecode on the measurement co=
rpus;
  the Grails domain-class extreme was 24 closure classes (80 KB) for one cl=
ass.
  Less classloading, verification, JIT profiling, metaspace; smaller jars.
* Throughput: capturing shapes (the ones GEP-27 exists for) run 1.4-1.9x FA=
STER
  packed than as generated classes (JMH, current master baseline including =
the
  Closure.call fast-path work), and the win holds under megamorphic dispatc=
h.
  On Grails-shaped in-situ workloads (domain pipelines, validation cycles) =
the
  blend nets +17-23% with identical outputs. Allocation at or below the
  closure-class baseline on capturing shapes.
* Semantics: the MOP matrix is fully green =E2=80=94 per-instance metaclass
interception,
  categories, and respondsTo behave identically to generated closure classe=
s on
  both the dynamic and Java/GDK paths (pinned by tests); corpus transcripts=
 are
  byte-identical flag on/off; verified equally under classic (indy=3Dfalse)
  compilation.
* JPMS: packed dispatch uses the capability model Java lambdas use, so it w=
orks
  under strict encapsulation with no --add-opens, where reflective dispatch=
 of
  closure classes does not.
* Back-out valves at every granularity: the flag, and @PackedClosures(mode =
=3D
  DISABLED) per class or per method.

Costs and residuals (all documented in the @PackedClosures javadoc and GEP-=
27)
---------------------------------------------------------------------------=
---
* The tight non-capturing collect shape runs ~0.6x in microbenchmarks =E2=
=80=94 the
  MOP-transparency guard's per-call checks, which are loop-invariant
and hoist to
  zero when the JIT inlines the chain. Assessed against realistic Grails-sh=
aped
  workloads: no measurable in-situ effect (the blend is +17-23% as above), =
so
  this is a micro-visible, macro-invisible cost.
* Serialization: a packed closure is not serializable, and this now has the=
 same
  three-layer treatment as the delegate boundary =E2=80=94 a literal VISIBL=
Y
  serialization-bound (cast/coerced to a Serializable type, or passed
directly to
  writeObject) declines at compile time and keeps its class (so it serializ=
es as
  before, and WARN/STRICT report it); a transitive route fails fast at runt=
ime
  with a message naming the closure and the opt-out (dehydrate()
cannot help =E2=80=94 the
  dispatch state remains); and the escape analysis already blocks the
main routes
  into serialized object graphs (no field stores, returns, or collections).
* Class identity: closure.getClass() no longer distinguishes literals =E2=
=80=94 tooling
  or code keyed on generated $_closureN names/types will not find them.
* Class-level metaclass changes on the shared adapter are global to packed
  closures (per-instance setMetaClass is fully honoured).
* Reflection surface: hoisted bodies and dispatch tables appear as private
  synthetic methods on the host class, as Java lambda impl methods do.

Remaining gates before a default flip
-------------------------------------
1. The human-only IDE/tooling pass: breakpoints and step-into on hoisted bo=
dies,
   variables view, JDWP, coverage/decompilers. The scriptable layer is done=
 for
   both legs (automated tests assert per-line LineNumberTables and fully na=
med
   LocalVariableTables =E2=80=94 captures including written ones, typed and=
 implicit-it
   parameters, body locals =E2=80=94 plus ACC_SYNTHETIC on everything gener=
ated). Note
   this is the same gate groovy.target.lambda.hoist is waiting on, and that
   feature has zero semantic residuals =E2=80=94 so lambdas reasonably flip=
 first, and
   packing follows rather than jumps the queue.
2. One ecosystem due-diligence: frameworks that serialize closures as progr=
am
   state (the Jenkins CPS pattern) should be checked at flip time. Expected
   unaffected =E2=80=94 such frameworks transform closures at the AST level=
 before our
   codegen sees them, and would not enable the flag for those compilations =
=E2=80=94 but
   expected is not verified.
3. Migrating our own tests that assert the old bytecode shape (mechanical;
   currently flag-skipped).

My reading: the mechanism work is complete and verified; what remains is
tooling verification and policy. The conservative sequencing that fits our
conventions is: ship opt-in in 6.0 as proposed, revisit the default alongsi=
de
lambda.hoist once the IDE pass is done.

Opinions welcome, especially from anyone who can help with the IDE verifica=
tion
side.


On Wed, Jul 15, 2026 at 5:43=E2=80=AFAM Jochen Theodorou <[email protected]=
> wrote:
>
> On 7/14/26 14:50, Paul King wrote:
> > Hi Jochen,
> >
> > AI read below. It could be wrong. I am still pondering but won't get
> > time again until my tomorrow. So this is something to read in the
> > meantime.
>
> take your time. I recently feel like I do reviews and comments on PRs in
> 100% of my spare time ;)
> Some comments inline
>
> [...]
> > 1) The doCall-only contract has never held on the dynamic path.
> >
> > I tested a Closure subclass declaring only `String call(String)` (no do=
Call)
> > across releases:
> >
> >                         dynamic cl("foo")    GDK collect(cl)
> >    Groovy 3.0.21-25     works                MissingMethodException
> >    Groovy 4.0.32        works                MissingMethodException
> >    Groovy 5.0.6         works                MissingMethodException
> >    Groovy 6.0.0-alpha-1 works                MissingMethodException
> >    master (12164/65)    works                works
>
> This is an obvious outcome. call(Object) delegates to call(Object...),
> which looks for doCall. GDK does not see call(String), only
> call(Object), thus the failure. Which is exactly why overloading call
> methods in Closure was never really supported and only sometimes working
> if you override the call method instead. But if you override
> call(Object) and then do the call through call(Object...), it was still
> failing, since the dispatch direction was designed the other way around.
> The only choice left, if you want to avoid doCall, is then to override
> call(Object...) instead. That aligns with the other mail I think. The
> javadoc of Closure even mentions, that if you want to use the short form
> like cl("foo") you have to provide a doCall method.
> I will not deny that cl("foo") works, but I don`t think that was really
> intended. More like accidentally worked and then nobody changed it.
>
> [...] > That leaves a genuine decision rather than a bug report: either
> custom call
> > overloads are accepted as the (now consistent) de-facto contract, or
> > doCall-only is enforced on BOTH paths =E2=80=94 which is a behavioural =
break with
> > 3.x-6.x dynamic dispatch.
>
> Which is why I wanted to start a discussion in the first place.
>
> [...]
> > 2) On PackedClosure,
> [...]
> > - one shared MetaClass for all packed closures: conceded, and it is the=
 one
> >    semantic difference that cannot be engineered away =E2=80=94 per-lit=
eral metaclass
> >    is definitionally what packing removes. Class-level metaClass change=
s on
> >    PackedClosure would affect every packed closure. The delegate/
> >    resolveStrategy axis is already fenced by the runtime guard; this ax=
is
> >    needs documenting as a caveat of the opt-in feature (and is one more=
 reason
> >    packing must stay off for code that plays MOP games).
>
> the problem is that it will stay an opt-in feature, since it cannot
> replace the regular construct for the same case.
>
> > - "the call path seems to be the only way to invoke the real method": t=
rue,
> >    but I'd frame what that path now IS: a statically compiled, metaclas=
s-free,
> >    inlinable route (adapter -> per-arity dispatch table -> direct invoc=
ation of
> >    the typed hoisted body). For statically compiled callers, invokevirt=
ual
> >    call() on a packed closure reaches a typed body with no MOP involvem=
ent at
> >    all =E2=80=94 which is the closest thing we currently have to your p=
oint (6),
> >    since the typed doCall of a closure class is hidden behind Closure<T=
> for
> >    static compilation, but a packed closure's entry is structurally fix=
ed.
> >    Recent measurements: packed closures now run 1.8-4.7x FASTER than ge=
nerated
> >    closure classes on capturing shapes with byte-identical behaviour, l=
argely
> >    because of this property.
>
> With GeneratedClosure the class is part of the same package and module
> of the class it was declared in. So if a class of module A gets an
> instance of GeneratedClosure from module B, and I do a dynamic call from
> A to B using that Closure, then the call will be a call from A via
> doCall into B, requiring JPMS to allow it. Now in case of PackedClosure,
> as long as we not bypass the call method and use the target directly, we
> effectively make it a call from A into <Groovy Runtime> into B.
> Bypassing any possible restriction between A and B. Of course I am only
> talking about A and B as modules written in Groovy. And yes, this is
> currently a theoretical case.
>
> Now... the other part to think about is what happens if the MOP
> involvement is intended? Like if EMC is used. Do we say it does not work
> if PackedClosure is enabled? As I said, that probably disqualifies it as
> an incubation feature that will become a standard without flag. If, on
> the other hand, we move the call code into a special metaclass, we
> should still achieve similar results in performance while we ensure a
> potential MOP path stays more open. And we can enable potential direct
> invocation paths with invokedynamic. Plus, most likely, optimizing that
> path will also suggest similar optimizations for ClosureMetaClass. Then
> the main difference between the two would be that PackedClosure saves on
> class loading overhead.
>
> > - one point in the architecture's favour under your criterion (2): dyna=
mic
> >    call sites invoking closures are megamorphic across per-literal clos=
ure
> >    classes, but monomorphic on the single PackedClosure class =E2=80=94=
 the per-target
> >    selection moves into a switch the JIT handles well. Indy caching get=
s
> >    easier, not harder.
>
> which is why I would like to have this as standard feature, not as opt-in
>   [...]
>
>
> bye Jochen
>