Re: Gradual Growth of Memory Use?

On 13 Mar 2025, at 15:10, Yuri Davidovsky (as work at disclosure dot ie) <[email protected]> wrote:
> 
> I am not convinced that GC is the issue, not entirely sure what it could be doing to slow things down so much, unless it constantly traverses every single cons trying to figure out if it is accessible or not, but I digress.

Neither am I.  But LW gives you detailed control over anything you are likely to need here.  I am sure Martin can advise better than I can, but you can control which generation is the blocking generation, what it takes to GC it (how much it has to grow), and how it should be GCd if needed (marking or copying).

You can then also control which generation allocation happens in some dynamic extent, which allows you to allocate directly into the blocking generation.

So for instance (generation 3 is the blocking generation in 64 bit LW):

> (room)
 > Generation 7 allocated 38021776
 > Generation 6 allocated 0
 > Generation 5 allocated 0
 > Generation 4 allocated 0
 > Generation 3 allocated 714688
 > Generation 2 allocated 8773224
 > Generation 1 allocated 1102080
 > Generation 0 allocated 2709224

Total allocated 51320992 (0x30F18A0), total size 182255616 (0xADD0000) 

nil

> (defvar *l* (system:apply-with-allocation-in-gen-num
               :cons
               3
               (lambda ()
                 (make-list 10000000))))
*l*

> (room)
 > Generation 7 allocated 38021776
 > Generation 6 allocated 0
 > Generation 5 allocated 0
 > Generation 4 allocated 0
 > Generation 3 allocated 170150848
 > Generation 2 allocated 855528
 > Generation 1 allocated 0
 > Generation 0 allocated 208728

Total allocated 209236880 (0xC78B390), total size 255836160 (0xF3FC000)

So you can see that this has pumped that big list directly into the blocking generation (it's almost certainly also caused several GCs of it as it's increased in size by 7 powers of 2, and doubling is the default threshold that triggers a GC.

You can do better than this.  Look at the generations: there are four generations above the blocking generation: nothing will ever be promoted into those generations except by things like clean-down, so a GC will never be triggered.

> (room)
 > Generation 7 allocated 38021776
 > Generation 6 allocated 0
 > Generation 5 allocated 0
 > Generation 4 allocated 0
 > Generation 3 allocated 714688
 > Generation 2 allocated 8773224
 > Generation 1 allocated 1102080
 > Generation 0 allocated 2077008

Total allocated 50688776 (0x3057308), total size 182255616 (0xADD0000) 

nil

> (defvar *l* (system:apply-with-allocation-in-gen-num
               :cons
               4
               #'make-list
               10000000))
*l*

> (room)
 > Generation 7 allocated 38021776
 > Generation 6 allocated 0
 > Generation 5 allocated 0
 > Generation 4 allocated 160000000
 > Generation 3 allocated 714752
 > Generation 2 allocated 8773224
 > Generation 1 allocated 1102080
 > Generation 0 allocated 4297072

Total allocated 212908904 (0xCB0BB68), total size 359612416 (0x156F4000) 

So now I've got lot of conses in generation 4: since nothing ever gets promoted into it, then it will never be GCd, unless I ask for that.

So now:

> (gc-generation t)
10898768

> (room)
 > Generation 7 allocated 38021776
 > Generation 6 allocated 0
 > Generation 5 allocated 0
 > Generation 4 allocated 160000000
 > Generation 3 allocated 8628104
 > Generation 2 allocated 1913648
 > Generation 1 allocated 357016
 > Generation 0 allocated 241968

Total allocated 209162512 (0xC779110), total size 240336896 (0xE534000) 

nil

> (setf *l* nil)
nil

> (gc-generation t)
10912304

> (room)
 > Generation 7 allocated 38021776
 > Generation 6 allocated 0
 > Generation 5 allocated 0
 > Generation 4 allocated 160000000
 > Generation 3 allocated 9686224
 > Generation 2 allocated 1181968
 > Generation 1 allocated 44112
 > Generation 0 allocated 338544

Total allocated 209272624 (0xC793F30), total size 240205824 (0xE514000) 

And now I decide I'd like to empty generation 4:

> (gc-generation 4)
10917152

> (room)
 > Generation 7 allocated 38021776
 > Generation 6 allocated 0
 > Generation 5 allocated 0
 > Generation 4 allocated 0
 > Generation 3 allocated 10007048
 > Generation 2 allocated 875192
 > Generation 1 allocated 34912
 > Generation 0 allocated 341120

Total allocated 49280048 (0x2EFF430), total size 79855616 (0x4C28000) 

One thing to remember is that the control of allocation using system:apply-with-allocation-in-gen-num is, I am pretty sure, based on the dynamic *extent* not scope, so any allocation from other processes will also end up in that generation.  Again, Martin will know better than I do, of course.

All of this is for 64-bit LW, 32-bit is different.

--tim



_______________________________________________
Lisp Hug - the mailing list for LispWorks users
[email protected]
http://www.lispworks.com/support/lisp-hug.html