Re: Gradual Growth of Memory Use?

So what could account for accumulating Actors? 

There exists SERIALIZER Actors which ensure that only one message at a time is delivered to an Actor subnetwork. These are used for activities, like writing to a file, that should not allow parallel concurrent executions of the actions. 

Once a Task (not a machine thread) completes its use of that protected Actor network, it is expected to SEND a message to the customer Actor (denoted in the original message) which has the side effect of allowing the next waiting message to be delivered to the protected Actor subnetwork.

A Task is any chain of messages and Actor behavior executions that are related to completing some action. Execution can occur across any number of machine threads. Once completed, a result is sent to the customer Actor. If that final SEND never happens, then a growing queue of messages from other Tasks are left hanging and never allowed delivery past the SERIALIZER. Messages may contain references to Actors. Hence Actors would be kept along with the growing queue of messages awaiting delivery.

But such errant behavior would be visible as the failure to make progress beyond the action for one Task. It is akin to a deadlock, but the Actor system remains alive for other uses that don’t involve the protected Actor subnetwork. I’m using Actors to coordinate data collection and display activities. My system would visibly grind to a halt if a SERIALIZER were blocking progress. I don’t see any blocked behavior. My graphs keep updating and showing new data results every 15 seconds.

There are only a few SERIALIZERS in the system. Examining them shows no more than 1 Actor enqueued in them. So this cannot account for the growing large number of Actors that survive GC.

There are just two SERIALIZER in the main data acquisition code, used for serialization of recorded data and for global system state changes. State shows visible evidence for its changes, in the graphs displayed on screen. Data recording works properly since I can review the data collection on another system from the recorded file.

Actor references can be kept in messages, closure bindings, and global lexical bindings. Messages are either kept in the main event queue, or the separate SERIALIZER queues. The main event queue shows no messages pending delivery, but grows and shrinks during regular activity. The main event queue is a LW Mailbox, which shows a current capacity of around 8k. Actor references can also be kept in closure bindings of other Actors, and in global lexical bindings. There are only a few dozen global lexical Actor bindings in the entire system.

Ephemeral anonymous continuation Actors probably dominate the message traffic, but those are each used just once and then the reference is dropped. I assume that they will have short lifetime, but perhaps could live beyond a GC 0 collection. Forking involves a higher level continuation Actor that serves as the customer to each parallel execution Task, so there can be multiple references to them. But once the continuation conditions are met, and all parallel activations complete, the references are dropped with the delivery of the result messages to the continuation Actor and then on to the ultimate customer. There are many frequent (FORK) parallel activations during data acquisition. If any of these parallel activities blocks, then data acquisition as a whole ceases. I do not observe such blockage.

Actors are CREATEd during runtime, but CREATE merely allocates the 1-slot structs and fills in their functional closure slot. No running record of created Actors is kept in the system. If an Actor is CREATEd and not referenced, or if the references are dropped later, then the Actor struct and closure should be GC’d.

So I suppose I could take Tim’s code and modify to tally what behaviors (closure functions) are kept in all the Actors being accumulated. That might reveal where they are coming from. These will most likely be sub-functions of some DEFUN in the system. Maybe the LW system will retain an identifier for the containing DEFUN?

- DM



> On Mar 19, 2025, at 06:50, David McClain <[email protected]> wrote:
> 
> I just did a (GC-GENERATION 3), about half-hour after the last dump. Here is the new dump: (Now shows 460,000 Actors, up from 342,000 !!??)
> 
> (#S(CLASS-COUNTER :NAME CONS
>                   :COUNT 8821834
>                   :TOTAL-SIZE 141149344
>                   :CUMULATIVE-SIZE 324811016)
>  #S(CLASS-COUNTER :NAME FUNCTION
>                   :COUNT 1456012
>                   :TOTAL-SIZE 71371256
>                   :CUMULATIVE-SIZE 183661672)
>  #S(CLASS-COUNTER :NAME SIMPLE-VECTOR
>                   :COUNT 2221920
>                   :TOTAL-SIZE 69226000
>                   :CUMULATIVE-SIZE 112290416)
>  #S(CLASS-COUNTER :NAME COM.RAL.ACTORS:ACTOR
>                   :COUNT 459690
>                   :TOTAL-SIZE 11032560
>                   :CUMULATIVE-SIZE 43064416)
>  #S(CLASS-COUNTER :NAME CLOS::SYSTEM-OBJECT
>                   :COUNT 127772
>                   :TOTAL-SIZE 7416080
>                   :CUMULATIVE-SIZE 32031856)
>  #S(CLASS-COUNTER :NAME VECTOR
>                   :COUNT 3110
>                   :TOTAL-SIZE 5701968
>                   :CUMULATIVE-SIZE 24615776)
>  #S(CLASS-COUNTER :NAME SIMPLE-STRING
>                   :COUNT 22319
>                   :TOTAL-SIZE 5269464
>                   :CUMULATIVE-SIZE 18913808)
>  #S(CLASS-COUNTER :NAME SYMBOL
>                   :COUNT 103353
>                   :TOTAL-SIZE 4960944
>                   :CUMULATIVE-SIZE 13644344)
>  #S(CLASS-COUNTER :NAME SIMPLE-BASE-STRING
>                   :COUNT 126192
>                   :TOTAL-SIZE 3817624
>                   :CUMULATIVE-SIZE 8683400)
>  #S(CLASS-COUNTER :NAME DOUBLE-FLOAT
>                   :COUNT 27586
>                   :TOTAL-SIZE 441376
>                   :CUMULATIVE-SIZE 4865776)
> 
> 
>> On Mar 19, 2025, at 06:43, David McClain (as dbm at refined-audiometrics dot com) <[email protected]> wrote:
>> 
>> 
>> 
>>> On Mar 19, 2025, at 06:02, Martin Simmons <[email protected]> wrote:
>>> 
>>> I suggest comparing those counters with another call to size-by-class/loop
>>> when the image has grown some more.  After that, you could do (gc-generation
>>> 3) and compare again to see if the objects were really live or just stuck in
>>> generation 3.
>> 
>> 
>> So, from your suggestion, it would make most sense to perform (gc-generation 3) before doing the memory dump from Tim? Allow the system to discard the accumulated garbage first? Then compare dumps from two separated times.
>> 
>> My experience running (gc-generation 3) in the past has shown that the memory image is indeed increasing. The results of the call do keep increasing, which would indicate memory actually in-use.
>