Re: Gradual Growth of Memory Use?

The telemetry data is simply being PUSH’d into a list. No permutations. But that list becomes very long after 24 hours, and could cost a lot of time chasing through all the links that never get discarded anyway. 

So mark/sweep is always going to find that nothing was discarded and should be kept. IOW, a more suitable allocation would be to use static memory that GC can ignore, or some scratch file on external disk.

> On Mar 13, 2025, at 02:26, wojciech.pasieka (as wojciech dot pasieka at ai dot pressiton dot com) (Adrian W. Pasieka Ikane) <[email protected]> wrote:
> 
> 
> From: "David McClain (as dbm at refined-audiometrics dot com)" <[email protected]>
> 
> 'LWW seems quite speedy as long as the memory image in use remains below 1.5 GB. Something happens
> once you cross above that allocation. Perhaps just overly long CONS chains?'
> 
> 
> I just found this recently and haven’t tested it yet, it might also be relevant to the
> recent discussion about overall Lisp's performance:
> 
> 
> 'This code implements a novel algorithm for iterating over all permutations of a standard Lisp list
> (a singly linked list). No cons operations are performed during the iteration, and the original
> nodes are reused throughout the entire process.'
> 
> https://git.univ-pau.fr/tbaruchel/numerical-routines/-/snippets/5
> 
> 
> -- 
> Regards,
> 
> Adrian W. Pasieka
> 
> 
> 
> From: "David McClain \(as dbm at refined-audiometrics dot com\)" <[email protected]>
> Subject: Re: Gradual Growth of Memory Use?
> Date: March 12, 2025 at 18:40:29 MST
> To: Martin Simmons <[email protected]>
> Cc: [email protected]
> Reply-To: David McClain <[email protected]>
> 
> 
> Okay, having tracked down things… it is becoming clear that the memory growth is primarily coming from treating memory as a virtual file system, accumulating a growing list of telemetry data for some of the graphic displays.
> 
> Shutting down those accumulators cut the memory growth to about 50 MB/hr, or half what it was.
> 
> Further shutting down the dynamic plotter displays cuts down the growth even more. Some of these graphs are quite elaborate and take lots of memory to produce them. I can still record data for later review, even without any plotter displays. But I really don’t want to do that.
> 
> I need to find a better way to accumulate telemetry without causing grief to the Lisp memory manager. Either pre-allocate a maximal array, or use a real file system. Either way the memory use would be contained and of definite size.
> 
> The worst case seems to be caused by ever growing chains of CONS cells pointing to freshly allocated data structures with the telemetry data (via PUSH). Perhaps using a maximal sized array would avoid some of the memory problems? There is never any discarding of this data unless I hit the CLEAR DATA button. Generally, I just keep accumulating for the entire 24 hour period.
> 
> The thread system is quite busy, since there is a lot of parallel computation going on. Any time I decimate data, I have 2 incoming data channels undergoing the same decimation. Then the data becomes complex values stored in separate real and imaginary arrays after heterodyning to DC. So both of those components can be treated in parallel for decimation. Add to that are the various infrastructure tasks that get executed by concurrent Actors - things like file writing for data recording, running the GUI, orchestrating the 50% overlap processing, and finally performing precision frequency measurements on the arriving data.
> 
> LWW seems quite speedy as long as the memory image in use remains below 1.5 GB. Something happens once you cross above that allocation. Perhaps just overly long CONS chains?
> 
> 
> 
> 
>> On Mar 12, 2025, at 14:24, Martin Simmons <[email protected]> wrote:
>> 
>> Unless background threads are very busy, it would be surprising that the REPL
>> runs slowly.  You could see if (CLEAN-DOWN T) changes anything (but it isn't
>> recommended for general use because it moves every into generation 7 where it
>> won't be collected automatically).
>> 
>> Can you try running it for a day without calling the plotting code?  That
>> would eliminate more of the non-Lisp code (in the libraries used to implement
>> the CAPI).
>> 
>> -- 
>> Martin Simmons
>> LispWorks Ltd
>> http://www.lispworks.com/
>> 
>> 
>> 
>>>>>>> On Wed, 12 Mar 2025 10:32:58 -0700, David McClain said:
>>> 
>>> I have the C buffer ring arranged with 5 buffers for 4096 data samples each.
>>> 
>>> My thinking was that there would be one buffer currently in use by Win/11
>>> MMIO, one buffer queued up for next use by Win/11, one buffer in use by Lisp
>>> for data processing, and two extra buffers in a FIFO queue awaiting
>>> processing by Lisp. That gives me some accordion flexibility in case Lisp
>>> gets seriously delayed in some of the processing.
>>> 
>>> This is probably a lot more elaborate than actually needed. Telemetry
>>> indicates that my processing times range from 10 ms to 300 ms, and the
>>> buffer filling takes 500 ms. So there ought to be plenty of spare time if I
>>> just used the simplest possible buffering scheme.
>>> 
>>> But once dropouts begin to occur, they occur in clusters, spaced
>>> sporadically, but consistently thereafter. It really does look like my Lisp
>>> might be at fault here. I should start seeing processing throughput delays
>>> approaching 1s or worse. (In fact, I have seen that behavior a few weeks
>>> back.)
>>> 
>>> So something about the GC changes beyond a threshold near 1.5 GB allocation
>>> which begins to impose serious delays.
>>> 
>>>> On Mar 12, 2025, at 10:10, David McClain (as dbm at refined-audiometrics
>>>> dot com) <[email protected]> wrote:
>>>> 
>>>> Hi Martin,
>>>> 
>>>> Thanks for chipping in here…
>>>> 
>>>> I can run (GC-GENERATION T) from the REPL and see something very close to
>>>> what is being shown in the Performance Monitor. It seems to grow by about
>>>> 100 MB/hr. There might be an intentional growing list of data somewhere
>>>> down in all my code, which could account for this growth. I haven’t found
>>>> it yet, but still searching.
>>>> 
>>>> The plotting graphics are using CAPI and CG in LW, my own Plotter
>>>> package. There is a background image/logo shown in all the graphs that
>>>> comes from an external graphics image. That is the part that needs to be
>>>> allocated specifically in Lisp, then intentionally freed again. And that
>>>> is the elision that I tried earlier. Elision made no difference to the
>>>> growth of the working set.
>>>> 
>>>> I’m not particularly bothered by the 100 MB/hr growth just yet, (after
>>>> all, I have 32 GB of RAM), but something happens when we reach 1.5 GB
>>>> after 24 hours, which seems to slow down the system. I can do things in
>>>> the REPL and see a significant response delay at that time. Just stopping
>>>> and restarting the data collection does not cure the problems at that
>>>> point. I have to perform a cold start in a fresh session in LWW.
>>>> 
>>>> What is striking is that when the Lisp begins its delayed response, that
>>>> is when I see dropped data blocks from the ADC. Perhaps the problem is
>>>> entirely with my Lisp code, in that it delays returning a processed buffer
>>>> back to the C buffer-ring. It is seeming more likely this, as I watch the
>>>> behavior over the past month.
>>>> 
>>>> So, I should really try hard to track down any growing lists in my data
>>>> collection and try to elide them to see if it makes a difference. Puzzling
>>>> is that there seems a 1.5 GB threshold where the system enters an entirely
>>>> different behavior pattern.
>>>> 
>>>> - DM
>>>> 
>>>> 
>>>> 
>>>>> On Mar 12, 2025, at 09:06, Martin Simmons <[email protected]> wrote:
>>>>> 
>>>>> Yes, HCL:GC-GENERATION affects all running threads.
>>>>> 
>>>>> Is the gradual growth up to 1.5 GB reported in the output of (room) or
>>>>> the return values of (system:room-values)?  In particular the "total
>>>>> allocated" and "total size" values.
>>>>> 
>>>>> If so, then the output of (room t) periodically might be useful to see.
>>>>> 
>>>>> If not, then the problem is outside Lisp.
>>>>> 
>>>>> What is the plotting library using for the graphics?
>>>>> 
>>>>> -- 
>>>>> Martin Simmons LispWorks Ltd http://www.lispworks.com/
>>>>> 
>>>>> 
>>>>> 
>>>>>>>>>> On Wed, 12 Mar 2025 05:34:15 -0700, David McClain (as dbm at
>>>>>> refined-audiometrics dot com) said:
>>>>>> 
>>>>>> The system produces live data plots, and there are some graphics
>>>>>> elements that need to be allocated and freed for each graph redraw. I
>>>>>> have tried eliding those allocations during plotting to see of that was
>>>>>> the cause of memory leaks, and it appears not to be the case.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Mar 12, 2025, at 05:27, David McClain <[email protected]>
>>>>>>> wrote:
>>>>>>> 
>>>>>>> PS: The computer is an AMD Rizen 5 with 32 GB RAM. So no shortage of
>>>>>>> physical memory, and no reason to be paging anything. The CPU load is
>>>>>>> around 5-7% during data acquisition and recording to external disk
>>>>>>> memory. Nothing else is overtly running in the computer, but Windows/11
>>>>>>> may have other ideas about that…
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Mar 12, 2025, at 05:17, David McClain <[email protected]>
>>>>>>> wrote:
>>>>>>> 
>>>>>>> My daily data acquisition system now runs about 24 hours between
>>>>>>> failures. The symptoms are ever increasing memory use, starting at 130
>>>>>>> MB and growing to 1.5 GB, and then DAC-read failures where samples are
>>>>>>> erratically and consistently dropped, causing growing distortions of the
>>>>>>> measured items.
>>>>>>> 
>>>>>>> I’m doing (HCL:GC-GENERATION T) every time through the outer loop, about
>>>>>>> once per 15 sec. Memory buffers for DAC reading were initially
>>>>>>> established in Windows/11 and locked down by the system, yet during
>>>>>>> these later acquisitions the system acts like the locked buffers have
>>>>>>> been paged out, causing excess delay in arriving for use by the
>>>>>>> underlying MMIO system of Windows/11.
>>>>>>> 
>>>>>>> My normal mode of operation has been to acquire data each day in two
>>>>>>> groups. First is several hours surrounding Sunrise, and second, several
>>>>>>> hours surrounding Sunset. Those contain the most interesting events. The
>>>>>>> system normally goes haywire around 4 am, at which time I simply shut it
>>>>>>> down, exit the Lisp system, and then restart from cold.
>>>>>>> 
>>>>>>> My assumption is that HCL:GC-GENERATION affects all running threads in
>>>>>>> the system, and not just the one thread performing the GC. Maybe that is
>>>>>>> mistaken?
>>>>>>> 
>>>>>>> - DM
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________ Lisp Hug - the mailing
>>>>>> list for LispWorks users [email protected]
>>>>>> http://www.lispworks.com/support/lisp-hug.html
>>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________ Lisp Hug - the mailing
>>>> list for LispWorks users [email protected]
>>>> http://www.lispworks.com/support/lisp-hug.html
>>> 
>>> 
> 
> 
> _______________________________________________
> Lisp Hug - the mailing list for LispWorks users
> [email protected]
> http://www.lispworks.com/support/lisp-hug.html
> 
>