Re: Gradual Growth of Memory Use?

Unless background threads are very busy, it would be surprising that the REPL
runs slowly.  You could see if (CLEAN-DOWN T) changes anything (but it isn't
recommended for general use because it moves every into generation 7 where it
won't be collected automatically).

Can you try running it for a day without calling the plotting code?  That
would eliminate more of the non-Lisp code (in the libraries used to implement
the CAPI).

-- 
Martin Simmons
LispWorks Ltd
http://www.lispworks.com/



>>>>> On Wed, 12 Mar 2025 10:32:58 -0700, David McClain said:
> 
> I have the C buffer ring arranged with 5 buffers for 4096 data samples each.
> 
> My thinking was that there would be one buffer currently in use by Win/11
> MMIO, one buffer queued up for next use by Win/11, one buffer in use by Lisp
> for data processing, and two extra buffers in a FIFO queue awaiting
> processing by Lisp. That gives me some accordion flexibility in case Lisp
> gets seriously delayed in some of the processing.
> 
> This is probably a lot more elaborate than actually needed. Telemetry
> indicates that my processing times range from 10 ms to 300 ms, and the
> buffer filling takes 500 ms. So there ought to be plenty of spare time if I
> just used the simplest possible buffering scheme.
> 
> But once dropouts begin to occur, they occur in clusters, spaced
> sporadically, but consistently thereafter. It really does look like my Lisp
> might be at fault here. I should start seeing processing throughput delays
> approaching 1s or worse. (In fact, I have seen that behavior a few weeks
> back.)
> 
> So something about the GC changes beyond a threshold near 1.5 GB allocation
> which begins to impose serious delays.
> 
> > On Mar 12, 2025, at 10:10, David McClain (as dbm at refined-audiometrics
> > dot com) <[email protected]> wrote:
> > 
> > Hi Martin,
> > 
> > Thanks for chipping in here…
> > 
> > I can run (GC-GENERATION T) from the REPL and see something very close to
> > what is being shown in the Performance Monitor. It seems to grow by about
> > 100 MB/hr. There might be an intentional growing list of data somewhere
> > down in all my code, which could account for this growth. I haven’t found
> > it yet, but still searching.
> > 
> > The plotting graphics are using CAPI and CG in LW, my own Plotter
> > package. There is a background image/logo shown in all the graphs that
> > comes from an external graphics image. That is the part that needs to be
> > allocated specifically in Lisp, then intentionally freed again. And that
> > is the elision that I tried earlier. Elision made no difference to the
> > growth of the working set.
> > 
> > I’m not particularly bothered by the 100 MB/hr growth just yet, (after
> > all, I have 32 GB of RAM), but something happens when we reach 1.5 GB
> > after 24 hours, which seems to slow down the system. I can do things in
> > the REPL and see a significant response delay at that time. Just stopping
> > and restarting the data collection does not cure the problems at that
> > point. I have to perform a cold start in a fresh session in LWW.
> > 
> > What is striking is that when the Lisp begins its delayed response, that
> > is when I see dropped data blocks from the ADC. Perhaps the problem is
> > entirely with my Lisp code, in that it delays returning a processed buffer
> > back to the C buffer-ring. It is seeming more likely this, as I watch the
> > behavior over the past month.
> > 
> > So, I should really try hard to track down any growing lists in my data
> > collection and try to elide them to see if it makes a difference. Puzzling
> > is that there seems a 1.5 GB threshold where the system enters an entirely
> > different behavior pattern.
> > 
> > - DM
> > 
> > 
> > 
> >> On Mar 12, 2025, at 09:06, Martin Simmons <[email protected]> wrote:
> >> 
> >> Yes, HCL:GC-GENERATION affects all running threads.
> >> 
> >> Is the gradual growth up to 1.5 GB reported in the output of (room) or
> >> the return values of (system:room-values)?  In particular the "total
> >> allocated" and "total size" values.
> >> 
> >> If so, then the output of (room t) periodically might be useful to see.
> >> 
> >> If not, then the problem is outside Lisp.
> >> 
> >> What is the plotting library using for the graphics?
> >> 
> >> -- 
> >> Martin Simmons LispWorks Ltd http://www.lispworks.com/
> >> 
> >> 
> >> 
> >>>>>>> On Wed, 12 Mar 2025 05:34:15 -0700, David McClain (as dbm at
> >>> refined-audiometrics dot com) said:
> >>> 
> >>> The system produces live data plots, and there are some graphics
> >>> elements that need to be allocated and freed for each graph redraw. I
> >>> have tried eliding those allocations during plotting to see of that was
> >>> the cause of memory leaks, and it appears not to be the case.
> >>> 
> >>> 
> >>> 
> >>>> On Mar 12, 2025, at 05:27, David McClain <[email protected]>
> >>>> wrote:
> >>>> 
> >>>> PS: The computer is an AMD Rizen 5 with 32 GB RAM. So no shortage of
> >>>> physical memory, and no reason to be paging anything. The CPU load is
> >>>> around 5-7% during data acquisition and recording to external disk
> >>>> memory. Nothing else is overtly running in the computer, but Windows/11
> >>>> may have other ideas about that…
> >>>> 
> >>>> 
> >>>> 
>>>>> On Mar 12, 2025, at 05:17, David McClain <[email protected]>
>>>>> wrote:
>>>>> 
>>>>> My daily data acquisition system now runs about 24 hours between
>>>>> failures. The symptoms are ever increasing memory use, starting at 130
>>>>> MB and growing to 1.5 GB, and then DAC-read failures where samples are
>>>>> erratically and consistently dropped, causing growing distortions of the
>>>>> measured items.
>>>>> 
>>>>> I’m doing (HCL:GC-GENERATION T) every time through the outer loop, about
>>>>> once per 15 sec. Memory buffers for DAC reading were initially
>>>>> established in Windows/11 and locked down by the system, yet during
>>>>> these later acquisitions the system acts like the locked buffers have
>>>>> been paged out, causing excess delay in arriving for use by the
>>>>> underlying MMIO system of Windows/11.
>>>>> 
>>>>> My normal mode of operation has been to acquire data each day in two
>>>>> groups. First is several hours surrounding Sunrise, and second, several
>>>>> hours surrounding Sunset. Those contain the most interesting events. The
>>>>> system normally goes haywire around 4 am, at which time I simply shut it
>>>>> down, exit the Lisp system, and then restart from cold.
>>>>> 
>>>>> My assumption is that HCL:GC-GENERATION affects all running threads in
>>>>> the system, and not just the one thread performing the GC. Maybe that is
>>>>> mistaken?
>>>>> 
>>>>> - DM
> >>>> 
> >>> 
> >>> 
> >>> _______________________________________________ Lisp Hug - the mailing
> >>> list for LispWorks users [email protected]
> >>> http://www.lispworks.com/support/lisp-hug.html
> >>> 
> > 
> > 
> > _______________________________________________ Lisp Hug - the mailing
> > list for LispWorks users [email protected]
> > http://www.lispworks.com/support/lisp-hug.html
> 
> 

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
[email protected]
http://www.lispworks.com/support/lisp-hug.html