Re: Gradual Growth of Memory Use?

It is clear from the Win/11 Resource Monitor, that explorer.exe is the hog here. It is accumulating memory at more than twice the rate of LWW.

So it looks like resources are being requested from Win/11 and not actually being freed up.

As for why LWW is accumulating memory, I also don’t have a good answer. My internal data accumulation is tiny in comparison to the growth seen here. But I’m at the mercy of the default GC at the moment. I am not taking any measures to perform GC on my own. And I am not explicitly retaining any pointers to images.

- DM


> On Mar 15, 2025, at 11:32, David McClain <[email protected]> wrote:
> 
> here is pw-plot-image, merely a client of the gp:image object:
> 
> (defun pw-plot-image (pane xv yv img wd ht)
>   (let* ((xlog   (plotter-xlog pane))
>          (xlogfn (logfn xlog))
>          (ylog   (plotter-ylog pane))
>          (ylogfn (logfn ylog)))
>     (declare (fixnum wd ht))
>     
>     (flet ((x-value (x)
>              (funcall xlogfn x))
>            (y-value (y)
>              (funcall ylogfn y)))
>       
>       (let+ ((xform  (plotter-xform pane))
>              ( (lf bt rt tp)
>                (gp:transform-points xform
>                                     (list (x-value (elt xv 0)) (y-value (elt yv 0))
>                                           (x-value (elt xv 1)) (y-value (elt yv 1)))
>                                     ))
>              (plt-wd (1+ (- rt lf)))
>              (plt-ht (1+ (- bt tp))))
>         
>         (with-plotview-coords (pane)
>           (gp:with-graphics-state (pane
>                                    :mask (plotter-mask pane))
>             (gp:draw-image pane img
>                            lf tp
>                            :from-width  wd
>                            :from-height ht
>                            :to-width    plt-wd
>                            :to-height   plt-ht)
>             ))))
>     ))
> 
>> On Mar 15, 2025, at 11:25, David McClain (as dbm at refined-audiometrics dot com) <[email protected]> wrote:
>> 
>> I have now done one better…
>> 
>> It is clear that the memory is dominated by my waterfall display. And I don’t want to run the data collection blindly. I need to see the data as it comes in. From time to time, I take a screenshot to preserve interesting phenomena along with the raw data recording.
>> 
>> But instead of burdening the data analysis portion with the graphics display, I update vital information during the analysis portion and then spin off to some other thread to perform the graphics displays. If that thread is already busy with a graphics display, then it simply ignores the request. That can only happen when I’m doing non-decimated realtime processing. But most of the time, I have 15 seconds between display updates, and that is plenty. Meanwhile the data analysis can report back and free the data buffer for the next collections.
>> 
>> So now, looking at why the memory is accumulating… Every image and image-access is performed with an UNWIND-PROTECT to free the access and image when finished. That should be kosher. But memory is still accumulating.
>> 
>> ;; ---------------------------------------------
>> ;; Define some safe image access macros...
>> ;;
>> (defun do-with-image (port image fn)
>>   (unwind-protect
>>       (funcall fn image)
>>     (gp:free-image port image)))
>> 
>> (defmacro with-image ((port (image imgexpr)) &body body)
>>   ;; returned value will be that of the body
>>   `(do-with-image ,port ,imgexpr
>>                   (lambda (,image)
>>                     ,@body)))
>> 
>> 
>> (defun do-with-image-access (acc fn)
>>   (unwind-protect
>>       (funcall fn acc)
>>     (gp:free-image-access acc)))
>> 
>> (defmacro with-image-access ((acc access-expr) &body body)
>>   ;; returned value will be that of the body
>>   `(do-with-image-access ,access-expr
>>                          (lambda (,acc)
>>                            ,@body)))
>> 
>> 
>> And here’s the portion that makes use of these:
>> 
>> ;; -----------------------------------------------------------------------
>> ;; At redraw time, we need to construct a CAPI image and stuff it with
>> ;; the BGRA vector contents, then hand off this image to whatever
>> ;; drawing routines need to be called.
>> 
>> (defun do-convert-array-to-color-image-for-pane (bgra-vec wd ht pane continuation)
>>   (with-image (pane (img #-:WIN32 (gp:make-image pane wd ht)
>>                          #+:WIN32 (gp:make-image pane wd ht :alpha nil)
>>                          ))
>>     (with-image-access (acc (gp:make-image-access pane img))
>>       (gp:image-access-transfer-from-image acc)
>>       (gp:image-access-pixels-from-bgra acc bgra-vec)
>>       (gp:image-access-transfer-to-image acc))
>>     (funcall continuation img)
>>     ))
>> 
>> (defmacro with-array-converted-to-color-image-for-pane ((bgra-vec wd ht pane img)
>>                                                         &body body)
>>   `(do-convert-array-to-color-image-for-pane
>>           ,bgra-vec ,wd ,ht ,pane
>>           (lambda (,img)
>>             ,@body)))
>> 
>> and here is the calling client of these access routines: Argument ARR (2D) contains the z-values for the display. XV and YV are simply used to provide axis labels on the graph.
>> 
>> (defun do-plot-image (pane xv yv arr
>>                            &rest args
>>                            &key
>>                            clear
>>                            &allow-other-keys)
>>   "Internal workhorse for image plotting."
>>   (let+ ((pane     (plotter-pane-of pane))
>>          (fresh    (or clear
>>                        (display-list-empty-p pane)))
>>          (bgra-vec (apply #'make-bgra-image arr args)) ;; convert to 0-256 pixel values in each of 4 color channels
>>          ((ht wd)  (array-dimensions arr))
>>          (action   (lambda (pane x y width height)
>>                      (declare (ignore x y width height))
>>                      (when fresh
>>                        (apply 'pw-axes pane args))
>>                      (with-array-converted-to-color-image-for-pane (bgra-vec wd ht pane img)
>>                        (pw-plot-image pane xv yv img wd ht)))
>>                    ))
>>     (apply 'pw-init-xv-yv pane xv yv args)
>>     (augment-display-list pane action fresh)
>>     ))
>> 
>> Action happens in CAPI thread at display time.
>> 
>> 
>>> On Mar 15, 2025, at 06:41, Bradford Miller <[email protected]> wrote:
>>> 
>>> Have you considered using 2 computers, one to run your collection and analysis, and the other to do your waterfall display? At least then when you fill up memory, it would only be on the display computer and not impact your data tools.
>>> 
>>> Best,
>>> Brad Miller
>>> 
>>>> On Mar 15, 2025, at 8:41 AM, David McClain (as dbm at refined-audiometrics dot com) <[email protected]> wrote:
>>>> 
>>>> Max analysis duration now showing 36 ms, down from 450 ms. Most durations are 5 ms, versus 150 ms. So waterfall display has a huge impact on processor use.
>>>> 
>>>> 
>>>> 
>>>>> On Mar 15, 2025, at 05:39, David McClain <[email protected]> wrote:
>>>>> 
>>>>> I am running another data acquisition, this time with the waterfall image display disabled. Memory use by LWW still grows, but more slowly. Explorer memory use not significantly growing after only 15 minutes. I’ll check back in a few hours.
>>>>> 
>>>>>> On Mar 15, 2025, at 05:24, David McClain (as dbm at refined-audiometrics dot com) <[email protected]> wrote:
>>>>>> 
>>>>>> Okay… Finally have more than 24 hours of non-stop data acquisition.
>>>>>> 
>>>>>> Conditions:
>>>>>> Telemetry data saved to prealloc arrays - so no growth, just fill-in
>>>>>> Full running graphic displays
>>>>>> Stop after 33 hours of data acquisition - only one buffer-fill fault shown, around hour 27, max analysis time 450 ms
>>>>>> Memory use at start was 130 MB, growing to over 2 GB at finish.
>>>>>> 
>>>>>> I still cannot account for massive memory growth, but I suspect that it has to do with the running graphic display of a synthesized image (waterfall display).
>>>>>> 
>>>>>> The overall memory use grows from 4 GB to over 9 GB, with no compressed memory showing. But the “hog” in the memory is Explorer.exe (!), at over 7 GB. I am not overtly running any instances of Explorer.
>>>>>> 
>>>>>> When I reboot the system and begin anew, the monitor shows that Explorer.exe is launched just after I launch LWW. LWW PID = 7028, Explorer PID = 7048. At start, we have LWW memory at 130 MB, and Explorer at 110 MB. I did not invoke explorer.exe, but maybe the CAPI graphics system does this?
>>>>>> 
>>>>>> 
>>>>>>> On Mar 14, 2025, at 23:04, Dmitry Ivanov (as ystok-systema at mail dot ru) <[email protected]> wrote:
>>>>>>> 
>>>>>>> Hello folks,
>>>>>>> 
>>>>>>>> Have a look at this article:
>>>>>>>> https://bardimin.com/en/windows-en/tips-and-tricks/easy-ways-to-enable-and-disable-memory-compression-to-improve-windows-11-performance/
>>>>>>> 
>>>>>>> I have tested on Windows 10.
>>>>>>> 
>>>>>>> PS> Disable-MMAgent -mc
>>>>>>> 
>>>>>>> Reboot
>>>>>>> 
>>>>>>> PS> Get-MMAgent
>>>>>>> 
>>>>>>> ApplicationLaunchPrefetching : True
>>>>>>> ApplicationPreLaunch         : True
>>>>>>> MaxOperationAPIFiles         : 256
>>>>>>> MemoryCompression            : False
>>>>>>> OperationAPI                 : True
>>>>>>> PageCombining                : True
>>>>>>> PSComputerName               :
>>>>>>> 
>>>>>>> Nevetheless, I see in the Task managaer:
>>>>>>> compressed 200 MB
>>>>>>> available 1.5 GB
>>>>>>> 
>>>>>>> -- 
>>>>>>> Dmitry Ivanov
>>>>>>> lisp.ystok.ru
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> Lisp Hug - the mailing list for LispWorks users
>>>>>>> [email protected]
>>>>>>> http://www.lispworks.com/support/lisp-hug.html
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Lisp Hug - the mailing list for LispWorks users
>>>>>> [email protected]
>>>>>> http://www.lispworks.com/support/lisp-hug.html
>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Lisp Hug - the mailing list for LispWorks users
>>>> [email protected]
>>>> http://www.lispworks.com/support/lisp-hug.html
>>>> 
>>> 
>> 
>