Re: Gradual Growth of Memory Use?

One more illustration of the innate beauty of Lisp - 

Very simple to change from linked list to an array. Can probably do in less than 5 minutes, without changing the API. Will do after this morning’s data collection.



> On Mar 13, 2025, at 08:10, Yuri Davidovsky (as work at disclosure dot ie) <[email protected]> wrote:
> 
> 
>> On 13 Mar 2025, at 15:40, David McClain <[email protected]> wrote:
>> 
>> 
>> But what *is* a concern for me is when the existence of a growing list of telemetry causes the system to slow, probably as a result of GC issues. There is absolutely no need to have this telemetry data be subjected to GC, just like there would be no need for a serial disk file to be constant re-read and adjusted.
> 
> I am not convinced that GC is the issue, not entirely sure what it could be doing to slow things down so much, unless it constantly traverses every single cons trying to figure out if it is accessible or not, but I digress.
> 
>> But when the list grows to 1.5 GB then we are seeing system performance issues. Lots of compressed data laying around and lots of time scanning a list for memory that is never released anyway.
> 
> I think compressed memory should only cause issues when you are trying to reach for the old data, which will have to be recompressed on the fly and written back to the spare ram. As I understand it, OS keeps track of how often memory pages get accessed and compresses those that are not accessed often, or haven’t been accessed in a while.
> 
>> From your setup it appears that you only need the most recent data (hence the (push) usage), so it should not be compressed. But. It might be that linked data is stored all over the heap, in some frequently accessed pages, as well as less frequently, which could be making the mess of the OS trying to keep track of what has to be compressed, and what does not.
> 
> Disclaimer: all that I am saying here is only a guess, so treat is as such.
> 
>> CONS cells a cheap way to grow a linear list, using PUSH. But they take double the number of read accesses to reach the data, compared to a linear array. And they are small and can fit just about anywhere that some memory is available, whereas an array takes a glob of contiguous memory.
> 
> PS: As been said, see if you could use arrays for data storage, never mind the fact that you need to preallocate a lot of memory for those, we’ll worry about that later if it is confirmed to solve the issue. Ideally that would be typed arrays too, so we are minimising the random access via pointers.
> 
> 
> 
> _______________________________________________
> Lisp Hug - the mailing list for LispWorks users
> [email protected]
> http://www.lispworks.com/support/lisp-hug.html


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
[email protected]
http://www.lispworks.com/support/lisp-hug.html