Re: Gradual Growth of Memory Use?
"David McClain (as dbm at refined-audiometrics dot com)" <[email protected]>
| Newsgroups | gmane.lisp.lispworks.general |
|---|---|
| Message-ID | <[email protected]> |
OH !!! I will do. > On Mar 13, 2025, at 11:26, Martin Simmons <[email protected]> wrote: > > Have you tried removing your calls to (HCL:GC-GENERATION T) from the outer > loop and just allowing the automatic GC to do its thing? Every call to that > will move all 1.5 GB of data, which is not sensible. > > -- > Martin Simmons > LispWorks Ltd > http://www.lispworks.com/ > > > >>>>>> On Thu, 13 Mar 2025 07:40:43 -0700, David McClain (as dbm at refined-audiometrics dot com) said: >> >> In my case, the telemetry data is simply a record of past events. Performance is not an issue for me with regard to that data. It is simply displayed in a running graph every 15s or so. The high performance data is allocated to pre-allocated buffers and dynamic arrays in the moment, then immediately discarded. The telemetry is just a few numbers from each analysis, along with a timestamp. >> >> But what *is* a concern for me is when the existence of a growing list of telemetry causes the system to slow, probably as a result of GC issues. There is absolutely no need to have this telemetry data be subjected to GC, just like there would be no need for a serial disk file to be constant re-read and adjusted. >> >> CONS cells a cheap way to grow a linear list, using PUSH. But they take double the number of read accesses to reach the data, compared to a linear array. And they are small and can fit just about anywhere that some memory is available, whereas an array takes a glob of contiguous memory. So PUSH is probably suitable for a few thousand entries. But when the list grows to 1.5 GB then we are seeing system performance issues. Lots of compressed data laying around and lots of time scanning a list for memory that is never released anyway. >> >>> On Mar 13, 2025, at 06:21, Yuri Davidovsky (as work at disclosure dot ie) <[email protected]> wrote: >>> >>> >>> >>>> On 13 Mar 2025, at 14:05, Anthony REY <[email protected]> wrote: >>>> >>>> Not a Lisp expert at all : but I was wondering if accessing data in a cons cell chain is similar to accessing data in a LinkedList, which is O(N) ? >>> >>> Yes, cons chains dbm is talking about are the linked lists you mentioned. Essentially each element in it is a pointer to a heap location and a pointer to the next cons element. You are essentially making two memory lookups for every single element traversed which can be bad news for performance if you have to do that continuously with huge lists. >>> >>>> Yuri's answers made me wonder if you might be running into heap fragmentation and then constantly cache miss when accessing the data pointed by your cells ? >>> >>> >>> Yes, the lack of spacial locality is a big problem for cache efficiency and linked lists are probably the worst offenders when it comes to that, although I would imagine there may be ways to mitigate it somewhat, for example by using continuous arrays of pointers behind the scenes which will save the time on looking up the next cell. >>> >>> >>>> I don´t know enough about Garbage Collection or LispWorks to give more than an hazardous hypothesis, but I do work in the video games industry and we developed some strategies and data structures to make sure that the memory is staying contiguous in memory to please the cache as much as possible and ensure we can reach our real time performance goals >>> >>> Game developers are quite known for being the top of the industry when it comes to memory management, I am not surprised that you are surprised to see how things are done in other areas. >>> >>> >>> >>> >>> _______________________________________________ >>> Lisp Hug - the mailing list for LispWorks users >>> [email protected] >>> http://www.lispworks.com/support/lisp-hug.html >> >> >> _______________________________________________ >> Lisp Hug - the mailing list for LispWorks users >> [email protected] >> http://www.lispworks.com/support/lisp-hug.html >> _______________________________________________ Lisp Hug - the mailing list for LispWorks users [email protected] http://www.lispworks.com/support/lisp-hug.html