Re: [Gc] Is there a way to walk the entire heap of live obj ects

Hi Cristian,
Do you use latest gc? (From any bdwgc repository)
I remember I had some out of memory issue resolved by fixing splitting in allocblk_nth. See commits ec5130a and 701f219 but they both relates to bdwgc compiled with USE MUNMAP.
Regards,
Ivan
--
Sat, 07 Jun 2014, 08:33 +04:00 from Bruce Hoult <[email protected]>:
You also seem to have two N7gctools17GCVector_moveableIN4core11CacheRecordEEE of 1.2 MB each.
So when you allocate one of those, the GC has to find 300 contiguous pages. That's a lot.
If those two are all you ever have, then no problem. If you're constantly allocating and GCing them then that's potentially a big big problem.
Unfortunately, in a non-moving memory manager there is no perfect algorithm to decide when or whether to split up or combine large memory blocks. That applies to both malloc and gc.
In my own code, I try not to allocate anything larger than a heap block. I try to avoid very large arrays (including strings), and when I do need them I use an implementation that uses lots of small blocks and an index.
That would probably be a big change to your code.
While you have a locally-modified version of the GC, may I suggest you go to GC_allochblk_nth() and add "may_split = FALSE;" as the first statement (overriding the parameter's passed value). Let me know if that makes any difference and if so we can try something a little more subtle :)
On Sat, Jun 7, 2014 at 3:54 PM, Christian Schafmeister  < [email protected] > wrote:
>Bruce,
>
>The C++ files are read by the Clang library - they are used to construct C++ abstract syntax trees that I search using the ASTMatcher facilities of the Clang library.    I can monitor clang/llvm allocations on OS X using the “heap” command line program because they are allocated using new/delete.  They don’t appear to leak memory.   
>
>I have some large arrays (largest is 393K, see below) but I don’t think that many - I could be wrong on that.  I’ll count how many are larger than 4K.
>
>After parsing/generating-AST/matching features in the AST for the same C++ file 10 times
>A summary of the memory usage according to the Boehm GC library routines...
>
>Done walk of memory  1093 ClassKinds   82 LispKinds   18 ContainerKinds   1 StringKinds
>TOTAL invalidHeaderTotalSize =      8682352
>TOTAL memory usage (bytes):        34446496
>TOTAL GC_get_heap_size()          352399360
>TOTAL GC_get_free_bytes()          35569664
>TOTAL GC_get_bytes_since_gc()     141379680
>TOTAL GC_get_total_bytes()     316783433851
>
>So the Boehm GC says it’s using 0.3GB (GC_get_heap_size()) but the OS X Activity monitor reports that the process is consuming 6.34GB of memory.
>
>If I can believe the Boehm GC number then I’m leaking memory somewhere else.
>
>
>An inventory of the largest reachable objects…
>
>-------------------- Reachable StringKinds -------------------
>strs: total_size:    5401416 count:    88618   avg.sz :       60  largest:     1024 N7gctools17GCString_moveableIcEE
>-------------------- Reachable ContainerKinds -------------------
>cont: total_size:    3569344 count:    47460   avg.sz :       75  largest:   393240 N7gctools17GCVector_moveableINS_9smart_ptrIN4core3T_OEEEEE
>cont: total_size:    2359344 count:        2   avg.sz :  1179672  largest:  1179672 N7gctools17GCVector_moveableIN4core11CacheRecordEEE
>cont: total_size:     177512 count:     5112   avg.sz :       34  largest:       88 N7gctools16GCArray_moveableINS_9smart_ptrIN4core3T_OEEELi0EEE
>cont: total_size:     160584 count:     1336   avg.sz :      120  largest:      288 N7gctools17GCVector_moveableIN4core16RequiredArgumentEEE
>cont: total_size:      54040 count:       73   avg.sz :      740  largest:     3296 N7gctools17GCVector_moveableINS_9smart_ptrIN4core8Symbol_OEEEEE
>cont: total_size:      32256 count:      112   avg.sz :      288  largest:      288 N7gctools17GCVector_moveableIN4core16OptionalArgumentEEE
>cont: total_size:      15736 count:      281   avg.sz :       56  largest:       56 N7gctools17GCVector_moveableINS_9smart_ptrIN4core6Cons_OEEEEE
>cont: total_size:      14784 count:       50   avg.sz :      295  largest:      672 N7gctools17GCVector_moveableIN4core15KeywordArgumentEEE
>cont: total_size:       8216 count:        1   avg.sz :     8216  largest:     8216 N7gctools17GCVector_moveableIN4core14DynamicBindingEEE
>cont: total_size:       3288 count:        1   avg.sz :     3288  largest:     3288 N7gctools17GCVector_moveableINS_9smart_ptrIN6clbind10ClassRep_OEEEEE
>cont: total_size:       2168 count:        1   avg.sz :     2168  largest:     2168 N7gctools17GCVector_moveableIN10asttooling12_GLOBAL__N_112RegistryMaps27SymbolMatcherDescriptorPairEEE
>cont: total_size:       1560 count:        1   avg.sz :     1560  largest:     1560 N7gctools17GCVector_moveableINS_9smart_ptrIN4core11Character_OEEEEE
>cont: total_size:       1560 count:        1   avg.sz :     1560  largest:     1560 N7gctools17GCVector_moveableINS_9smart_ptrIN4core5Str_OEEEEE
>cont: total_size:       1448 count:        1   avg.sz :     1448  largest:     1448 N7gctools17GCVector_moveableIN4core14ExceptionEntryEEE
>cont: total_size:        808 count:       13   avg.sz :       62  largest:      136 N7gctools17GCVector_moveableINS_9smart_ptrIN4core9Package_OEEEEE
>cont: total_size:        784 count:       14   avg.sz :       56  largest:       56 N7gctools17GCVector_moveableIPN10asttooling8internal17MatcherDescriptorEEE
>cont: total_size:        152 count:        1   avg.sz :      152  largest:      152 N7gctools17GCVector_moveableIN4core11AuxArgumentEEE
>cont: total_size:         56 count:        1   avg.sz :       56  largest:       56 N7gctools17GCVector_moveableINS_9smart_ptrIN4core16SourceFileInfo_OEEEEE
>-------------------- Reachable LispKinds -------------------
>lisp: total_size:   11301888 count:   353160  avg.sz :       32 N4core6Cons_OE
>lisp: total_size:    2454176 count:    76693  avg.sz :       32 N4core5Str_OE
>lisp: total_size:    1724448 count:    53889  avg.sz :       32 N4core8Bignum_OE
>lisp: total_size:    1607040 count:    25110  avg.sz :       64 N4core10Instance_OE
>lisp: total_size:    1295120 count:    32378  avg.sz :       40 N4core13BranchSNode_OE
>lisp: total_size:     852768 count:    26649  avg.sz :       32 N4core11LeafSNode_OE
>lisp: total_size:     657440 count:    16436  avg.sz :       40 N4core8Symbol_OE
>lisp: total_size:     446408 count:    11160  avg.sz :       40 N4core15VectorObjects_OE
>lisp: total_size:     418920 count:    10473  avg.sz :       40 N4core26VectorObjectsWithFillPtr_OE
>lisp: total_size:     268912 count:     4802  avg.sz :       56 N4core18CompiledFunction_OE
>lisp: total_size:     229760 count:     7180  avg.sz :       32 N4core14CompiledBody_OE
>lisp: total_size:     203680 count:     5092  avg.sz :       40 N4core12ValueFrame_OE
>lisp: total_size:     194112 count:     8088  avg.sz :       24 N4core8Fixnum_OE
>lisp: total_size:     148064 count:     4627  avg.sz :       32 N4core16StrWithFillPtr_OE
>lisp: total_size:     136064 count:     1546  avg.sz :       88 N4core19LambdaListHandler_OE
>lisp: total_size:      96376 count:     1721  avg.sz :       56 N4core9BuiltIn_OE
>lisp: total_size:      74272 count:     2320  avg.sz :       32 N4core15SourcePosInfo_OE
>
>
>On Jun 6, 2014, at 9:04 PM, Bruce Hoult < [email protected] > wrote:
>>How are you reading the c++ files? Are they, or anything else, GC objects larger than 4 KB?
>>
>>
>>
>>On Sat, Jun 7, 2014 at 11:07 AM, Christian Schafmeister  < [email protected] > wrote:
>>>Apologies if this is a duplicate - I ran into trouble with the size due to a graph I included.
>>>
>>>I used the code that Peter sent me as-is - it works very well - thanks!
>>>
>>>I’m writing a Common Lisp implementation that interoperates with C++ and uses LLVM as the back end.  I’ve written a static analyzer for my C++ code to help me clean it up and add features. When I run the static analyzer on the 165 C++ source files using the Boehm GC it blows up.
>>>
>>>At the end of processing 165 source files the process consumes about 30GB (gigabytes) of memory.
>>>
>>>So I run the static analyzer on one source file 10 times and use the code that Peter sent me and walk the reachable objects and add up their memory footprint.   There are a handful of objects that don’t have valid headers - I filter them out and currently don’t count them. I find that the total reachable memory used by the system remains pretty constant at 62MB to 34MB (it spikes and then goes down).
>>>
>>>But the total memory used by my process goes up and up and up very quickly!
>>>
>>>I ran the OS X program “heap” on the process and it reports “non-object” heap memory - which I assume is allocated by the Boehm GC (this assumption may be wrong - please correct me if you know better).
>>>
>>>Here is a graph that shows all of the memory usage each time after parsing/generating an AST/searching the AST for the same C++ file. A bit of explanation… The Y axis is log(Bytes)  The red points are what “heap” reports as “non-object” memory - I assume this is total Boehm memory. The green points are the total reachable Boehm memory that has a valid header that I built.   The blue points are the reachable Boehm memory that was allocated just since the start of loading the most recent C++ file. I measure this by writing an integer marker into the header of each newly allocated object and changing that marker each time I read the source file - this lets me track what objects are allocated since the last operation.
>>>
>>>Graph link:  http://imgur.com/tSN5jRL
>>>
>>>
>>>
>>>This can’t be memory fragmentation can it?
>>>Is Boehm not reusing memory properly?
>>>Am I not configuring Boehm properly?
>>>_______________________________________________
>>>bdwgc mailing list
>>>[email protected]
>>>https://lists.opendylan.org/mailman/listinfo/bdwgc
>>>
>>>--
>>>This message has been scanned for viruses and
>>>dangerous content by MailScanner, and is
>>>believed to be clean.
>>>
>>
>
>
>--
>This message has been scanned for viruses and
>dangerous content by MailScanner , and is
>believed to be clean.
_______________________________________________
bdwgc mailing list
[email protected]
https://lists.opendylan.org/mailman/listinfo/bdwgc

_______________________________________________
bdwgc mailing list
[email protected]
https://lists.opendylan.org/mailman/listinfo/bdwgc