Re: [Gc] Is there a way to walk the entire heap of live objects
Christian Schafmeister <[email protected]> Fri, 06 Jun 2014 23:54:31 -0400
| Newsgroups | gmane.comp.programming.garbage-collection.boehmgc |
|---|---|
| Message-ID | <[email protected]> |
Bruce, The C++ files are read by the Clang library - they are used to construct C++ abstract syntax trees that I search using the ASTMatcher facilities of the Clang library. I can monitor clang/llvm allocations on OS X using the “heap” command line program because they are allocated using new/delete. They don’t appear to leak memory. I have some large arrays (largest is 393K, see below) but I don’t think that many - I could be wrong on that. I’ll count how many are larger than 4K. After parsing/generating-AST/matching features in the AST for the same C++ file 10 times A summary of the memory usage according to the Boehm GC library routines... Done walk of memory 1093 ClassKinds 82 LispKinds 18 ContainerKinds 1 StringKinds TOTAL invalidHeaderTotalSize = 8682352 TOTAL memory usage (bytes): 34446496 TOTAL GC_get_heap_size() 352399360 TOTAL GC_get_free_bytes() 35569664 TOTAL GC_get_bytes_since_gc() 141379680 TOTAL GC_get_total_bytes() 316783433851 So the Boehm GC says it’s using 0.3GB (GC_get_heap_size()) but the OS X Activity monitor reports that the process is consuming 6.34GB of memory. If I can believe the Boehm GC number then I’m leaking memory somewhere else. An inventory of the largest reachable objects… -------------------- Reachable StringKinds ------------------- strs: total_size: 5401416 count: 88618 avg.sz: 60 largest: 1024 N7gctools17GCString_moveableIcEE -------------------- Reachable ContainerKinds ------------------- cont: total_size: 3569344 count: 47460 avg.sz: 75 largest: 393240 N7gctools17GCVector_moveableINS_9smart_ptrIN4core3T_OEEEEE cont: total_size: 2359344 count: 2 avg.sz: 1179672 largest: 1179672 N7gctools17GCVector_moveableIN4core11CacheRecordEEE cont: total_size: 177512 count: 5112 avg.sz: 34 largest: 88 N7gctools16GCArray_moveableINS_9smart_ptrIN4core3T_OEEELi0EEE cont: total_size: 160584 count: 1336 avg.sz: 120 largest: 288 N7gctools17GCVector_moveableIN4core16RequiredArgumentEEE cont: total_size: 54040 count: 73 avg.sz: 740 largest: 3296 N7gctools17GCVector_moveableINS_9smart_ptrIN4core8Symbol_OEEEEE cont: total_size: 32256 count: 112 avg.sz: 288 largest: 288 N7gctools17GCVector_moveableIN4core16OptionalArgumentEEE cont: total_size: 15736 count: 281 avg.sz: 56 largest: 56 N7gctools17GCVector_moveableINS_9smart_ptrIN4core6Cons_OEEEEE cont: total_size: 14784 count: 50 avg.sz: 295 largest: 672 N7gctools17GCVector_moveableIN4core15KeywordArgumentEEE cont: total_size: 8216 count: 1 avg.sz: 8216 largest: 8216 N7gctools17GCVector_moveableIN4core14DynamicBindingEEE cont: total_size: 3288 count: 1 avg.sz: 3288 largest: 3288 N7gctools17GCVector_moveableINS_9smart_ptrIN6clbind10ClassRep_OEEEEE cont: total_size: 2168 count: 1 avg.sz: 2168 largest: 2168 N7gctools17GCVector_moveableIN10asttooling12_GLOBAL__N_112RegistryMaps27SymbolMatcherDescriptorPairEEE cont: total_size: 1560 count: 1 avg.sz: 1560 largest: 1560 N7gctools17GCVector_moveableINS_9smart_ptrIN4core11Character_OEEEEE cont: total_size: 1560 count: 1 avg.sz: 1560 largest: 1560 N7gctools17GCVector_moveableINS_9smart_ptrIN4core5Str_OEEEEE cont: total_size: 1448 count: 1 avg.sz: 1448 largest: 1448 N7gctools17GCVector_moveableIN4core14ExceptionEntryEEE cont: total_size: 808 count: 13 avg.sz: 62 largest: 136 N7gctools17GCVector_moveableINS_9smart_ptrIN4core9Package_OEEEEE cont: total_size: 784 count: 14 avg.sz: 56 largest: 56 N7gctools17GCVector_moveableIPN10asttooling8internal17MatcherDescriptorEEE cont: total_size: 152 count: 1 avg.sz: 152 largest: 152 N7gctools17GCVector_moveableIN4core11AuxArgumentEEE cont: total_size: 56 count: 1 avg.sz: 56 largest: 56 N7gctools17GCVector_moveableINS_9smart_ptrIN4core16SourceFileInfo_OEEEEE -------------------- Reachable LispKinds ------------------- lisp: total_size: 11301888 count: 353160 avg.sz: 32 N4core6Cons_OE lisp: total_size: 2454176 count: 76693 avg.sz: 32 N4core5Str_OE lisp: total_size: 1724448 count: 53889 avg.sz: 32 N4core8Bignum_OE lisp: total_size: 1607040 count: 25110 avg.sz: 64 N4core10Instance_OE lisp: total_size: 1295120 count: 32378 avg.sz: 40 N4core13BranchSNode_OE lisp: total_size: 852768 count: 26649 avg.sz: 32 N4core11LeafSNode_OE lisp: total_size: 657440 count: 16436 avg.sz: 40 N4core8Symbol_OE lisp: total_size: 446408 count: 11160 avg.sz: 40 N4core15VectorObjects_OE lisp: total_size: 418920 count: 10473 avg.sz: 40 N4core26VectorObjectsWithFillPtr_OE lisp: total_size: 268912 count: 4802 avg.sz: 56 N4core18CompiledFunction_OE lisp: total_size: 229760 count: 7180 avg.sz: 32 N4core14CompiledBody_OE lisp: total_size: 203680 count: 5092 avg.sz: 40 N4core12ValueFrame_OE lisp: total_size: 194112 count: 8088 avg.sz: 24 N4core8Fixnum_OE lisp: total_size: 148064 count: 4627 avg.sz: 32 N4core16StrWithFillPtr_OE lisp: total_size: 136064 count: 1546 avg.sz: 88 N4core19LambdaListHandler_OE lisp: total_size: 96376 count: 1721 avg.sz: 56 N4core9BuiltIn_OE lisp: total_size: 74272 count: 2320 avg.sz: 32 N4core15SourcePosInfo_OE On Jun 6, 2014, at 9:04 PM, Bruce Hoult <[email protected]> wrote: > How are you reading the c++ files? Are they, or anything else, GC objects larger than 4 KB? > > > > On Sat, Jun 7, 2014 at 11:07 AM, Christian Schafmeister <[email protected]> wrote: > Apologies if this is a duplicate - I ran into trouble with the size due to a graph I included. > > I used the code that Peter sent me as-is - it works very well - thanks! > > I’m writing a Common Lisp implementation that interoperates with C++ and uses LLVM as the back end. I’ve written a static analyzer for my C++ code to help me clean it up and add features. When I run the static analyzer on the 165 C++ source files using the Boehm GC it blows up. > > At the end of processing 165 source files the process consumes about 30GB (gigabytes) of memory. > > So I run the static analyzer on one source file 10 times and use the code that Peter sent me and walk the reachable objects and add up their memory footprint. There are a handful of objects that don’t have valid headers - I filter them out and currently don’t count them. I find that the total reachable memory used by the system remains pretty constant at 62MB to 34MB (it spikes and then goes down). > > But the total memory used by my process goes up and up and up very quickly! > > I ran the OS X program “heap” on the process and it reports “non-object” heap memory - which I assume is allocated by the Boehm GC (this assumption may be wrong - please correct me if you know better). > > Here is a graph that shows all of the memory usage each time after parsing/generating an AST/searching the AST for the same C++ file. A bit of explanation… The Y axis is log(Bytes) The red points are what “heap” reports as “non-object” memory - I assume this is total Boehm memory. The green points are the total reachable Boehm memory that has a valid header that I built. The blue points are the reachable Boehm memory that was allocated just since the start of loading the most recent C++ file. I measure this by writing an integer marker into the header of each newly allocated object and changing that marker each time I read the source file - this lets me track what objects are allocated since the last operation. > > Graph link: http://imgur.com/tSN5jRL > > > > This can’t be memory fragmentation can it? > Is Boehm not reusing memory properly? > Am I not configuring Boehm properly? > _______________________________________________ > bdwgc mailing list > [email protected] > https://lists.opendylan.org/mailman/listinfo/bdwgc > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > _______________________________________________ bdwgc mailing list [email protected] https://lists.opendylan.org/mailman/listinfo/bdwgc