Re: OCaml Memory Management Best Practices

Am Montag, den 29.06.2015, 13:23 +0000 schrieb Kenneth Miller
[email protected] [ocaml_beginners]:
>   
> Replied inline.
> 
> 
> 
> 
> On Monday, June 29, 2015 8:55 AM, "Gerd Stolpmann
> [email protected] [ocaml_beginners]"
> <[email protected]> wrote:
> 
> 
> 
>   
> Am Montag, den 29.06.2015, 04:11 +0000 schrieb Kenneth Miller
> [email protected] [ocaml_beginners]:
> 
> > Garbage collection "leaks" can be one of several, right? I'm sure
> you
> > know, but I'll just describe a few here to make it clear what I'm
> > saying. A program that uses a GC could leak in that it doesn't
> > maintain it's data structures, so collection is untimely or
> > impossible. 
> 
> I understand this part. Is there a way for me to help the GC
> distinguish in your example below with x1? That would be the general
> jest of the email as per title.

Unfortunately, there is no good way to get directly the information from
the compiler whether a variable is alive or dead. You could look at the
assembler output from "ocamlopt -s", and check the mentioned stack
descriptors, but this is really hard to decode stuff.

What you can do is to define a finalizer for your value, and print
something when it gets collected, e.g.

let print_when_collected tag v =
  Printf.eprintf "Value is collected: %s\n%!" tag

and then just do

Gc.finalize (print_when_collected "x1") x1

That way you can at least exclude variables from your analysis.
(Culprit: do not define a finalizer function like print_when_collected
as local function because this would systematically prevent values from
being collected.)

Sometimes it is also helpful to force a garbage collection
(Gc.full_major) and check which variables are collected.


> =====
> Ah! So you are distinguishing the specifics of the problem to allow it
> to become decidable. Ok ok, I am convinced.

Well, we were talking about Java and OCaml.
 
> (This happens often btw, a category of problem is undecidable or too
> difficult to calculate but a very similar niche case is solvable and
> pragmatic.)

Agreed. Close neighbor problems can be totally different.

Gerd

> 
> > In either case, my program is currently eating up so much memory
> that
> > it crashes due to memory exhaustion. Let me explain my consumption
> > scenario. 
> > I have a file, it's kind of big, just 300 MB. I read that file and
> > translate that into a data structure that shouldn't really be all
> that
> > big; +- a some bytes.
> > I then iterate over the data structures that the file contains, and
> > translate them; the final result wanted, couldn't be but at maximum
> 4
> > to 8 times bigger. So, between 1.2 to 2.4 gigs additional. *Not* 20
> > gigs. I don't know what I did wrong though because I wrote all of my
> > code in a straight forward fashion; there's no undue complexity
> issue
> > that I know of, the task is inherently linear in time and space, as
> is
> > my code in my knowledge. So I really do care which specific
> definition
> > of leak you apply; I presume that I'm doing something wrong with
> > regards to how the GC would collect information and that there's
> > something that I *can* improve. Because what you seem to be saying
> > that the GC is perfect, it can always collect memory, therefore
> > there's nothing I can do. Well if there's nothing I can do, I will
> be
> > sad.
> 
> It is hard to say what goes wrong without knowing the code. It could
> be
> that you unintentionally keep pointers alive that should be dead. It
> could be that the chosen data structures are actually bigger than you
> expect.
> 
> Regarding the latter, one option is to use the objsize library to
> measure the bytes consumed by data.
> =====
> Thanks for the tip about objsize, I was looking for a way to learn how
> many actual bytes were taken up by my variables.
> 
> > Yeah, I was just thinking that List.map was heavy for me, because as
> I
> > transform from list 1 to list 2, I do not need each element of one.
> > One and two are each big, so I'd like to be able to free more
> > frequently, to reduce the load overall.
> 
> List.map is nothing more than
> 
> let rec map f = function
> [] -> []
> | a::l -> let r = f a in r :: map f l
> 
> and it is easy to have your own specially crafted version of it that
> omits elements not needed anymore.
> 
> Gerd
> 
> Thank you so so much! I thought initially that you possibly were being
> diminutive, but I can appreciate that you are indeed very helpful and
> kind :) 
> 
> > Thanks so much Gerd! :)
> > 
> > Gerd
> > 
> > > I had an assignment once where we used a tree-set data structure
> to
> > > manage indexes to another reference array for fast lookup. We had
> to
> > > do exactly this.
> > > 
> > > 
> > > 
> > > 
> > > On Sunday, June 28, 2015 9:17 AM, "Gerd Stolpmann
> > > [email protected] [ocaml_beginners]"
> > > <[email protected]> wrote:
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Am Samstag, den 27.06.2015, 17:06 +0000 schrieb Kenneth Miller
> > > [email protected] [ocaml_beginners]:
> > > > 
> > > > For operations like List.map, I would imagine that the what is
> > > > actually occurring, is a new list of equal size is being
> > constructed
> > > > where an operation on the old map is happening for every
> element.
> > I
> > > > would like to know what the real implications are for the
> garbage
> > > > collector, and how I can minimize the performance and memory
> > impact.
> > > 
> > > You cannot do anything, and you need not.
> > > 
> > > > Things like List.append calculate a new list by constructing a
> > copy.
> > > > That's incredibly inefficient to me, both in the sense that each
> > > > element afterward (time) needs to be copied, and in that more
> > memory
> > > > (space) is needed.
> > > > 
> > > > 
> > > > In addition, is there a set of advice points that can be given
> to
> > > make
> > > > sure that the garbage collector will appropriately see what it
> > needs
> > > > to in order to free items? In java, everything is reference
> > counted,
> > > 
> > > No, this is not true. Java, like OCaml, uses at least a
> > mark-and-sweep
> > > collector (of course, this depends on the JVM used, and there is
> > > normally even a generational collector). Reference-counting is a
> bad
> > > choice for any higher-level language because it cannot detect
> cyclic
> > > pointer chains, and these remain uncollected.
> > > 
> > > > so if you want an object to be freed and you're maintaining it
> in
> > a
> > > > tree the lack of use of that object is better treated by going
> to
> > > the
> > > > tree and updating the node that references it to mark it null.
> In
> > > this
> > > > way, the garbage collector can collect the item that you were
> > > pointing
> > > > to.
> > > 
> > > There is no need to do this (breaking cycles) in Java, and no need
> > to
> > > do
> > > this in OCaml.
> > > 
> > > > OCaml is a different animal though, and it doesn't use reference
> > > > counting. There doesn't seem to me to be any simple go-to
> practice
> > > in
> > > > order to ensure that the GC will in fact collect an item that I
> > know
> > > > at the moment because it's operation is so different.
> > > 
> > > The OCaml collector is fully automatic like the one in Java.
> Cycles
> > > are
> > > collected when the whole cycle becomes unreachable.
> > > 
> > > Gerd
> > > 
> > > [Non-text portions of this message have been removed]
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> 
> 
> 
> 
> 
> 
>