Re: How to benchmark memory managers?

On Jan 3, 2005, at 3:53 PM, Emery Berger wrote:

> If you'd like to see a way around these difficulties, I'd refer you to 
> a
> paper by my student Matthew Hertz and myself. We have developed a
> methodology that gets around these complications, and we have compared
> the performance of numerous garbage collection algorithms to a
> widely-used explicit memory allocator (the Lea allocator). The paper is
> in submission, but there's an earlier technical report version 
> available
> at http://www.cs.umass.edu/~emery/pubs/04-17.pdf.
>
> regards,
> -- emery
>

wow if true, this is going to come down like an anvil. from the 
abstract:

> We show that, at large heap sizes and under no memory pressure, the 
> runtime performance of some garbage collection algorithms is 
> competitive with the Lea memory allocator and occasionally outperforms 
> it by up to 4%. However, our results confirm that garbage collection 
> requires six times the physical memory to achieve this performance and 
> suffers order-of-magnitude performance penalties when paging occurs.


However:
> We use exact object reachability traces [27, 28] as an oracle to 
> indicate when objects should be deallocated. By implementing this 
> oracle inside a detailed architectural simulator, we can execute and 
> precisely measure unaltered Java applications as if they were written 
> using explicit memory management.

by using an oracle, you eliminate the cost of the explicit management 
that a programmer using explicit memory management would have had to 
implement in order to determine when objects become garbage. Thus any 
cost of explicit destruction or reference counting which may have to be 
done is not accounted for. So I think this methodology is biased since 
it completely eliminates the cost on one side that is being counted on 
the other.

-- james mccartney