Re: Watch out for memory leaks with EhCache

On Wed, Dec 11, 2019 at 3:13 PM John Huss <[email protected]> wrote:

>
>
> On Wed, Dec 11, 2019 at 1:59 PM John Huss <[email protected]> wrote:
>
>>
>>
>> On Wed, Dec 11, 2019 at 10:59 AM Andrus Adamchik <[email protected]>
>> wrote:
>>
>>>
>>>
>>> > On Dec 10, 2019, at 9:39 PM, John Huss <[email protected]> wrote:
>>> >
>>> >> My projects are set up to use soft references using "
>>> >> cayenne.server.object_retain_strategy=soft". I wouldn't expect this to
>>> >> behave differently than weak under memory pressure, but my unit test
>>> with
>>> >> soft is not releasing this memory. I'll try running the test with
>>> "weak"
>>> >> and see if that changes it. And I'll look at your project (thanks for
>>> >> that).
>>> >>
>>> >
>>> > To clarify for future readers, we're talking about objects being
>>> retained
>>> > by this reference path:
>>> > QueryCache -> cached_object -> ObjectContext ->
>>> > other_objects_that_weren't_cached
>>> >
>>> > Sorry, I was wrong about the cause here. The persistent objects ARE
>>> > released by the context's objectStore (whether using weak or soft).
>>> But the
>>> > context is still retaining a lot of extra memory. I'm having a hard
>>> time
>>> > determining the specific cause. It might be the dataRowCache?
>>>
>>> If you have "Use Shared Cache" unchecked in the modeler, then you get a
>>> single dataRowCache per context, so that would definitely explain it. I am
>>> using the default - one dataRowCache per stack, shared by all contexts, so
>>> that's never a problem.
>>>
>>
>> I've narrowed it down - the "extra" memory being retained by the
>> ObjectContext is: entries in ObjectStore.objectMap - but not DataObjects
>> themselves (those get cleared), it's just the references to those objects:
>> the mapping from ObjectId -> WeakReference. These entries stay present
>> after the WeakReference is cleared. All those ObjectIds (though small) add
>> up to a significant amount of memory over time. It looks like these are
>> supposed to be cleared ReferenceMap.checkReferenceQueue(), which calls
>> ReferenceQueue.poll() to find the cleared WeakReferences and remove those
>> entries from the ObjectStore's objectMap. However, poll() doesn't ever seem
>> to return any results (a cleared WeakReference). If I add in a call to
>> ReferenceQueue.remove(5) manually (which will block for five milliseconds
>> while it finds cleared references), it does return them and clear that
>> memory. I need to read more about ReferenceQueue, but the current
>> implementation of checkReferenceQueue() does not appear to be working since
>> poll() never returns anything.
>>
>
> This is actually a timing problem, not a problem with poll().
> checkReferenceQueue() would need to be called *after* the context is done
> being used *and* the gc has occurred. As it currently is
> checkReferenceQueue() is called while the context is being used, which can
> help clear *some* memory that can be freed while in use, but doesn't help
> after the context is unused (like when it is just sitting being referenced
> by an object in the QueryCache.
>

This can be addressed by brute force by checking all the ObjectContexts
that are being retained by the QueryCache and touching the
ObjectStore.objectMap to make it garbage collect.

ExecutorService pool = Executors.*newFixedThreadPool*(4);


*for* (String cacheName : cacheManager.getCacheNames()) {

Cache cache = cacheManager.getCache(cacheName);

*for* (Object key : cache.getKeys()) {

Element element = cache.get(key);

List<Persistent> list = (List<Persistent>) element.getObjectValue();

*if* (!list.isEmpty() && list.get(0).getObjectContext() *instanceof*
DataContext) {

ObjectStore objectStore = ((DataContext)list
.get(0).getObjectContext()).getObjectStore();


pool.submit(() -> {

objectStore.registeredObjectsCount(); // this will trigger
ObjectStore.objectMap.checkReferenceQueue()

});

}

}

}


pool.shutdown();

...

But this is a poor solution really (specific to each QueryCache
implementation, wasteful of resources). The better choice by far would be
to not retain the original ObjectContext at all - either by replacing it
with a fresh context, or by nulling it out. What problems are there in
doing that?


>
>>
>>>
>>> Andrus
>>>
>>>