Re: Outrigger memory leak?

Hi John:

        Thank you for the reply.  See answers interspersed. Oddly enough, I've
just begun having difficulty replicating the problem on Solaris

Cheers,

Greg.

On Wed, 2007-01-03 at 17:50, John McClain - Sun Microsystems, Inc.
wrote:
> Hmmm, seems odd - not sure why your test client would result in OoMEs. I
> assume your entry is just a Uuid and a byte[]?
>
My apologies for not including it before -
public class SampleEntry implements Serializable,Entry  {

    public Uuid uuid=null;

    public Object payload=null;


}

> Given your access patters I think it makes sense that change the
> reapingInternal or Priority didn't have any effect. The takes your are
> doing should have the save the same effect as the reaping thread.
>
I haven't dug deep enough in the source code for Outrigger yet; I
wondered if the reaping happened solely in the background, or if it
would be done when the take() is done.

> Random thought - Java will throw OoME for other conditions besides
> running out of memory, it could have memory but no block is big enough
> to satisfy the requested allocation, it could be out of threads (seems
> unlikely that would be the case here though). Don't suppose there is any
> sort of useful stack trace from the OoME?
>
See Exhibit A below.  I don't see anything significant in it, but you
might.

> What was the smallest payload size you tried? 250KB seems a bit large (I
> may just be showing my age - when I was your age all we had was 128
> bytes, and WE LIKED IT...)
Yeah, yeah, I had to save up for six months to put that second 4K into
the OSI Challenger 4P....You tell that to the kids today, they won't
believe you.

Anyway, the tests looked like this:

Payload(bytes)          Iterations to fail      Total
throughput(MB)=Payload*Iterations
=======================================================================================
1000000                           197                   197
 500000                           225                   112.5
 250000                          1342                   335
 100000                          3018                   301.8
  50000                          8410                   420.5
  25000                          8194                   204.8
  10000                         11336                   113.4
   5000                         22330                   111.7
   2500                         32976                    82.4
   1000                         53471                    53.5

All tests were run with -Xmx128m.

> I could imagine the GC is getting into some
> odd state, when you let it sit idle for a short time how long would you
> wait? Was the waiting between testSpace calls or the between the write
> and take?
>
My hypothesis was that reaping happened on a background thread and it
didn't get to run when the main I/O threads were ping-ponging back and
forth.  So I gave it 100mS between testSpace calls; hopefully one or two
time slices.  No effect.  Also, stopping the test partway through and
letting Outrigger sit for several minutes had no effect on the memory
usage (as seen by the Netbeans profiler).

> Greg Trasuk wrote:
> > Hi all:
> >
> >         I'm running up against out-of-memory errors using transient outrigger.
> > I got the errors in a production system that runs a fair bit of data
> > through the space, then wrote a test script to try and isolate the
> > problem (code attached below).  When running with max heap size at the
> > default (512M, I believe), the Outrigger instance runs out of heap after
> > about 2100 iterations with a payload size of 250e3 bytes.  With smaller
> > payloads, I get more iterations (although curiously not in any kind of
> > linear relationship.  hmmm...), but I still get the failure eventually.
> >
> >         Note that the test case verifies that the entry is gone after I 'take'
> > it.  I've also confirmed in other tests that it disappears on its own
> > when the lease time expires, so everything seems to be behaving as
> > expected.  Letting the space sit idle for a short time between accesses
> > also has no effect (I wondered if perhaps the reaping thread didn't get
> > a chance to run).  I've tried setting 'reapingInterval=5000;
> > reapingPriority=Thread.MAX_PRIORITY', which had no effect.
> >
> >         Am I missing something in the configuration?  The problem is
> > independent of whether Outrigger is run under Harvester or from the
> > Launch-All script in the starter kit.  I haven't found any other bug
> > reports, and I find it hard to believe I've found a bug in a mature
> > product.  Any suggestions?
> >
> > Thanks in advance,
> >
> > Greg.
> >
> > Environment details:
> > --------------------
> > JSK 2.1
> > JDK 1.5.0_09 Server VM on Solaris 10 (same issue on Windows XP running
> > JDK6)
> > Opteron 165 Processor (dual-core), 2GB physical memory
> >
> > Test Case (run in a different VM from Outrigger):
> > -------------------------------------------------
> > public class TestSpace extends TestCase {
> >
> >     private static final int PAYLOAD_SIZE=250000;
> >
> >     /** Creates a new instance of TestSpace */
> >     public TestSpace(String s) {
> >         super(s);
> >     }
> >
> >     private ApplicationContext applicationContext=null;
> >     private JavaSpace space=null;
> >
> >     public void setUp() {
> >         setApplicationContext(StaticApplicationContext.getContext());
> >     }
> >
> >     /** We should be able to create an entry, drop it in and retrieve
> > it. */
> >     public void testSpace() throws Exception {
> >         DynamicResolver resolver=new DynamicResolver(this);
> >         try {
> >
> >             resolver.resolve();
> >             assertTrue("No JavaSpace was found", getSpace()!= null);
> >
> >             SampleEntry entry = new SampleEntry();
> >             entry.uuid=UuidFactory.generate();
> >             byte[] payload=new byte[PAYLOAD_SIZE];
> >             entry.payload=payload;
> >
> >             getSpace().write(entry, null, 5000);
> >             entry.payload=null;
> >
> >             SampleEntry retrievedEntry=(SampleEntry)
> > getSpace().takeIfExists(entry, null, JavaSpace.NO_WAIT);
> >             assertTrue("Didn't get my entry back",
> > retrievedEntry!=null);
> >             assertTrue("Didn't get the payload back",
> > retrievedEntry.payload!=null);
> >             byte[] retrievedPayload=(byte[]) retrievedEntry.payload;
> >             assertTrue("Payload size wasn't right.",
> > retrievedPayload.length==payload.length);
> >             /* Once we have retrieved it, it should not be retrievable
> > again. */
> >             retrievedEntry=(SampleEntry) getSpace().takeIfExists(entry,
> > null, JavaSpace.NO_WAIT);
> >             assertTrue("Entry was still there after I took it",
> > retrievedEntry==null);
> >         } catch(RuntimeException ex) {
> >             resolver.fail();
> >             throw ex;
> >         } catch(Exception t) {
> >             resolver.fail();
> >             throw t;
> >         } finally {
> >             resolver.clear();
> >         }
> >     }
> >
> >     public void testSpaceALot() throws Exception {
> >         int i=0;
> >         try {
> >             for(i=0; i <5000; i++) {
> >                 testSpace();
> >             }
> >         } catch(Exception ex) {
> >             fail("Exception " + ex.getMessage() + " at " + i + "
> > iterations and payload size=" + PAYLOAD_SIZE+ ".");
> >         }
> >     }
> >
> >     public ApplicationContext getApplicationContext() {
> >         return applicationContext;
> >     }
> >
> >     public void setApplicationContext(ApplicationContext
> > applicationContext) {
> >         this.applicationContext = applicationContext;
> >     }
> >
> >     public JavaSpace getSpace() {
> >         return space;
> >     }
> >
> >     public void setSpace(JavaSpace space) {
> >         this.space = space;
> >     }
> > }
> > --
> > Greg Trasuk, President
> > StratusCom Manufacturing Systems Inc. - We use information technology to
> > solve business problems on your plant floor.
> > http://stratuscom.com
>

Exhibit A: Stack trace from Out of Memory Error:
================================================
    java.rmi.ServerError: Error in server thread; nested exception is:
        java.lang.OutOfMemoryError: Java heap space
        at
net.jini.jeri.BasicInvocationDispatcher.dispatch(BasicInvocationDispatcher.java:647)
        at
com.sun.jini.jeri.internal.runtime.ObjectTable$6.run(ObjectTable.java:597)
        at
net.jini.export.ServerContext.doWithServerContext(ServerContext.java:103)
        at
com.sun.jini.jeri.internal.runtime.ObjectTable$Target.dispatch0(ObjectTable.java:595)
        at
com.sun.jini.jeri.internal.runtime.ObjectTable$Target.access$700(ObjectTable.java:212)
        at
com.sun.jini.jeri.internal.runtime.ObjectTable$5.run(ObjectTable.java:568)
        at java.security.AccessController.doPrivileged(Native Method)
        at
com.sun.jini.jeri.internal.runtime.ObjectTable$Target.dispatch(ObjectTable.java:565)
        at
com.sun.jini.jeri.internal.runtime.ObjectTable$Target.dispatch(ObjectTable.java:540)
        at
com.sun.jini.jeri.internal.runtime.ObjectTable$RD.dispatch(ObjectTable.java:778)
        at
net.jini.jeri.connection.ServerConnectionManager$Dispatcher.dispatch(ServerConnectionManager.java:148)
        at
com.sun.jini.jeri.internal.mux.MuxServer$2.run(MuxServer.java:244)
        at java.security.AccessController.doPrivileged(Native Method)
        at
com.sun.jini.jeri.internal.mux.MuxServer$1.run(MuxServer.java:241)
        at
com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:136)
        at java.lang.Thread.run(Thread.java:595)
        at
com.sun.jini.jeri.internal.runtime.Util.__________EXCEPTION_RECEIVED_FROM_SERVER__________(Util.java:108)
        at
com.sun.jini.jeri.internal.runtime.Util.exceptionReceivedFromServer(Util.java:101)
        at
net.jini.jeri.BasicInvocationHandler.unmarshalThrow(BasicInvocationHandler.java:1303)
        at
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:832)
        at
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethod(BasicInvocationHandler.java:659)
        at
net.jini.jeri.BasicInvocationHandler.invoke(BasicInvocationHandler.java:528)
        at com.sun.jini.outrigger.$Proxy27.write(Unknown Source)
        at
com.sun.jini.outrigger.SpaceProxy2.write(SpaceProxy2.java:296)
        at
com.stratuscom.fairview.tests.TestSpace.testSpace(TestSpace.java:53)
        at
com.stratuscom.fairview.tests.TestSpace.testSpaceALot(TestSpace.java:79)
        at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at junit.framework.TestCase.runTest(TestCase.java:166)
        at junit.framework.TestCase.runBare(TestCase.java:140)
        at junit.framework.TestResult$1.protect(TestResult.java:106)
        at junit.framework.TestResult.runProtected(TestResult.java:124)
        at junit.framework.TestResult.run(TestResult.java:109)
        at junit.framework.TestCase.run(TestCase.java:131)
        at junit.framework.TestSuite.runTest(TestSuite.java:173)
        at junit.framework.TestSuite.run(TestSuite.java:168)
        at junit.swingui.TestRunner$17.run(TestRunner.java:644)
Caused by: java.lang.OutOfMemoryError: Java heap space
--
Greg Trasuk, President
StratusCom Manufacturing Systems Inc. - We use information technology to
solve business problems on your plant floor.
http://stratuscom.com

===========================================================================
To unsubscribe, send email to [email protected] and include in the body
of the message "signoff JAVASPACES-USERS".  For general help, send email to
[email protected] and include in the body of the message "help".

To view past JAVASPACES-USERS postings, please see:
http://archives.java.sun.com/archives/javaspaces-users.html