Problems with helma Swarm and jgroups

Franz Philipp Moser <[email protected]> Thu, 10 Jan 2008 15:35:10 +0100

Newsgroups	gmane.comp.java.helma.general
Message-ID	<[email protected]>

Hi list,

I experinced latetly some outOfMemory Exceptions when using current 
helmaswarm. With a profiler I was able to find 35824 org.jgroups.Message 
Objects that consume a total of 300.000.000 Bytes of Memory (see 
screenshot, retained size). This only happens on the app server that is 
NOT the Swarm Master.

I don't know why this happens but it looks like if the "notMaster" 
(acceptor) doesn't purge the no longer needed Message objects.

"Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8009]" 
java.lang.OutOfMemoryError: GC overhead limit exceeded

I have appended a log file with the outOfMemory errors.

Can anybody help?

cu Philipp

_______________________________________________
Helma-user mailing list
[email protected]
http://helma.org/mailman/listinfo/helma-user

helma-out.log (text/plain, 15.3 KB)

Exception in thread "SwarmSessionMgr-twoday_net" java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
        at java.util.HashMap$KeyIterator.next(HashMap.java:828)
        at java.util.AbstractCollection.toArray(AbstractCollection.java:124)
        at helma.swarm.SwarmSessionManager.broadcastIds(SwarmSessionManager.java:250)
        at helma.swarm.SwarmSessionManager.run(SwarmSessionManager.java:130)
        at java.lang.Thread.run(Thread.java:619)
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid1694.hprof ...
Exception in thread "AJP13Listener1-187" Heap dump file created [514915279 bytes in 13.606 secs]
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at org.mortbay.util.LineInput$LineBuffer.<init>(LineInput.java:702)
        at org.mortbay.util.LineInput.<init>(LineInput.java:127)
        at org.mortbay.http.HttpInputStream.<init>(HttpInputStream.java:74)
        at org.mortbay.http.HttpConnection.<init>(HttpConnection.java:120)
        at org.mortbay.http.ajp.AJP13Connection.<init>(AJP13Connection.java:62)
        at org.mortbay.http.ajp.AJP13Listener.createConnection(AJP13Listener.java:230)
        at org.mortbay.http.ajp.AJP13Listener.handleConnection(AJP13Listener.java:209)
        at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
        at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
Exception in thread "AJP13Listener1-190" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at org.mortbay.util.LineInput$LineBuffer.<init>(LineInput.java:702)
        at org.mortbay.util.LineInput.<init>(LineInput.java:127)
        at org.mortbay.http.HttpInputStream.<init>(HttpInputStream.java:74)
        at org.mortbay.http.HttpConnection.<init>(HttpConnection.java:120)
        at org.mortbay.http.ajp.AJP13Connection.<init>(AJP13Connection.java:62)
        at org.mortbay.http.ajp.AJP13Listener.createConnection(AJP13Listener.java:230)
        at org.mortbay.http.ajp.AJP13Listener.handleConnection(AJP13Listener.java:209)
        at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
        at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
Exception in thread "AJP13Listener1-181" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at org.mortbay.util.ByteArrayPool.getByteArray(ByteArrayPool.java:59)
Exception in thread "AJP13Listener1-185" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at org.mortbay.util.ByteArrayPool.getByteArray(ByteArrayPool.java:59)
        at org.mortbay.http.ajp.AJP13Packet.<init>(AJP13Packet.java:91)
        at org.mortbay.http.ajp.AJP13RequestPacket.<init>(AJP13RequestPacket.java:56)
        at org.mortbay.http.ajp.AJP13InputStream.<init>(AJP13InputStream.java:37)
        at org.mortbay.http.ajp.AJP13Connection.<init>(AJP13Connection.java:62)
        at org.mortbay.http.ajp.AJP13Listener.createConnection(AJP13Listener.java:230)
        at org.mortbay.http.ajp.AJP13Listener.handleConnection(AJP13Listener.java:209)
Exception in thread "Thread-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
Error in XML-RPC listener loop (java.lang.OutOfMemoryError: GC overhead limit exceeded).
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Thread-1" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "AJP13Listener1-203" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "AJP13Listener1-75" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: Java heap space
Exception in thread "AJP13Listener1-221" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "AJP13Listener1-222" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "AJP13Listener1-92" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "AJP13Listener1-223" java.lang.OutOfMemoryError: GC overhead limit exceeded
Error in XML-RPC listener loop (java.lang.OutOfMemoryError: GC overhead limit exceeded).
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "AJP13Listener1-227" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Timer-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8009]" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "AJP13Listener1-121" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: Java heap space
helma.framework.TimeoutException: Request timed out
        at helma.objectmodel.db.Node.checkWriteLock(Node.java:272)
        at helma.objectmodel.db.Node.setString(Node.java:1963)
        at helma.objectmodel.db.Node.getGroupbySubnode(Node.java:1221)
        at helma.objectmodel.db.NodeManager.getNodeByRelation(NodeManager.java:1591)
        at helma.objectmodel.db.NodeManager.getNode(NodeManager.java:278)
        at helma.objectmodel.db.NodeManager.getNode(NodeManager.java:184)
        at helma.objectmodel.db.WrappedNodeManager.getNode(WrappedNodeManager.java:60)
        at helma.objectmodel.db.NodeHandle.getNode(NodeHandle.java:75)
        at helma.scripting.rhino.HopObject.checkNode(HopObject.java:175)
        at helma.scripting.rhino.HopObject.getFromNode(HopObject.java:826)
        at helma.scripting.rhino.HopObject.get(HopObject.java:814)
        at org.mozilla.javascript.ScriptableObject.getProperty(ScriptableObject.java:1544)
        at org.mozilla.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2032)
        at org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:117)
        at org.mozilla.javascript.gen.c144._c3(/home/twoday_net/helma/apps/twoday_net/code/Site/renderFunctions.js:97)
        at org.mozilla.javascript.gen.c144.call(/home/twoday_net/helma/apps/twoday_net/code/Site/renderFunctions.js)
        at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:66)
        at org.mozilla.javascript.gen.c140._c1(/home/twoday_net/helma/apps/twoday_net/code/Site/actions.js:15)
        at org.mozilla.javascript.gen.c140.call(/home/twoday_net/helma/apps/twoday_net/code/Site/actions.js)
        at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
        at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2834)
        at org.mozilla.javascript.gen.c140.call(/home/twoday_net/helma/apps/twoday_net/code/Site/actions.js)
        at org.mozilla.javascript.Context.call(Context.java:538)
        at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:296)
        at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:390)
        at java.lang.Thread.run(Thread.java:619)
helma.framework.TimeoutException: Request timed out
        at helma.objectmodel.db.Node.checkWriteLock(Node.java:272)
        at helma.objectmodel.db.Node.setString(Node.java:1963)
        at helma.objectmodel.db.Node.getGroupbySubnode(Node.java:1221)
        at helma.objectmodel.db.NodeManager.getNodeByRelation(NodeManager.java:1591)
        at helma.objectmodel.db.NodeManager.getNode(NodeManager.java:278)
        at helma.objectmodel.db.NodeManager.getNode(NodeManager.java:184)
        at helma.objectmodel.db.WrappedNodeManager.getNode(WrappedNodeManager.java:60)
        at helma.objectmodel.db.NodeHandle.getNode(NodeHandle.java:75)
        at helma.scripting.rhino.HopObject.checkNode(HopObject.java:175)
        at helma.scripting.rhino.HopObject.getFromNode(HopObject.java:826)
        at helma.scripting.rhino.HopObject.get(HopObject.java:814)
        at org.mozilla.javascript.ScriptableObject.getProperty(ScriptableObject.java:1544)
        at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:265)
        at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:868)
        at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:670)
        at helma.framework.core.Skin$Macro.render(Skin.java:752)
        at helma.framework.core.Skin.render(Skin.java:276)
        at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:223)
        at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:155)
        at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:474)
        at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:76)
        at org.mozilla.javascript.gen.c95._c10(/home/twoday_net/helma/apps/twoday_net/code/HopObject/macros.js:202)
        at org.mozilla.javascript.gen.c95.call(/home/twoday_net/helma/apps/twoday_net/code/HopObject/macros.js)
        at org.mozilla.javascript.Context.call(Context.java:531)
        at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:296)
        at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:868)
        at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:670)
        at helma.framework.core.Skin$Macro.render(Skin.java:752)
        at helma.framework.core.Skin.render(Skin.java:276)
        at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:223)
        at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:155)
        at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:474)
        at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:76)
        at org.mozilla.javascript.gen.c95._c10(/home/twoday_net/helma/apps/twoday_net/code/HopObject/macros.js:202)
        at org.mozilla.javascript.gen.c95.call(/home/twoday_net/helma/apps/twoday_net/code/HopObject/macros.js)
        at org.mozilla.javascript.Context.call(Context.java:531)
        at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:296)
        at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:868)
        at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:670)
        at helma.framework.core.Skin$Macro.render(Skin.java:752)
        at helma.framework.core.Skin.render(Skin.java:276)
        at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:223)
        at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:155)
        at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:474)
        at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:76)
        at org.mozilla.javascript.gen.c144._c4(/home/twoday_net/helma/apps/twoday_net/code/Site/renderFunctions.js:130)
        at org.mozilla.javascript.gen.c144.call(/home/twoday_net/helma/apps/twoday_net/code/Site/renderFunctions.js)
        at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2234)
        at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:257)
        at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:127)
        at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:76)
        at org.mozilla.javascript.gen.c21._c3(/home/twoday_net/helma/apps/twoday_net/code/Global/Helma.Aspects.js:37)
        at org.mozilla.javascript.gen.c21.call(/home/twoday_net/helma/apps/twoday_net/code/Global/Helma.Aspects.js)
        at org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:119)
        at org.mozilla.javascript.gen.c207._c1(/home/twoday_net/helma/apps/twoday_net/code/Day/actions.js:24)
        at org.mozilla.javascript.gen.c207.call(/home/twoday_net/helma/apps/twoday_net/code/Day/actions.js)
        at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
        at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2834)
        at org.mozilla.javascript.gen.c207.call(/home/twoday_net/helma/apps/twoday_net/code/Day/actions.js)
        at org.mozilla.javascript.Context.call(Context.java:538)
        at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:296)
        at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:390)
        at java.lang.Thread.run(Thread.java:619)
helma.framework.TimeoutException: Request timed out
        at helma.objectmodel.db.Node.checkWriteLock(Node.java:272)
        at helma.objectmodel.db.Node.setString(Node.java:1963)
        at helma.objectmodel.db.Node.getGroupbySubnode(Node.java:1221)
        at helma.objectmodel.db.NodeManager.getNodeByRelation(NodeManager.java:1591)
        at helma.objectmodel.db.NodeManager.getNode(NodeManager.java:278)
        at helma.objectmodel.db.NodeManager.getNode(NodeManager.java:184)
        at helma.objectmodel.db.WrappedNodeManager.getNode(WrappedNodeManager.java:60)
        at helma.objectmodel.db.NodeHandle.getNode(NodeHandle.java:75)
        at helma.scripting.rhino.HopObject.checkNode(HopObject.java:175)
        at helma.scripting.rhino.HopObject.getFromNode(HopObject.java:826)
        at helma.scripting.rhino.HopObject.get(HopObject.java:814)
        at org.mozilla.javascript.ScriptableObject.getProperty(ScriptableObject.java:1544)
        at org.mozilla.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2032)
        at org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:117)
        at org.mozilla.javascript.gen.c144._c3(/home/twoday_net/helma/apps/twoday_net/code/Site/renderFunctions.js:97)
        at org.mozilla.javascript.gen.c144.call(/home/twoday_net/helma/apps/twoday_net/code/Site/renderFunctions.js)
        at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:66)
        at org.mozilla.javascript.gen.c140._c1(/home/twoday_net/helma/apps/twoday_net/code/Site/actions.js:15)
        at org.mozilla.javascript.gen.c140.call(/home/twoday_net/helma/apps/twoday_net/code/Site/actions.js)
        at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
        at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2834)
        at org.mozilla.javascript.gen.c140.call(/home/twoday_net/helma/apps/twoday_net/code/Site/actions.js)
        at org.mozilla.javascript.Context.call(Context.java:538)
        at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:296)
        at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:390)
        at java.lang.Thread.run(Thread.java:619)
Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: Java heap space
Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: Java heap space

Bild 1.png (image/png, 119.2 KB) - not displayed