Re: Problems with helma Swarm and jgroups

"Hannes Wallnoefer" <[email protected]> Fri, 11 Jan 2008 10:22:08 +0100
Newsgroups	gmane.comp.java.helma.general
Message-ID	<[email protected]>
Hi Philipp,

first, an administrative note: your message was held back by the
mailing list software because of the attachment. The list is
configured for a max message size of 40kb. I accepted your message for
this time, next time please post attachments on the web and just
include a link!

2008/1/10, Franz Philipp Moser <[email protected]>:
> Hi list,
>
> I experinced latetly some outOfMemory Exceptions when using current
> helmaswarm. With a profiler I was able to find 35824 org.jgroups.Message
> Objects that consume a total of 300.000.000 Bytes of Memory (see
> screenshot, retained size). This only happens on the app server that is
> NOT the Swarm Master.

I think this is probably a problem with the JGroups configuration
stack. A google search for "jgroups memory leak" revealed this:

http://osdir.com/ml/java.javagroups.general/2006-12/msg00022.html

This looks very much like your problem. A few questions:

- Can you find out by which class the retained Message objects are
referenced? This should be possible to find out with Yourkit profiler
(don't ask me how).
- Which Jgroups stack do you use? Helmaswarms standard UDP stack does
contain pbcast.Stable, but it doesn't have
"discard_delivered_msgs="true"" in its NACKAK config.

I also found that the current JGroups release is 2.6.1, whereas
helmaswarm currently comes with 2.4.1-SP3. Updating JGroups might be
worth a try, too.

hannes

> I don't know why this happens but it looks like if the "notMaster"
> (acceptor) doesn't purge the no longer needed Message objects.
>
> "Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8009]"
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> I have appended a log file with the outOfMemory errors.
>
> Can anybody help?
>
> cu Philipp
>
> Exception in thread "SwarmSessionMgr-twoday_net" java.util.ConcurrentModificationException
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
>         at java.util.HashMap$KeyIterator.next(HashMap.java:828)
>         at java.util.AbstractCollection.toArray(AbstractCollection.java:124)
>         at helma.swarm.SwarmSessionManager.broadcastIds(SwarmSessionManager.java:250)
>         at helma.swarm.SwarmSessionManager.run(SwarmSessionManager.java:130)
>         at java.lang.Thread.run(Thread.java:619)
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> Dumping heap to java_pid1694.hprof ...
> Exception in thread "AJP13Listener1-187" Heap dump file created [514915279 bytes in 13.606 secs]
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>         at org.mortbay.util.LineInput$LineBuffer.<init>(LineInput.java:702)
>         at org.mortbay.util.LineInput.<init>(LineInput.java:127)
>         at org.mortbay.http.HttpInputStream.<init>(HttpInputStream.java:74)
>         at org.mortbay.http.HttpConnection.<init>(HttpConnection.java:120)
>         at org.mortbay.http.ajp.AJP13Connection.<init>(AJP13Connection.java:62)
>         at org.mortbay.http.ajp.AJP13Listener.createConnection(AJP13Listener.java:230)
>         at org.mortbay.http.ajp.AJP13Listener.handleConnection(AJP13Listener.java:209)
>         at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
>         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> Exception in thread "AJP13Listener1-190" java.lang.OutOfMemoryError: GC overhead limit exceeded
>         at org.mortbay.util.LineInput$LineBuffer.<init>(LineInput.java:702)
>         at org.mortbay.util.LineInput.<init>(LineInput.java:127)
>         at org.mortbay.http.HttpInputStream.<init>(HttpInputStream.java:74)
>         at org.mortbay.http.HttpConnection.<init>(HttpConnection.java:120)
>         at org.mortbay.http.ajp.AJP13Connection.<init>(AJP13Connection.java:62)
>         at org.mortbay.http.ajp.AJP13Listener.createConnection(AJP13Listener.java:230)
>         at org.mortbay.http.ajp.AJP13Listener.handleConnection(AJP13Listener.java:209)
>         at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
>         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> Exception in thread "AJP13Listener1-181" java.lang.OutOfMemoryError: GC overhead limit exceeded
>         at org.mortbay.util.ByteArrayPool.getByteArray(ByteArrayPool.java:59)
> Exception in thread "AJP13Listener1-185" java.lang.OutOfMemoryError: GC overhead limit exceeded
>         at org.mortbay.util.ByteArrayPool.getByteArray(ByteArrayPool.java:59)
>         at org.mortbay.http.ajp.AJP13Packet.<init>(AJP13Packet.java:91)
>         at org.mortbay.http.ajp.AJP13RequestPacket.<init>(AJP13RequestPacket.java:56)
>         at org.mortbay.http.ajp.AJP13InputStream.<init>(AJP13InputStream.java:37)
>         at org.mortbay.http.ajp.AJP13Connection.<init>(AJP13Connection.java:62)
>         at org.mortbay.http.ajp.AJP13Listener.createConnection(AJP13Listener.java:230)
>         at org.mortbay.http.ajp.AJP13Listener.handleConnection(AJP13Listener.java:209)
> Exception in thread "Thread-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Error in XML-RPC listener loop (java.lang.OutOfMemoryError: GC overhead limit exceeded).
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "Thread-1" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "AJP13Listener1-203" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "AJP13Listener1-75" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "AJP13Listener1-221" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "AJP13Listener1-222" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "AJP13Listener1-92" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "AJP13Listener1-223" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Error in XML-RPC listener loop (java.lang.OutOfMemoryError: GC overhead limit exceeded).
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "AJP13Listener1-227" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "Timer-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8009]" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "AJP13Listener1-121" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: Java heap space
> helma.framework.TimeoutException: Request timed out
>         at helma.objectmodel.db.Node.checkWriteLock(Node.java:272)
>         at helma.objectmodel.db.Node.setString(Node.java:1963)
>         at helma.objectmodel.db.Node.getGroupbySubnode(Node.java:1221)
>         at helma.objectmodel.db.NodeManager.getNodeByRelation(NodeManager.java:1591)
>         at helma.objectmodel.db.NodeManager.getNode(NodeManager.java:278)
>         at helma.objectmodel.db.NodeManager.getNode(NodeManager.java:184)
>         at helma.objectmodel.db.WrappedNodeManager.getNode(WrappedNodeManager.java:60)
>         at helma.objectmodel.db.NodeHandle.getNode(NodeHandle.java:75)
>         at helma.scripting.rhino.HopObject.checkNode(HopObject.java:175)
>         at helma.scripting.rhino.HopObject.getFromNode(HopObject.java:826)
>         at helma.scripting.rhino.HopObject.get(HopObject.java:814)
>         at org.mozilla.javascript.ScriptableObject.getProperty(ScriptableObject.java:1544)
>         at org.mozilla.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2032)
>         at org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:117)
>         at org.mozilla.javascript.gen.c144._c3(/home/twoday_net/helma/apps/twoday_net/code/Site/renderFunctions.js:97)
>         at org.mozilla.javascript.gen.c144.call(/home/twoday_net/helma/apps/twoday_net/code/Site/renderFunctions.js)
>         at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:66)
>         at org.mozilla.javascript.gen.c140._c1(/home/twoday_net/helma/apps/twoday_net/code/Site/actions.js:15)
>         at org.mozilla.javascript.gen.c140.call(/home/twoday_net/helma/apps/twoday_net/code/Site/actions.js)
>         at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
>         at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2834)
>         at org.mozilla.javascript.gen.c140.call(/home/twoday_net/helma/apps/twoday_net/code/Site/actions.js)
>         at org.mozilla.javascript.Context.call(Context.java:538)
>         at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:296)
>         at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:390)
>         at java.lang.Thread.run(Thread.java:619)
> helma.framework.TimeoutException: Request timed out
>         at helma.objectmodel.db.Node.checkWriteLock(Node.java:272)
>         at helma.objectmodel.db.Node.setString(Node.java:1963)
>         at helma.objectmodel.db.Node.getGroupbySubnode(Node.java:1221)
>         at helma.objectmodel.db.NodeManager.getNodeByRelation(NodeManager.java:1591)
>         at helma.objectmodel.db.NodeManager.getNode(NodeManager.java:278)
>         at helma.objectmodel.db.NodeManager.getNode(NodeManager.java:184)
>         at helma.objectmodel.db.WrappedNodeManager.getNode(WrappedNodeManager.java:60)
>         at helma.objectmodel.db.NodeHandle.getNode(NodeHandle.java:75)
>         at helma.scripting.rhino.HopObject.checkNode(HopObject.java:175)
>         at helma.scripting.rhino.HopObject.getFromNode(HopObject.java:826)
>         at helma.scripting.rhino.HopObject.get(HopObject.java:814)
>         at org.mozilla.javascript.ScriptableObject.getProperty(ScriptableObject.java:1544)
>         at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:265)
>         at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:868)
>         at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:670)
>         at helma.framework.core.Skin$Macro.render(Skin.java:752)
>         at helma.framework.core.Skin.render(Skin.java:276)
>         at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:223)
>         at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:155)
>         at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:474)
>         at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:76)
>         at org.mozilla.javascript.gen.c95._c10(/home/twoday_net/helma/apps/twoday_net/code/HopObject/macros.js:202)
>         at org.mozilla.javascript.gen.c95.call(/home/twoday_net/helma/apps/twoday_net/code/HopObject/macros.js)
>         at org.mozilla.javascript.Context.call(Context.java:531)
>         at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:296)
>         at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:868)
>         at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:670)
>         at helma.framework.core.Skin$Macro.render(Skin.java:752)
>         at helma.framework.core.Skin.render(Skin.java:276)
>         at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:223)
>         at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:155)
>         at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:474)
>         at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:76)
>         at org.mozilla.javascript.gen.c95._c10(/home/twoday_net/helma/apps/twoday_net/code/HopObject/macros.js:202)
>         at org.mozilla.javascript.gen.c95.call(/home/twoday_net/helma/apps/twoday_net/code/HopObject/macros.js)
>         at org.mozilla.javascript.Context.call(Context.java:531)
>         at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:296)
>         at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:868)
>         at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:670)
>         at helma.framework.core.Skin$Macro.render(Skin.java:752)
>         at helma.framework.core.Skin.render(Skin.java:276)
>         at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:223)
>         at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:155)
>         at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:474)
>         at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:76)
>         at org.mozilla.javascript.gen.c144._c4(/home/twoday_net/helma/apps/twoday_net/code/Site/renderFunctions.js:130)
>         at org.mozilla.javascript.gen.c144.call(/home/twoday_net/helma/apps/twoday_net/code/Site/renderFunctions.js)
>         at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2234)
>         at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:257)
>         at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:127)
>         at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:76)
>         at org.mozilla.javascript.gen.c21._c3(/home/twoday_net/helma/apps/twoday_net/code/Global/Helma.Aspects.js:37)
>         at org.mozilla.javascript.gen.c21.call(/home/twoday_net/helma/apps/twoday_net/code/Global/Helma.Aspects.js)
>         at org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:119)
>         at org.mozilla.javascript.gen.c207._c1(/home/twoday_net/helma/apps/twoday_net/code/Day/actions.js:24)
>         at org.mozilla.javascript.gen.c207.call(/home/twoday_net/helma/apps/twoday_net/code/Day/actions.js)
>         at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
>         at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2834)
>         at org.mozilla.javascript.gen.c207.call(/home/twoday_net/helma/apps/twoday_net/code/Day/actions.js)
>         at org.mozilla.javascript.Context.call(Context.java:538)
>         at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:296)
>         at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:390)
>         at java.lang.Thread.run(Thread.java:619)
> helma.framework.TimeoutException: Request timed out
>         at helma.objectmodel.db.Node.checkWriteLock(Node.java:272)
>         at helma.objectmodel.db.Node.setString(Node.java:1963)
>         at helma.objectmodel.db.Node.getGroupbySubnode(Node.java:1221)
>         at helma.objectmodel.db.NodeManager.getNodeByRelation(NodeManager.java:1591)
>         at helma.objectmodel.db.NodeManager.getNode(NodeManager.java:278)
>         at helma.objectmodel.db.NodeManager.getNode(NodeManager.java:184)
>         at helma.objectmodel.db.WrappedNodeManager.getNode(WrappedNodeManager.java:60)
>         at helma.objectmodel.db.NodeHandle.getNode(NodeHandle.java:75)
>         at helma.scripting.rhino.HopObject.checkNode(HopObject.java:175)
>         at helma.scripting.rhino.HopObject.getFromNode(HopObject.java:826)
>         at helma.scripting.rhino.HopObject.get(HopObject.java:814)
>         at org.mozilla.javascript.ScriptableObject.getProperty(ScriptableObject.java:1544)
>         at org.mozilla.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2032)
>         at org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:117)
>         at org.mozilla.javascript.gen.c144._c3(/home/twoday_net/helma/apps/twoday_net/code/Site/renderFunctions.js:97)
>         at org.mozilla.javascript.gen.c144.call(/home/twoday_net/helma/apps/twoday_net/code/Site/renderFunctions.js)
>         at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:66)
>         at org.mozilla.javascript.gen.c140._c1(/home/twoday_net/helma/apps/twoday_net/code/Site/actions.js:15)
>         at org.mozilla.javascript.gen.c140.call(/home/twoday_net/helma/apps/twoday_net/code/Site/actions.js)
>         at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
>         at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2834)
>         at org.mozilla.javascript.gen.c140.call(/home/twoday_net/helma/apps/twoday_net/code/Site/actions.js)
>         at org.mozilla.javascript.Context.call(Context.java:538)
>         at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:296)
>         at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:390)
>         at java.lang.Thread.run(Thread.java:619)
> Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "TX-twoday_net" java.lang.OutOfMemoryError: Java heap space
>
> _______________________________________________
> Helma-user mailing list
> [email protected]
> http://helma.org/mailman/listinfo/helma-user
>
>
>