Re: Outrigger memory leak?

I think the memory leak you are referring to (which I still have on my
list to track down....) is some interaction between a particular space
access pattern, Outrigger, transactions, and JERI/TCP. It goes away if
one uses JRMP (and if memory serves it also goes away if one uses
JERI/HTTP as the RMI implementation)[*].

My gut tells me that Greg is seeing a different problem (but my gut is
often wrong in this sort of situation).

[*] One shouldn't lay the blame at JERI/TCP's feet though, since the
problem doesn't occure when you use a trivial JERI based service that
mimics the same remote call pattern. Seems to be some interaction
between Outrigger and JERI/TCP.

JeaMY Lee wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello,
>
> i have no answer least of all a solution but we encountered the same
> problem first time in 2003 after setting up a test server for our portal
> based on rio/jini.
> Under heavy load the transient outrigger failed after an hour or so.
> The logs showed lots of OutOfMemoryError exceptions, same as you see it
> in your test app.
> We tested different memory,cpu,OS,jvm configurations
> (Win2000,Solaris,linux on amd, intel and sparc).
> The result was always the same, different configurations (e.g more
> memory) only delays the problem.
> Starting Outrigger and Mahalo (we are using Transactions) from command
> line or within Rio does not change the behaviour.
>
> The workflow is very simple: A module is writing an entry to space, a
> client waits for notification from space and takes the entry out of space.
>
> All what we found was, as John stated in his answer, the GC is getting
> into some troubles.
>
> Sample test started today:
>
> 4GB Memory
> 2*Intel(R) XEON(TM) CPU 1.80GHz
> uname -a
> Linux dba2 2.6.16.27-0.6-bigsmp #1 SMP Wed Dec 13 09:34:50 UTC 2006 i686
> i686 i386 GNU/Linux
>
> /usr/local/java java -version -server
> java version "1.6.0"
> Java(TM) SE Runtime Environment (build 1.6.0-b105)
> Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)
>
> Rio3.2,Jini2.1
>
> Entry sizes between < 1Kb to 1.5Mb
>
> /usr/local/java -server -Xmx512m -verbose:gc ...
>
>
> 04.01.2007 17:44:50
> [GC 3072K->372K(7680K), 0.0049380 secs]
> [GC 1983K->456K(7680K), 0.0125360 secs]
>
> 15 minutes later:
>
> 04.01.2007 17:59:57
> [GC 115292K->69462K(122944K), 0.0290720 secs]
> [Full GC 69462K->63902K(130688K), 0.5178370 secs]
>
> after an hour:
>
> [Full GC 501055K->500879K(512384K), 0.7101180 secs]
> 04.01.2007 18:44:32
> [Full GC 501055K->500729K(512384K), 0.8147690 secs]
> [Full GC 501055K->500765K(512384K), 1.0540600 secs]
> 04.01.2007 18:44:34
> [Full GC 501055K->498970K(512384K), 0.7306400 secs]
>
> and shortly (between 2 and 10 minutes) before the out of memory
> exceptions happens (and if the server load isn't already up in the sky):
>
> 04.01.2007 18:58:54
> [Full GC 501055K->501055K(512384K), 0.7279520 secs]
> [Full GC[Unloading class
> sun.reflect.GeneratedSerializationConstructorAccessor1999]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1997]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor2005]
>  501055K->499289K(512384K), 1.4706750 secs]
> [Full GC 501056K->501056K(512384K), 0.7607370 secs]
> [Full GC[Unloading class
> sun.reflect.GeneratedSerializationConstructorAccessor2017]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1375]
> [20+ identical entries deleted]
> [Unloading class sun.reflect.GeneratedMethodAccessor138]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor2016]
> [Unloading class sun.reflect.GeneratedMethodAccessor154]
>  501056K->499352K(512384K), 1.3770250 secs]
> [Full GC 501055K->501055K(512384K), 0.8272070 secs]
> [Full GC 501055K->501055K(512384K), 0.7326680 secs]
> [Full GC 501055K->501055K(512384K), 0.8064620 secs]
> [Full GC 501055K->499327K(512384K), 0.7300490 secs]
> [Full GC 501055K->501055K(512384K), 0.7275900 secs]
> [Full GC 501055K->501055K(512384K), 0.7293740 secs]
> [Full GC 501055K->501055K(512384K), 0.8068770 secs]
> [Full GC 501055K->499489K(512384K), 1.3840900 secs]
>
>
>>         I'm running up against out-of-memory errors using transient outrigger.
>> I got the errors in a production system that runs a fair bit of data
>> through the space, then wrote a test script to try and isolate the
>> problem (code attached below).  When running with max heap size at the
>> default (512M, I believe), the Outrigger instance runs out of heap after
>> about 2100 iterations with a payload size of 250e3 bytes.  With smaller
>> payloads, I get more iterations (although curiously not in any kind of
>> linear relationship.  hmmm...), but I still get the failure eventually.
>>
>>         Note that the test case verifies that the entry is gone after I 'take'
>> it.  I've also confirmed in other tests that it disappears on its own
>> when the lease time expires, so everything seems to be behaving as
>> expected.  Letting the space sit idle for a short time between accesses
>> also has no effect (I wondered if perhaps the reaping thread didn't get
>> a chance to run).  I've tried setting 'reapingInterval=5000;
>> reapingPriority=Thread.MAX_PRIORITY', which had no effect.
>>
>>         Am I missing something in the configuration?  The problem is
>> independent of whether Outrigger is run under Harvester or from the
>> Launch-All script in the starter kit.  I haven't found any other bug
>> reports, and I find it hard to believe I've found a bug in a mature
>> product.  Any suggestions?
>>
>
> - --
> .mostly harmless
>  jeamy lee
>  http://www.jeamy.com
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFFnVtJW+f8aPeJggsRArD+AJ9r6QXs6C5BXDByKtJVtmK3KnmzCgCeMNsJ
> SpR8MD7O9kczdTlNvoeOQAg=
> =Rsa9
> -----END PGP SIGNATURE-----
>
> ===========================================================================
> To unsubscribe, send email to [email protected] and include in the body
> of the message "signoff JAVASPACES-USERS".  For general help, send email to
> [email protected] and include in the body of the message "help".
>
> To view past JAVASPACES-USERS postings, please see:
> http://archives.java.sun.com/archives/javaspaces-users.html


--
BTW, if you want to reply to this message, please direct your reply
to the list, thanks

John McClain                                    [email protected]
Sun Microsystems, Inc.
Burlington, MA

And it is that way today. We are tricked by hope into starting
companies, beginning books, immigrating to this country and investing
in telecom networks. The challenges turn out to be tougher than we
imagined. Our excessive optimism is exposed. New skills are demanded.
But nothing important was ever begun in a prudential frame of mind.

         - David Brooks

===========================================================================
To unsubscribe, send email to [email protected] and include in the body
of the message "signoff JAVASPACES-USERS".  For general help, send email to
[email protected] and include in the body of the message "help".

To view past JAVASPACES-USERS postings, please see:
http://archives.java.sun.com/archives/javaspaces-users.html