Re: Outrigger memory leak?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

i have no answer least of all a solution but we encountered the same
problem first time in 2003 after setting up a test server for our portal
based on rio/jini.
Under heavy load the transient outrigger failed after an hour or so.
The logs showed lots of OutOfMemoryError exceptions, same as you see it
in your test app.
We tested different memory,cpu,OS,jvm configurations
(Win2000,Solaris,linux on amd, intel and sparc).
The result was always the same, different configurations (e.g more
memory) only delays the problem.
Starting Outrigger and Mahalo (we are using Transactions) from command
line or within Rio does not change the behaviour.

The workflow is very simple: A module is writing an entry to space, a
client waits for notification from space and takes the entry out of space.

All what we found was, as John stated in his answer, the GC is getting
into some troubles.

Sample test started today:

4GB Memory
2*Intel(R) XEON(TM) CPU 1.80GHz
uname -a
Linux dba2 2.6.16.27-0.6-bigsmp #1 SMP Wed Dec 13 09:34:50 UTC 2006 i686
i686 i386 GNU/Linux

/usr/local/java java -version -server
java version "1.6.0"
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)

Rio3.2,Jini2.1

Entry sizes between < 1Kb to 1.5Mb

/usr/local/java -server -Xmx512m -verbose:gc ...


04.01.2007 17:44:50
[GC 3072K->372K(7680K), 0.0049380 secs]
[GC 1983K->456K(7680K), 0.0125360 secs]

15 minutes later:

04.01.2007 17:59:57
[GC 115292K->69462K(122944K), 0.0290720 secs]
[Full GC 69462K->63902K(130688K), 0.5178370 secs]

after an hour:

[Full GC 501055K->500879K(512384K), 0.7101180 secs]
04.01.2007 18:44:32
[Full GC 501055K->500729K(512384K), 0.8147690 secs]
[Full GC 501055K->500765K(512384K), 1.0540600 secs]
04.01.2007 18:44:34
[Full GC 501055K->498970K(512384K), 0.7306400 secs]

and shortly (between 2 and 10 minutes) before the out of memory
exceptions happens (and if the server load isn't already up in the sky):

04.01.2007 18:58:54
[Full GC 501055K->501055K(512384K), 0.7279520 secs]
[Full GC[Unloading class
sun.reflect.GeneratedSerializationConstructorAccessor1999]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1997]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor2005]
 501055K->499289K(512384K), 1.4706750 secs]
[Full GC 501056K->501056K(512384K), 0.7607370 secs]
[Full GC[Unloading class
sun.reflect.GeneratedSerializationConstructorAccessor2017]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1375]
[20+ identical entries deleted]
[Unloading class sun.reflect.GeneratedMethodAccessor138]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor2016]
[Unloading class sun.reflect.GeneratedMethodAccessor154]
 501056K->499352K(512384K), 1.3770250 secs]
[Full GC 501055K->501055K(512384K), 0.8272070 secs]
[Full GC 501055K->501055K(512384K), 0.7326680 secs]
[Full GC 501055K->501055K(512384K), 0.8064620 secs]
[Full GC 501055K->499327K(512384K), 0.7300490 secs]
[Full GC 501055K->501055K(512384K), 0.7275900 secs]
[Full GC 501055K->501055K(512384K), 0.7293740 secs]
[Full GC 501055K->501055K(512384K), 0.8068770 secs]
[Full GC 501055K->499489K(512384K), 1.3840900 secs]


>         I'm running up against out-of-memory errors using transient outrigger.
> I got the errors in a production system that runs a fair bit of data
> through the space, then wrote a test script to try and isolate the
> problem (code attached below).  When running with max heap size at the
> default (512M, I believe), the Outrigger instance runs out of heap after
> about 2100 iterations with a payload size of 250e3 bytes.  With smaller
> payloads, I get more iterations (although curiously not in any kind of
> linear relationship.  hmmm...), but I still get the failure eventually.
>
>         Note that the test case verifies that the entry is gone after I 'take'
> it.  I've also confirmed in other tests that it disappears on its own
> when the lease time expires, so everything seems to be behaving as
> expected.  Letting the space sit idle for a short time between accesses
> also has no effect (I wondered if perhaps the reaping thread didn't get
> a chance to run).  I've tried setting 'reapingInterval=5000;
> reapingPriority=Thread.MAX_PRIORITY', which had no effect.
>
>         Am I missing something in the configuration?  The problem is
> independent of whether Outrigger is run under Harvester or from the
> Launch-All script in the starter kit.  I haven't found any other bug
> reports, and I find it hard to believe I've found a bug in a mature
> product.  Any suggestions?
>

- --
.mostly harmless
 jeamy lee
 http://www.jeamy.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFnVtJW+f8aPeJggsRArD+AJ9r6QXs6C5BXDByKtJVtmK3KnmzCgCeMNsJ
SpR8MD7O9kczdTlNvoeOQAg=
=Rsa9
-----END PGP SIGNATURE-----

===========================================================================
To unsubscribe, send email to [email protected] and include in the body
of the message "signoff JAVASPACES-USERS".  For general help, send email to
[email protected] and include in the body of the message "help".

To view past JAVASPACES-USERS postings, please see:
http://archives.java.sun.com/archives/javaspaces-users.html