Re: JavaSpace Problems

Robert Daley wrote:
> Over the last couple of years I have been implementing the SAMUEL
> (Genetic Algorithm) learning system on top of a JINI/JAVASPACES based
> master-worker computation model.  As expected the tasks of evaluating
> the members of the current population are put  in the (persistent)
> JavaSpace by the master, completed and returned by the workers.  All of
> this is done using the Transactiion Manager (persistent Mahalo) to
> insure reliability.  Some experiments can take several days to complete
> using a cluster, so reliability is a critical issue.  In general
> everything has performed very well, but I have had capacity problems
> (OutOfMemoryError)  with the JavaSpace (Outrigger) and now allocate
> more memory for the JavaSpace's shared-memory JVM.  The reason for this

I believe Outrigger keeps a complete copy of the database content in
memory and backs it up to disk using logs etc.  i.e.  Outrigger does not
"swap" to disk - your memory must be big enough to hold all Entry
instances plus operational overhead (a lease for each entry,
transactional state etc - note also that, in general, you can't assume
that a lease expired Entry or a taken Entry won't consume memory of disk
space for some period of time).

> message is the following problem.
>        I have a student running SAMUEL experiments on a small (16-node)
> cluster.  Sometimes he starts too many experiments at the same time
> with the result that the disk partition on the node where the JavaSpace
> is running fills up in one of two ways.  One way is that the
> outrigger.log directory contains a large number of (what appear to be)
> incremental log files -- normally there is one main log and one or two
> incremental logs.

Hmmm, well you'll have some number of outrigger logs and then whatever
files the ODI database which Outrigger uses generates.

The basic model is this:

(1)     Outrigger "front-end" generates logs of operations that it is
performing against it's memory copy.

(2)     A separate thread in the "back-end" "consumes each log" trickling
updates into the database.  When a log has been consumed it is deleted
but the pace of log consumption will determine how long a log file lives
for so if Outrigger is under high load you _might_ see speed of log
generation go up whilst speed of consumption goes down.

(3)     After a certain number of log files are consumed, the ODI database
is garbage collected/checkpointed which will cause ODI to clear out log
files (if it has any - implementation dependent) and consolidate it's
database files.

There are two knobs you can twiddle - both via Outrigger's configuration
file - gcInterval which controls the number of log files that must be
consumed before a gc/checkpoint occurs.  maxOps which determines how
many operations Outrigger saves to a log before starting a new log.
This will, therefore, have some impact on log size.

You may find that by reducing gcInterval and maxOps you can get smaller
log files and less total disk space taken on average but there will
likely be a performance cost as a result of such tweaking.

I don't know how much disk storage you have available and I don't know
what the average size of an Entry or log file is but the cheapest
solution may just be to buy bigger disks which can cope with the space
demands.  I suspect also you may still need to do some further work to
get long term stability such as introducing some throttling mechanisms etc.

> The other way is that, since I save the rmid output into a file, the
> JavaSpace repeatedly attempts to reactivate, but produces an exception
> and quits -- this eventually fills up the partition as well.  I don't
> have a lot of detailed information for you at the moment.  When this
> happens I get a lot of complaints, so I need to restore the partition
> ASAP.  I'm in the process of setting up a machine separate from the
> cluster for hosting the JavaSpace so that I can find out exactly what
> is causing this, and how to work around it.
>        Any suggestions at this point would be welcome.

What exception are you getting from RMID?  Be useful to see this as it's
not really expected behaviour.  It could be trivially to do with the
fact that the partition is full but to eliminate it as a concern the
trace would be useful.


Hope that helps,

Dan.

===========================================================================
To unsubscribe, send email to [email protected] and include in the body
of the message "signoff JAVASPACES-USERS".  For general help, send email to
[email protected] and include in the body of the message "help".

To view past JAVASPACES-USERS postings, please see:
http://archives.java.sun.com/archives/javaspaces-users.html

JDC members can download the JavaSpaces(tm) Technology from:
http://developer.java.sun.com/developer/products/jini/