Re: JavaSpace Problems

What version are you running? What JDK?

Robert Daley wrote:
> Over the last couple of years I have been implementing the SAMUEL
> (Genetic Algorithm) learning system on top of a JINI/JAVASPACES based
> master-worker computation model.  As expected the tasks of evaluating
> the members of the current population are put  in the (persistent)
> JavaSpace by the master, completed and returned by the workers.  All of
> this is done using the Transactiion Manager (persistent Mahalo) to
> insure reliability.  Some experiments can take several days to complete
> using a cluster, so reliability is a critical issue.  In general
> everything has performed very well, but I have had capacity problems
> (OutOfMemoryError)  with the JavaSpace (Outrigger) and now allocate
> more memory for the JavaSpace's shared-memory JVM.  The reason for this
> message is the following problem.
>         I have a student running SAMUEL experiments on a small (16-node)
> cluster.  Sometimes he starts too many experiments at the same time
> with the result that the disk partition on the node where the JavaSpace
> is running fills up in one of two ways.  One way is that the
> outrigger.log directory contains a large number of (what appear to be)
> incremental log files -- normally there is one main log and one or two
> incremental logs.

To be clear, normally you see one or two incremental logs, but when this
happens you see a lot of incremental log files. How many?

With 2.n we have not been seeing backups of incremental log files. It might
become a problem with the new batch ops (I assume you are not using the
new batch ops). [note to self, adding some sort of throttle inside of
Outrigger might be a good idea....]. What sort of throughput do you
think you are generating? How much/what sort of data is in the entries?

Are there any exceptions coming out when this happens? Can you get a thread
dump from Outrigger?

> The other way is that, since I save the rmid output into a file, the
> JavaSpace repeatedly attempts to reactivate, but produces an exception
> and quits -- this eventually fills up the partition as well.  I don't
> have a lot of detailed information for you at the moment.  When this
> happens I get a lot of complaints, so I need to restore the partition
> ASAP.  I'm in the process of setting up a machine separate from the
> cluster for hosting the JavaSpace so that I can find out exactly what
> is causing this, and how to work around it.

RMID does not have a lot options around controlling re-start. It won't
try more than twice on any given re-start attempt, but if you have a bunch
of clients repeatedly asking Outrigger to be restarted I could see running
into this sort of problem.

In 2.n you could switch to using JERI with durable references and craft your own
restart system - one that would stop retries after some fixed number of times.
I could also see piping RMID's output to a program that uses some sort
of circular buffer scheme instead of one big file so you can limit the
amount of disk space consumed by RMID's output.

In any event it would be good to know why Outrigger can't restart,
do you have any good stack traces in these log files?

Hope this helps

--
BTW, if you want to reply to this message, please direct your reply
to the list, thanks

John McClain                                    [email protected]
Sun Microsystems, Inc.
Burlington, MA

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. - George Bernard Shaw

===========================================================================
To unsubscribe, send email to [email protected] and include in the body
of the message "signoff JAVASPACES-USERS".  For general help, send email to
[email protected] and include in the body of the message "help".

To view past JAVASPACES-USERS postings, please see:
http://archives.java.sun.com/archives/javaspaces-users.html

JDC members can download the JavaSpaces(tm) Technology from:
http://developer.java.sun.com/developer/products/jini/