Making multiple instances of a RMI method in (from?) one server implementation

I'm starting on a JavaSE RMI application and having some trouble working
out some points in the architecture. Let me first lay our what I'm
trying to accomplish, then let you know where I'm at and finally ask the
2 specific questions I need answered.

I have fully functioning program that processes data sets that are given
to it at random times. It takes a few minutes to an hour to process each
data set into a result set.  Instead of  processing each data set in
sequence, my goal is to hand out data sets, one set to each available
computer on a lan segment, and then collect the results sets as they
become available for further organization, storage and distribution.
This way I'll be processing data sets in parallel, to the limit of the
number of computers on the lan. Data sets in excess of the number of
computers available would be queued, waiting for a computer to become
available  to process them.

I have a partial design which I've implemented in skeletal  form just to
get a better understanding of RMI and remote synchronization. I have a
single  "distributor" program and a set (actually 2 for now) of  data
"processor" programs. The "distributor" has a threadsafe incoming queue
which  holds the names of  data "processor" programs that are available
to process a data set. The names are submitted to the queue by the
"processor" programs when they become idle. I pull names out of that
queue and call the instance of the data "processor" program
corresponding to that name using  RMI. Since the one "distributor" and
the multiple "processor" programs access services from one another using
RMI they act as both server and client to one another. At this point I
have this bare skeleton of  a  subsystem (without doing any actual
processing of data) working ok.

The two inter-related design problems I'd like advice on have to do with
devising a method for the data "processor" instances to return their
result sets to the "distributor" once they're done.

Problem 1: There is no guarantee that a data "processor" will succeed in
sending its result set to the "distributor" (the "processor" program,
the vm its running on, the computer hosting the vm or the communications
link can all die). And I'm afraid that if a "processor" program crashes
while sending its result set back to the "distributor" no other
"processors" will be able to send their result sets (e.g. that a single
failure while sending to a synchronized method on the "distributor" will
kill the whole system). Is this a valid concern? If so, maybe each data
processor ought to have an individual point of contact with the
distributor for the purpose of returning its result set. This way if one
processor dies while "owning" the interface to the "distributor" other
"processors" can still send their result sets; since they each have a
separate data return point.. But how would I do this? Could I make a
thread in the "distributor" for each "point of contact" between each
"processor" and the "distributor"? If so, how is something like that
mechanized?  I can't imagine that the "distributor" thread could pass
its "this" pointer as an arg to an RMI method on another vm on another
computer and have it resolve to something usable. So am I woried aobut a
real problem or not ? And if so, is there a way  (hopefully a simple,
easy to implement way) to accomplish my goal of reliably reporting
result sets to the distributor. (Remember I'm not interested in
reporting result sets in parallel,  just getting result sets from all
live data "processors" eventually, even if a "processor" fails while
reporting its results) ?

Problem 2: The results set produced by the "processor" may be quite
large.  Large enough that the results can't be sent as the RMI method
return value from the call  to the "processor". The vm runs out of
memory (at least in Eclipse it does) when I try and return a a test byte
array of 10MB from the "processor" method. Instead, I'm thinking that
I'd  like to send the results over to the "distributor" piecemeal, by
defining a "resultReciever" RMI method in the "distributor" that the
"processor" could call as many times as necessary to move its whole
result set.  The method signature would look something like

public void resultReceiver(int blockNr, int blockTotal, int byteCnt,
byte[] result);

Where blockNr is the sequence number of the current block being sent,
blockTotal is the number of the last block that will be sent, byteCnt is
the size of the current block being sent and result is the data block
itself. To make sure that no blocks are missing I'd keep a lastBlockNr
in each instance of the object that implements the thread containing the
resultReceiver method at the "distributor". Does this sound like a
reasonable approach?

Thanks in advance for your advice.

Lenny Wintfeld

===========================================================================
To unsubscribe, send email to [email protected] and include in the body
of the message "signoff RMI-USERS".  For general help, send email to
[email protected] and include in the body of the message "help".

For a list of frequently asked RMI questions please refer to:
http://java.sun.com/j2se/1.3/docs/guide/rmi/faq.html

To view past RMI-USERS postings, please see:
http://archives.java.sun.com/archives/rmi-users.html