Re: question about persistence

Building clustered highly available JavaSpaces is not so simple.

We have invested and still investing great efforts building stable and
reliable cluster aware JavaSpaces solution.



There are many issues that need to be taken into account:

- Data replication - There are synchronous/asynchronous replication issues
that need to be resolved. Advanced applications would require reliable
parallel multicast replication implementation to provide fast and optimized
data replication. This involves heartbeat , handshake and other mechanisms
that insure data coherency. You can't expect the OS or have some file to be
used to sync the different spaces with the latest state - it is very costly
and binds you to specific OS cluster technology. Replication should be done
to entries data, lease info , take/read templates , notify templates and
other space internal data.

- Primary/backup space notion - the space need to be aware of other spaces
running in order there would not be multiple primary active spaces. One of
the most interesting problems here knows as the shot in the head approach.

- Transaction state - The source and target space might need to be aware the
transaction state or at least throw exception when current primary space
failed to allow the client to retry the operation.

- Lease state - the space which took over need to maintain the entry lease

- Notifications - the space which took over should start send notifications
to clients

- Silent failover - Connected users should move into the backup space
silently. This means the client should have proxies to all relevant spaces
to allow smooth fail in short time.

- Recovery - when the original space restarted the current active space
should sync its data with the restarted space. This means you need to build
special king of iterator that will send the restarted space the active space
data.

- Persistency - if the space is persistent the durable storage (database)
need to be re-sync when the original space restarted. This means the spaces
should maintain both redo log and to do log to allow the restarted space to
get only the changes since the time it failed from the active space and
become available only when all data was fully recovered.



As you can imagine the above is just the tip of the iceberg when it comes to
building clustered JavaSpaces.

GigaSpaces handles the above using advanced algorithms we have developed
across the last few years. More details can be found at the product manual:

http://www.gigaspaces.com/docs/manual/Data_Grid.htm

http://www.gigaspaces.com/docs/manual/Cluster_Configuration.htm



Best Regards,

        Shay

----------------------------------------------------

Shay Hassidim

Product Manager, GigaSpaces Technologies

Email: [email protected]

Website: www.gigaspaces.com <http://www.gigaspaces.com/>



-----Original Message-----
From: John McClain - Sun Microsystems, Inc. [mailto:[email protected]]
Sent: Monday, April 24, 2006 5:26 PM
To: [email protected]
Subject: Re: question about persistence



So I got this message and I realized that the subject was "question

about persistence" but most of the messages was about proxies, though it

might be helpful to tie the two together, a short summary of my message

might be this:



    For fault recovery you need two thing, durable (or persistent)

    state and durable (or persistent) references. Durable state is

    accessible to the new incarnation of the service while a

    durable reference can be used access the new incarnation of the

    service, even if it was originally generated by an old

    incarnation.



A lot of my message was about how to make sure the references to

Outrigger (that is its proxies) are durable.





John McClain - Sun Microsystems, Inc. wrote:

> Suzanne Morelli wrote:

>

>> Hi,

>>

>> I understand using persistent JavaSpace might help to recover from a

>> fault. But if the fault is more severe than just a problem with the

>> JavaSpace process, it might not be suffisant; for instance, in the case

>> where the machine hosting the JavaSpace crashes totally. In that case,

>> I'd like to restart the JavaSpace, but somewhere else, using the log of

>> the crashed machine...

>> Is it possible to indicate a persitenceDirectory that is not local?

>

>

> There are two things to worry about. One is can the new incarnation of

> the space recover the state from the old incarnation, the second is can

> clients talk to the new incarnation?

>

> Setting persitenceDirectory to point to a remotely mounted directory can

> help the first issue (assuming the remote file server doesn't fail too

> of course). How well this is going to work depends on how good the

> remote file system is about syncing. If when the file system is asked to

> fsync by Outrigger it blocks until the file's state is committed to disk

> things should be ok, if not you may lose the effects of committed

> operations and/or end up with a corrupted store.

>

> The second issue has to do with how Outrigger is exported. There are two

> parts, first how Outrigger's proxies identify the "right" object in the

> server, second how the proxies connect to the server. All current RMI

> implementations have a notion of an "object id", when making a remote

> call the object id is used to find the right object in the server VM to

> accept the call. If the new incarnation doesn't have the same object id

> as the old incarnation then the old proxies won't be able to make calls

> on the new incarnation. In JRMP (the impl of RMI that comes with Java)

> unless you use activation there is no (supported) way to set the object

> id, it is picked at random  so the new incarnation wouldn't have the

> same object id as the old incarnation. In JERI (the new RMI

> implementation that was developed for Jini 2.0) you can set the object

> id, so you can arrange for the new and old incarnation to have the same

> object id.

>

> Once you get the object id issues sorted out, Outrigger's proxies need

> some way to establish a connection to the server. All the ways that are

> supported "out of the box" involve making TCP connections to a specified

> host and port. If the new incarnation comes up on a machine with the

> same "address" and on the same port then then old proxies will work with

>  new incarnation. The host in the proxy can be specified using either a

> host name or an IP address, obviously if you use a host name you will

> have additional options in making the new machine "look" like the old one.

>

> [It is worthing noting, in some cases, you can arrange things so

>  it is ok if the old proxies stop working by having the clients get

>  new proxies if the old proxies stop working. Where this gets difficult

>  is when it is hard/imposable for clients to get new proxies - which

>  comes up when using transactions and leases (though in the leasing case

>  there may be other solutions)]

>

> One way to address both problems (storage and addressing) is to use some

> sort of clustering solution (e.g. Sun Cluster, I think there are similar

> clustering solution for other operating systems) where you run machines

> in pairs with shared disks. Outrigger runs on one machine, if that

> machine crashes the second machines automatically takes control over the

> disks, takes the first machine's IP address, and restarts Outrigger. (I

> think Dan Creswell had a talk at the 7th Jini Community Meeting about

> doing this sort of thing with Blitz (on Linux?), but if memory serves

> what he did should work with any restartable service (including

> Outrigger)).

>

> If you are more of a code-it-your-self type there are also various

> layers in JERI  where you could plug in code to deal with the addressing

> issue, and in Outrigger where you could plug-in code to address the

> storage issue.

>

> Hope this helps





--

BTW, if you want to reply to this message, please direct your reply

to the list, thanks



John McClain                                    [email protected]

Sun Microsystems, Inc.

Burlington, MA



no files, just services



===========================================================================

To unsubscribe, send email to [email protected] and include in the body

of the message "signoff JAVASPACES-USERS".  For general help, send email to

[email protected] and include in the body of the message "help".



To view past JAVASPACES-USERS postings, please see:

http://archives.java.sun.com/archives/javaspaces-users.html


===========================================================================
To unsubscribe, send email to [email protected] and include in the body
of the message "signoff JAVASPACES-USERS".  For general help, send email to
[email protected] and include in the body of the message "help".

To view past JAVASPACES-USERS postings, please see:
http://archives.java.sun.com/archives/javaspaces-users.html