Proposal for inter-process communication

For the purpose of this discussion, I will exclude all
reference to shared memory or the sharing of resources
in a time-sensitive manner. I will also purposely
exclude all reference to sequential coherence.

One of the popular objections to clusters is you need
special software, using special-purpose communications
methods. This is bogus in the abstract, and only true
if the implementation doesn't support any IPC.

I will start by describing simple message-passing.
This is the easiest and simply requires that the
message be intercepted, relayed to the remote machine
with the target process, and then re-sent. If the
cluster uses UPIDs, this isn't too bad, as you'll know
if the UPID exists on your node and can always look up
what node the UPID is on if you've not already cached
it.

For message-passing (and other IPC) that can be
relayed over tipc, you simply use that. Nothing
further is needed. So, the above message-passing
mechanism is only useful for message-passing that
isn't trapped by tipc already.

Semaphores - and, indeed, any other IPC that requires
only the sending of a single byte or bit - cannot be
done efficiently. The overheads on even the simplest
packet massively exceed the size of data. Semaphores
are also used a lot for synchronization, and networks
have latencies in the order of micoseconds
(Dolphinics, Infiniband), milliseconds (Ethernet) or
longer (IP-over-Avian). Way too slow for a lot of
things.

However, synchronization over a cluster is always a
pain and unless you can now buy tachyon-based home
networking gear at Best Buy, anything written for a
cluster will be subject to the same limitations, and
MPI barrier operations are going to be a LOT slower
than just delivering a trivial packet over the
network.

That is sufficient for basic IPC between processes to
occur between nodes on a fairly transparent basis.
When it's too slow, just migrate the processes
together on the same node. Problem solved.

What about other mechanisms?

Well, socket connections will be fun, as you will need
to reflect then redirect the connection to not only
the correct node but quite possibly the correct port
on that node - unless port assignments are
cluster-wide. "Localhost" is the worst case, because
you have to go to where the user-side stub is, check
for port mappings, find out the user-side stub
attached to that port, find the machine it is now on,
get the process stub to identify the port, then check
mappings to see what the actual port is. And then make
the connection.

RPC is about the same, only you'd have to do the RPC
mappings as well.

CORBA (yes, people use it, mostly for avionics from
what I understand) is totally evil. When a process
migrates, it also has to handle all of the ORB
communication needed to resituate itself in addition
to migrating all of the connections.

In the cases of RPC and CORBA, you'd need a dedicated
extension for each to support the necessary activity.

And what do you get, if all of this is done? Well, you
can't cluster shmem operations, but you can cluster
nearly everything else in use without the need of a
solution hard-coded into the user apps.

What about other possibilities?

It would be good to have extensions that would allow
brave adventurers in GCC-land to support an OpenMosix
target for a subset of OpenMP. Then even when you did
want hard-coded solutions, the compiler can give you
native ones.

How central are these?

I would regard none of them as inherently CORE, but
I'd probably have TIPC and other really primitive
operations plugged as extensions directly on top of
CORE. Everything else listed here I'd build as
extensions on top of those IPC primitives.



       
____________________________________________________________________________________
Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated for today's economy) at Yahoo! Games.
http://get.games.yahoo.com/proddesc?gamekey=monopolyherenow  

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/