End-of-line or just carriage-return

First off, I'd like to thank Moshe Bar and all the
other programmers involved in openMosix - I firmly
believe that it is the best of the SSI solutions out
there at present.

I also believe that SSI is going to be more important,
not less, on the next generation of machines. The next
version of PCI Express supports multiple masters on a
single network of PCI switches. However, QoS seems to
be limited to bandwidth reservation for a master to be
established at setup. There is no way of dynamically
configuring the next generation PCI-e switches
according to varying demands.

How does this impact SSI? Well, if you can't move the
allocation, move the process. If you can migrate
between masters, you can load-balance a PCI-e system
in a way that the hardware does not support (by
design).

There is also the case that programs are usually
smaller than the data sets being manipulated. In a
general-purpose cluster, it is sometimes the case that
shifting the program image is vastly less expensive
than shifting the data. SIMD would seem to be the
perfect market for this, because you're guaranteed to
be running the same code on all nodes, but each node
will have its own data - which it would therefore need
to collect, rather than having a central collection
point.

DSM is a problem - and a problem I have worked hard on
for the last few years. Part of the problem is that
when you copy to N nodes using a unicast delivery
system, you must transmit the data N times. That is
slow. NACK-Oriented Reliable Multicast would reduce
the copies needed, cutting bandwidth requirements and
the total elapsed time between start of first packet
to end of last packet.

RDMA (Remote Direct Memory Access) is also an
interesting technology, allowing one node to copy from
local memory to remote memory, or remote memory to
local memory, in a single operation without
intervention by the kernel on either side. However,
most machines are not provided with RNICs or
Infiniband cards. Pity.

A project of mine has been to extend RDMA to support
multicasting - something the specs don't currently
allow for - so that you can transfer to multiple
memories in one go. Devising a protocol that the RDMA
groups can sign on to is proving difficult. There are
a large number of corner cases and the more I fix, the
more I find. If RDMA proves impossible, then any
zero-copy kernel-bypass protocol would work in
principle. This would avoid having specialized
hardware but you'd need an emulation layer to provide
compatibility to the bulk of interesting software.

Multicast with zero-copy does not totally solve the
problem of DSM, but it reduces some of the issues. You
shed a whole bunch of context switches, stack
latencies, buffer copies, backoffs caused by switch
flooding, etc. It's hard to quantify, but my
calculations repeatedly show that the latency is
roughly 5.6 microseconds plus the per hop latency on
the longest path for every 4K, regardless of the
number of nodes actually delivered to.

Another networking technology I have been taking a
close look at is Active Messaging. There's a project -
GAMMA - that implements AM on Linux. Again, the idea
is to reduce the wasted cycles involved. AM looks very
interesting, but the implementation seems to be very
card-specific and doesn't play nice with other
protocols. To use AM in a general-purpose cluster
would need a fair amount of work put into it.

TIPC is also worthy of examination, as anything that
reduces what a clustering solution needs to do to
cluster transparently is a Good Thing.

Ok, I've now completely flogged, beaten and trampled
this poor dead horse into the ground, I'll mutilate a
few other things, the first of which is the SSI
concept.

Strictly speaking, openMosix does not need to be SSI.
Indeed, it would make some sense to use the IBCS
patches to load and start applications for a range of
Intel-based *nixes and then migrate the process space
to an instance of the actual OS it is intended for. We
now have a multi system image clustering technology.

The second issue is with real-time support. Real-time,
in this context, means that if process X is guaranteed
a timeslice of Y out of a total timeslice of Z, with a
margin of error M, then no matter how time is divided
up, how busy/free the system is, whether the system
voluntarily hands back control, etc, it WILL occupy
that amount of the available time, give or take only
the margin of error, nothing more.

This can be fun when clustering. You don't hear of
hard real-time clusters for a reason, and the reason
has nothing to do with a lack of need. What does
real-time mean for openMosix? It means that where such
state information exists, openMosix needs to copy it.
It means that processes need to migrate according to a
packing algorithm that maximizes the chances of always
meeting the timeslice goals. If real-time networking
is also being used, then you also need the packing
algorithm to consider network latencies.

Real-time does not mean the fastest solution, which is
what most SSI strategies look for. Nearest neighbors
and all that. You will occupy a certain length of time
even if the transfer takes less, so you must place at
the greatest distance that the transfers will still
meet deadlines even under the greatest-expected
network loads.

Last, but by no means least, if the project is to go
unmaintained, does anyone object to it being listed on
the Unmaintained Free Projects pages?

Jonathan

P.S. Ok, I lied, that wasn't last, this is. If there
are people from this list going to the Open Source
conference in Portland, Oregon, next week, I suggest
we arrange some sort of mini meetup. Accepting that
Moshe has found nobody with the time and ability to
run the project at this time, I propose that those of
us who know gifted coders work on finding some, and
that those who have contacts in the moneyed world see
if the project can be sponsored somehow to the point
where a team of developers can hack it to perfection
on a viable salaried basis.



       
____________________________________________________________________________________
Need a vacation? Get great deals
to amazing places on Yahoo! Travel.
http://travel.yahoo.com/

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/