Stubs and wrappers and possibilities, oh my!

A number of things came out of the BOF meeting on
Tuesday, but I shall leave most of the discussion on
that to the formal writeup of the minutes of the
meeting by Wes. I will only say that everyone there
was excited by the idea of having development
continue, with good feelings about where we are.

One of the things examined was the use of a stub that
was not migratable between nodes. I was asked to write
a short piece describing a proposed modification to
this, so here it is. Note that suggestions, comments
and clue-stick beatings are welcome, but if you could
pad the clue-stick a little, I'd be grateful.

Brief Synopsis: Linux offers a bazillion features,
network engineers have written a bazillion others. The
I/O-side stub, as it is, works great but there may be
ways to make it more powerful and/or faster. We should
consider if this would be a useful/good thing to do,
and if so what our priorities should be.


My suggestion is to first extend the stub. Linux has a
whole host of new features which should expand the
usefulness of the stub, and it would be good to
discuss what we'd want the stub to actually do.

The second step I proposed was to allow the stub to
migrate. If the stub migrates, then all user-side I/O
would migrate with it to the new node, which must
(therefore) support the same I/O features as are being
used. So long as the information is available, I don't
see that being a problem.

(By migrating the stub rather than just the data, you
carry all of the process state, security/permission
labels, etc, for free and you can reuse existing
code.)

The problem lies in how to migrate the stubs safely
and maintain contact with the controlled process,
without adding any security complications and with
taking into account any data sent by the process
during the migration.

Linux - after applying a couple of patches - provides
methods for supporting the failover of network
connections, transient addressing, network mobility,
kernel-level switching for layers 2, 3, 4 and 7, and
virtual network devices. In short, we have so many
ways of getting a connection to switch that it's more
a case of what method we should use rather than
whether we can do it.

A worthy place to look for inspiration is the research
that has gone into IPv6 mobility. The current version
uses a transient home base that all connections link
to. The home base then talks to the user node. When
the user node moves, a redirect is sent through the
network. Any connection not yet redirected bounces off
the home base to wherever the user is. Redirected
connections go directly to the user. Depending on the
version of the mobility draft, once all connections
are redirected, the user becomes the new home base and
the old home base is closed down.

Alternatively, why keep track at all? Have a
process-side stub that listens for anycasts. When the
user-side stub migrates, it sends out an anycast
request for the UPID it was connected to. The process'
stub will be the only one to respond, so the
connection can then be restored.

What about kernel threads? Again, same thing applies.
We're not short of methods. Everything from HA
solutions to suspend-to-file would allow you to
transfer kernel-level operations from one machine to
another without problem.

What information is needed by the stub? Well, we
definitely need all file descriptors, netlink sockets,
unix sockets, pipes, memory tags, tipc sockets to the
remote process and regular sockets for handling the
I/O. You'll also want the security tokens for both the
stub and the remote process, and the stub's state
information. Almost all of that already exists and the
others are relatively minor additions.

Is there anything significant that's needed that's
new? Well, the stub should export to userspace any
information needed to perform migration and perhaps
resource allocation. At the meeting, we thought about
various strategies for this. /proc and /sys were
suggested, as was using a page of shared memory. I'd
love to hear other people's thoughts on this.

(The point is that the data changes extremely
infrequently, very few processes will want it, there
is a potential security hole if anything can access
that data and modify it, and copying from the kernel
is slow so should only ever occur on an event, never
on a per-read basis.)

I also suggested that since Linux now supports
real-time and that time-slicing is much more
predictable under new scheduler, we might want to
consider all processes under the control of OpenMosix
as real-time. Each process then has a fixed-sized
bucket of time, and the scheduler can use a
conventional packing herustic to pack those buckets
into the cluster.

How is this better than using, say, the load average?
The load average doesn't tell you what's taking the
time or why, or whether the remaining time is usable.
(I'm not certain how much you can fragment the block
of time that the new scheduler uses to timeslice, but
I can be pretty certain that any desktop system is
going to have enough events to fragment the time
block, leaving unusable chunks.)

Ok, so we want to export the timeslice requirements.
What else? Any universal ID would be good to export,
along with what machine the process is running on. We
also need to export memory tags, as RDMA should not be
done in the kernel.

Why not export everything? Because 99% of what is on
the stub is strictly for I/O to other kernel-based
components. Very little is needed or wanted by
userspace, and the less you have to copy, track or
update, the less latency you'll suffer and the lower
your requirements. You also get better security.

This ends the reading of the stub.



       
____________________________________________________________________________________
Pinpoint customers who are looking for what you sell. 
http://searchmarketing.yahoo.com/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/