[Fresco-devel] [ReFresco 03] Proposed network architecture overview

ReFresco 03: Proposed network architecture overview
===================================================

In this note I give a high-level view of how my proposed network
architecture works.  There are details left out, but it will hopefully
suffice to sketch out a big picture view, and give a feel for how much
work is actually required to meet our needs.

I'll start by repeating from ReFresco 1 my list of Fresco's
requirements and desirements for a networking layer:

  1) simple and understandable, to make reasoning, fixing, and
     implementation easy
  2) connection oriented, to allow tunneling, basic authentication,
     and connection management
  3) routable: clients should be able to exchange messages among
     themselves using the server as an intermediary; does not require
     arbitrary many-to-many communication
  4) extensible and object based: CORBA got this right, though we can
     do a better job at an on the wire format than IIOP
  5) secure from the ground up
  6) asynchronous by default, with "one way" calls that work

I think that overall, we have a bit of learned helplessness when it
comes to network protocols.  The whole CORBA philosophy is that you're
a bad, bad person if you even think about what's actually happening
under the covers.  But really, network programming isn't any harder
than other sorts of programming.  We can do this if we need to.

We don't have infinite resources, though, obviously.  Simplicity is
always a critical design goal, but even more so in this case -- if we
don't make things as simple as possible, we're less likely to make
them happen at all.  (And won't it be a refreshing change from CORBA?)
When designing this system I have been ruthlessly simple.

We start with a spoke topology network.  There is a server in the
center, and there are clients that connect to the server via some sort
of stream communication protocol (with the option of negotiating
stream-level encryption etc. at start up), e.g. TCP or Unix domain
sockets.  Clients never talk to each other directly; all communication
is either from a client to the server, or the server to a client.

Each program, whether client or server, contains a multitude of
objects.  By "object" here we mean "an address you can send a message
to", where at this level, a "message" is "a bunch of octets".  Each
object is identified by a longish (say, 128 bit) bit-string.  Object
identities are made globally unique by the simple expedient of making
them (cryptographically strong pseudo-) random numbers; this is the
same trick is used by, e.g., UUIDs.  This is an incredibly powerful
technique -- not only does it give us globally unique identities
without any expensive and complicated coordination, it means that ids
are unguessable; if I know the identity of an object, I can send it a
message; if I don't, then I can't.  I.e., we have a capability
system[0].  Capabilities are an extremely powerful, flexible, and
understandable way to handle security; the most wonderful thing about
using them is that security "just works" without having to think about
it too much.  In their honor I will refer to object identities as
"caps" from now on.

[0] http://www.eros-os.org/project/novelty.html#capabilities
    http://www.skyhunter.com/marcs/narratedIntros.html

The server keeps a master table mapping each cap to the program that
implements it; clients keep a table mapping local caps to the objects
implementing them.  When a client wants to send a message to cap A, it
first checks whether A is implemented locally, and if so, delivers the
message directly.  Otherwise, it sends the message off to the server.
The server, upon receiving a message, extracts the target cap, and
looks it up in its master table.  If the target is implemented
in the server, it is delivered directly to the object; otherwise, the
message is forwarded to whichever program does implement it.

Messages also contain a second cap, to which any return values or
errors are to be reported.  If a program does not wish to receive any
response, this cap should be set to all zeroes; this is a special
address whose object is the equivalent of /dev/null -- it will eat any
messages sent to it and never do anything with them.  This is the only
mechanism for matching requests with replies; we don't have sequence
numbers or anything like that, because replies need to not be
forgeable; only the object to whom we send the request should have the
capability to reply to it.

Aside from messages to objects, clients can inform the server that
they now implement cap FOO, or that they no longer implement cap BAR;
this is how the server's master table is maintained.  If two clients
both attempt to register the same cap, whichever client does so first
wins, and the other's attempt is ignored.

Note that viewed at this layer, the actual content of the messages is
entirely irrelevant; they are simply opaque byte arrays.  (The only
exception to this is that error-reporting code must decide how to
format messages reporting errors.)

On top of this layer we define a packet format for the message
contents, describing how to marshal arguments and errors, and indicate
what method is being called.  This is straightforward, so we omit the
details for now.

I claim that this is a sufficient architecture to achieve all of the
listed goals.  (1) is hopefully obvious.  (2) is certainly obvious.
(3) and (4) are achieved directly.  (5) comes from the use of
capabilities.  The only question remaining is how to achieve (6),
asynchronicity (including pipelining).  This can be done without
modifying any of the core protocol, by adding in "promises"[1].  The
idea of a promise is that it is an object that will eventually be the
result of some operation, but can be used now as if it were already
the result of the that operation.  In our protocol, we achieve this by
giving each client access to a Promise Factory, which takes a cap and
immediately creates a promise object in the server with that cap.  A
promise object takes all messages sent to it and simply queues them up
-- except for "message reply" or "message error" messages.  If it
receives a "message reply" message, it extracts the reply value
-- assumed for simplicity to be a simply a cap to the object returned
by a request, though more complex mechanisms are possible -- and then
forwards all queued messages to that cap; any later messages are
simply forwarded without queuing.  If it receives a "message error"
message, it becomes a broken promise, and sends back errors to all
messages it has received.

[1] http://www.skyhunter.com/marcs/ewalnut.html#SEC20

This needs an example.  Say I want to create a Frame object, and then
immediately call its method "set_background_color".  The simple way
would be to do the following:
    1) send message "create_frame" to some factory object
    2) wait for a reply containing a cap to the newly created Frame
    3) send that cap a "set_background_color" message.
Note that we have to wait for a full round-trip to and from the server
between steps 1 and 2, which is bad; latency is not our friend.  What
we can do instead with promises is:
    0) send message "create_promise(A)" to some factory object (where
       A is a new cap we just created, with no associated object
    1) send message "create_frame" to some factory object, giving A as
       the cap to send a reply to
    3) send message "set_background_color" to A
Now there is no step 2; we simply send 3 messages in quick
succession.  Internally, what will happen is:
    1) server sees "create_promise(A)", creates a promise and
       registers cap A to point at it
    2) server sees "create_frame", creates a Frame B, and sends B as
       its reply value to cap A
    3) the promise object receives a reply, so it becomes a forwarding
       object to cap B
    4) A receives message "set_background_color" and forwards it
       directly to B
    5) B receives message "set_background_color" and executes it
Here there is a "round trip" between 3 and 4 -- except that this round
trip occurs entirely within the server, so the latency is zero.

"create_frame" is probably executed immediately, but this all still
works if for some reason it takes awhile and the
"set_background_color" message arrives before "create_frame" has
finished; the promise will simply hold onto the "set_background_color"
until the frame is ready to receive it.

Note that a good binding could hide most of the details of creating
promises.

-- Nathaniel

-- 
Eternity is very long, especially towards the end.
  -- Woody Allen