Re: [Fresco-devel] [ReFresco 03] Proposed network architecture overview

On Sun, Feb 01, 2004 at 02:02:55PM +0100, Tobias Hunger wrote:
> Nathaniel Smith <[email protected]> wrote:
> > Current packet layout in my prototype is as follows:
> >    1 octet: packet type identifier
> >       (Currently this is one of MESSAGE, MESSAGE_FRAGMENT,
> >        MESSAGE_FRAGMENT_END)
> >    5 octets: packet length
> <snip>
> > ... sender ... reply-to ... fragment stream field
> 
> Let me do the following assumptions:
> 
>   * connections will not loose messages in transit and messages will
>     arrive in the same order that they were send down one connection.
> 
>     TCP, pipes etc. should fullfill this requirement

Yes, we definitely assume this.
 
>   * Objects will process messages in the order they receive them

I would say rather that objects will receive messages in the order
they were sent, and process them in an order that depends on the
object.  E.g., a promise factory guarantees that any message to it
will be processed immediately and synchronously before any more
messages on that connection are processed; but other objects may not
provide such a strong guarantee.  In particular, it's entirely likely
that if I call a slow method A and then a fast method B, that B will
complete before A does.

Not sure whether this contradicts you or not, but it should clarify
things perhaps :-).

> Then I'd propose the following header for messages:
> 
>    2 octects message flags:
>       first flag: meaning MESSAGE or REPLY
> 
>         This flag is needed so the receiver may figure out whether to
>         send this on to an object initiating a new method call (once
>         marshalling is doen) or whether this needs to get added to the
>         result queue.

I don't understand what this paragraph means.  Perhaps you could
expand a bit on your assumptions re: the difference between messages
and replies?  Like, what's a "result queue"?  And why shouldn't reply
delivery be via a method call?  (In an asynchronous world, replies
always come as callbacks in one way or another...)

Not sure why you want to distinguish these cases.  I've been imagining 

>       second flag: meaning FRAGMANT_PART or FRAGMENT_END
> 
>         An unfragmented message will be marked "FRAGMENT_END" to signal
>         that the receiver can immediently pass on the massage and does
>         not need to wait on follow up messages.

Doesn't work, because as explained below, fragment packets do need a
"fragment stream" identifier; so fragment packets and message packets
have different layout.  Unless we just include that field in every
packet, and treat unfragmented messages as complete fragment
streams...

>       third flag: meaning NO_ERROR or ERROR
> 
>         We do need a way to signal "object not found" and similar
>         occurances at the message passing level. We can not say we will
>         handle this at the marchalling level (which will need its own
>         error handling).

I've been going back and forth in my own mind whether marking some
messages as errors makes sense.  Error handling is the ickiest part of
the protocol; if we didn't have to report errors, we could get out of
the policy business altogether.  But we do have to report errors, so
we have to know about reply-to caps and decide on a way to format
error messages and so on.  One way or another, here, we have to break
the nice boundary between the messaging protocol and the marshalling
protocol.

The issues are sufficiently murky that I think we should wait until
we've had a chance to experiment with things more before picking a
solution.  Right now it's not obvious to me if there are any errors
to report besides "object not found", or what the right way to format
messages is, or if clients will want to distinguish between "no such
object"-type errors and "no such method"-type errors (the first is
part of the messaging protocol, the second part of the marshalling
protocol), ...

>       Rest reserved for future use.
> 
>    2 octets message length (including header)
> 
>       For practical reasons I suggest not having huge messages: One object
>       could DOS a connection by sending one huge package without other
>       objects having the chance of getting anything across the same wire.
> 
>       Of course both client and server need to be able to fragment messages
>       under this size constraint.

No, this section has problems.

Firstly, 2 octets is way too small; that puts the maximum message size
at ~65 kilobytes.  Uploading image files, or even just text buffers,
will go over this _really_ easily.  And then when connection speed
doubles a few more times, everyone will laugh, because megabyte-sized
packets won't seem huge.

Your DOS scenario isn't a problem either.  The server doesn't want to
buffer arbitrary amounts of data, so whenever it's received more than
some (server chosen) amount of data in a message, it'll just turn it
into a fragment and send it off.  So it's not possible to send
arbitrarily large packets at people; the server will split them up
anyway.  And it's better to let the server decide how big a packet can
get, instead of building it into the protocol, for the reasons above.

Other DOS scenarios are trickier to deal with, in particular when
there is a disparity of connection speeds (i.e., the local client can
pump messages at the modem-bound client and fill up its queue
arbitrarily deeply).  I've thought about how to deal with some of
these issues, but I'll save that for another missive... it all comes
down to buffer handling policy in the server.

>   16 octets recpient
> 
>   16 octets reply-to
> 
>      I put this last since we might want to leave this out at some
>      point, adding a flag to specify that the reply-to is gone missing;-)
> 
>      And the recipient is needed first anyway. 
> 
> A fragment stream field is not necessary: Under the assumptions made above
> the packages arrive in order. One object will of course not sprinckle other
> communication inbetween one message it is sending, so there's no danger of
> a mix up there either.

No, this is wrong.  Say we have

   A --\
        Server --- C
   B --/

A and B both start sending long messages to C.  The server doesn't
want to buffer these long messages; a requirement of the design is
that the server's per-connection memory usage be bounded by a
constant.  So as soon as the server is unwilling to buffer any more,
it starts fragmenting the messages from A and B, and sending the
fragments to C.  These fragments will arrive intermixed.

The Python server prototype you've seen does this already.

> > > I still fail to see why we don't need sequence numbers. Is the cap thrown
> > > away after receiving a single message?
> 
> Well, I do so now after having slept a night over ewalnut;-) I think I'm
> slowly getting into this;-)
> 
> > I suspect that often a cap will be thrown away after receiving a
> > single message, yes.
> 
> I don't think that's necessary under the assumptions I made above.
> 
> Let's assume Alice sends Bob two messages, expecting a reply for both of them.
> Bob gets to work on those in the order he received them (and under the
> assumptions made above that's the order in which Alice made them, maybe with
> some of Clive's and Dave's requests thrown in between which does not matter here).
> Bob finishes the first one, sending a message marked REPLY back to Alice. She
> knows that this must be the answer for her first request.

No she doesn't.  Suppose Alice's first message required some
information from Clive, or even stopping to prompt the user for
something, while the second message didn't.  Then the second message
will probably complete first, as I was saying above.

> Then Bob does some work for Clive and must get some information from Alice for
> that: He send a message marked "MESSAGE" to Alice requesting that info. Alice
> will not mix that one up with the expected reply due to the missing REPLY-flag.

Hrm, I think the names are causing some confusion here, being
ambiguous between clients and objects.  I would expect most caps to
either provide a normal interface, or gather replies, but not both.

> Alice replies, Bob finishes his work for Clive and replies to him, Bob starts
> to work on the second request for Alice and sends his next reply, all works out.
> 
> Let's assume Bob is funny and send an unrequested third reply: Alice does not wait
> for one and will just throw it out.
> 
> Of cource Alice could get a new Cap for each request and throw that away after
> each reply, thus making sure Bob can not be funny, but in a "trusted" local
> case this seems extremly wasteful to me.

I think this is sorta silly :-).  Firstly, as explained above, you
probably do need multiple caps anyway.  Secondly, caps are just not in
short supply.  My silly little unoptimized and highly paranoid
prototype, running in Python with optimization turned off, can
generate 25,000 a second; one can definitely do better than that.
Thirdly, lessening security in the name of performance is almost
always a bad idea, especially when one does it in an ad hoc and
unpredictable way. C'mon, man, Principle of Least Authority!

-- Nathaniel

-- 
Details are all that matters; God dwells there, and you never get to
see Him if you don't struggle to get them right. -- Stephen Jay Gould

This email may be read aloud.