Re: [Fresco-devel] [ReFresco 03] Proposed network architecture overview

(Finally replying to this, sorry it took so long; not sure how much
you still think various things, so I'll just pretend the message is
current and you can ignore comments that miss the mark.)

On Mon, Feb 02, 2004 at 02:14:10PM +0100, Tobias Hunger wrote:
> Nathaniel Smith <[email protected]> wrote:
> > > Let me do the following assumptions:
> > > 
> > >   * connections will not loose messages in transit and messages will
> > >     arrive in the same order that they were send down one connection.
> <snip>
> > Yes, we definitely assume this.
> 
> Wow, we do agree on something right on the first try! ;-)

Yay :-).

> > > Then I'd propose the following header for messages:
> 
> Aehm... make that packets, not messages! Sorry, I got confused by the
> fragmentation flag in the message... Why do you have that by the way?
> With 5byte message length field you shouldn't get into trouble there
> anytime soon.

Hrm, I _know_ I went over this, maybe on IRC?  Anyway, for
posterity...

The server is getting packets from clients A, B, C, and needs to
forward them to client F.  For safety, the server cannot queue a
packet to F until it has buffered the entire packet in memory,
because if the server were to start forwarding a packet from A and A
suddenly stopped transmitting data, the server would be stuck --
there's no way to tell F "oops, never mind, let's abort this packet in
the middle", so as a result, F would be totally starved.

So the server needs to buffer whole packets in memory; if it has a
complete packet, it can guarantee timely delivery.  However, the
server doesn't want to be forced to use arbitrarily large amounts of
memory to do so; that would be too easy a DoS.  Furthermore, it
doesn't want to send arbitrarily large packets on; they take an
arbitrarily long amount of time to transfer, and while that's
happening F can't receive other (possibly urgent) packets.

Similarly, as you mention, clients may not want to commit to sending
big huge packets that take a while to transfer; they may need to
preserve the ability to send other urgent packets in the middle, and
queuing a single giant packet will kill latency.

The solution is discretionary fragmentation: whenever the server has
decided that it's buffered as much as it's willing to, it declares
that to be a complete packet and sends it on.  Clients that want to
send a lot of data can split it into appropriately sized chunks and
send them one at a time, leaving themselves room to insert other
packets in between.

> > >    2 octects message flags:
> > >       first flag: meaning MESSAGE or REPLY
> > 
> > I don't understand what this paragraph means.  Perhaps you could
> > expand a bit on your assumptions re: the difference between messages
> > and replies?  Like, what's a "result queue"?  And why shouldn't reply
> > delivery be via a method call?  (In an asynchronous world, replies
> > always come as callbacks in one way or another...)
> 
> Having sleapt another night over ths problem: You are right. This
> flag is not necessary.

Yay again :-).

> > Not sure why you want to distinguish these cases.  I've been imagining 
> <snip> 
> > >       second flag: meaning FRAGMANT_PART or FRAGMENT_END
> > 
> > Doesn't work, because as explained below, fragment packets do need a
> > "fragment stream" identifier;
> <snip>
> 
> I am beginning to suspect that you are right here too, but I'm not sure
> yet.

How about now? ;-)

> >  Right now it's not obvious to me if there are any errors
> > to report besides "object not found",
> 
> "Connection lost", "Illegal fragment/packet", "Out of memory"
> (message is too long), ...

Are any of those reported across the network, though?  They seem like
something each client should worry about inside their own network
library, not things that we need to worry about coding in the
protocol.

Now, I have this feeling that on IRC someone suggested another error
that could occur and that would need forwarding across the network,
but I can't for the life of me remember it...

> <snip>
> > Your DOS scenario isn't a problem either.
> 
> I still think it is: Lets assume there are two object A and B in the same
> client. They share one connection to the server. A can now block that
> connection for long times by sending huge messages (the max. message length
> takes several days to go over a 100MBit network). B can't send data to the
> server during that time.
> 
> You argue that this is not an issue as it is the client's responsibility
> to look after its own latencys... I see that differently.

No, you don't see it differently, is the problem :-).  The client has
to look after its own latency either way.  In actual practice, both of
our clients end up doing the same thing -- taking packets and
splitting them up.  The difference is that you impose a particular and
fairly arbitrary policy, while I leave it to the clients/client
libraries to choose whatever they like.  Note also that your policy
doesn't even necessarily help.  If you replace:
   queue_packet(foo)
with
   for chunk in foo.split(bufsize=4096):
       queue_partial_packet(chunk)
then you haven't gained anything; you still have a single contiguous
blob of data queued, it just has some extra headers stuck in the
middle of it now.  To actually get an advantage, you have to do some
special handling where you don't _actually_ queue any fragment until
you know that the previous one has actually reached the wire, etc.,
and this complexity doesn't care which solution we use.

Worse, the previous can be hidden in the network library, but in
general things can't be; the client itself has to care, because
fragmenting packets like this changes ordering semantics.  If I do
  send_message(foo); send_message(bar)
correct behavior really demands that foo be received before bar; only
the client knows if this constraint can be relaxed in a particular
case, and has to do something to say so.  (Maybe have a special
send_eventually() library call that does what I described in the
previous paragraph with a separate queue and all.)

> <snip>
> > > A fragment stream field is not necessary: Under the assumptions made above
> > > the packages arrive in order. One object will of course not sprinckle other
> > > communication inbetween one message it is sending, so there's no danger of
> > > a mix up there either.
> > 
> > No, this is wrong.  Say we have
> > 
> >    A --\
> >         Server --- C
> >    B --/
> > 
> > A and B both start sending long messages to C.
> <snip>
> > So as soon as the server is unwilling to buffer any more,
> > it starts fragmenting the messages from A and B, and sending the
> > fragments to C.  These fragments will arrive intermixed.
> 
> ... which is not a problem as long as they were send by two separate
> originators. That's basically a stream number... of course this is
> assuming that a originator will be set, even on asynchronous messages
> which is not the case with the architecture you are proposiing. Mine
> is different in that respect.

I'm really not sure I understand this.  Perhaps by "originator" you
mean the cap included for error delivery?  That's not true; e.g., I
expect lots of messages to have the null cap as their errors-to
address, meaning, "I don't care if this goes through or not".  (Most
method replies, one-way callbacks, etc. have this property.)

> <snip>
> > > > > I still fail to see why we don't need sequence numbers. Is the cap thrown
> > > > > away after receiving a single message?
> 
> Of course it is! I got it now!

Yay three times! :-)

-- Nathaniel

-- 
.i dei jitfa fanmo xatra