Re: I-D ACTION:draft-tewari-webi-wcdp-00.txt,.ps

Ted,

Thanks for your comments. My responses in-line with your questions and
comments.

-Niranjan.



Ted Anderson <[email protected]> on 04/10/2002 12:36:14 PM

   To:    [email protected], Renu Tewari/Watson/IBM@IBMUS
   cc:    Srikanth Ramamurthy/Pittsburgh/IBM@IBMUS, Thirumale
          Niranjan/Pittsburgh/IBM@IBMUS
   Subject: Re: I-D ACTION:draft-tewari-webi-wcdp-00.txt,.ps


   Comments on Web Content Distribution Protocol (WCDP) based on
   draft-tewari-webi-wcdp-00.txt, by Tewary, Niranajan [sic], and
   Ramamurthy dated February 2002.

   Looks like Niranjan's name is spelled wrong.  There are some non-ascii
   characters in the .txt file which appear okay in the postscript version.

Will fix.

   In the terminology section, "heartbeat" is described as being sent from
   the invalidation server to the client caches.  However, neither of these
   two entities is defined in this section.  I'm guess that these are the
   WCDP server and WCDP client, respectively, but maybe this should be
   clarified.

OK.


   Odd that heartbeats travel from server to client, not the more
   traditional direction, from client to server.  Is there a justification
   for this?  I suppose the reason is that invalidations are emitted from
   the server, so it is the one that wants to determine proactively if an
   invalidation might timeout due to a client failure.  In the case of
   delta consistency, a missing heartbeat indicates that an invalidation
   could also have been missed.  In the case of strong consistency, the
   receipt of the heartbeat effectively extends to the lease granted by the
   server to the client.  So too many missed heartbeats allows the lease to
   expire and invalidates the corresponding cached objects.  [ Just
   thinking out loud here. ]


Since the clients represent the data repositories (caches/data servers),
they
need to know about the invalidation servers state that pertains to them
(the clients).
That is, a client would need to figure out whether the invalidation server
considers
the client "active". It needs to differentiate between a situation where
there is no
activity (e.g., nothing has been invalidated for a while), and a situation
where the
invalidation server is not sending it any invalidations because it has seen
a
network partition or some other failure. Therefore, the server needs to
send the client
a heartbeat, so that can know the state of the client as the server sees
it.


   Does scalability goal include the ability for WCDP clients to
   communicate with very many WCDP servers?

We did not consider that a requirement. However, if a client is at an ISP
(say, as a
forward proxy), I can imagine that it would have to receive invalidations
from a large
number of invalidation servers, assuming that each domain (or CDSP, in some
scenarios)
correspond to one invalidation server. In any case, our clients are
basically passive
listeners, so scalability in this respect should be easier to achieve.


   It doesn't seem like the refresh directive avoids any extra messages
   compared to simple invalidation.  Is it only intended to reduce latency?

Yes. It's an "update" feature, as opposed to plain invalidates. Our
experience shows
that the capability to preload content is key to avoiding latency. Often,
content is
accessed in a burst at predefined times, so the latency will affect not
just the first
requestor, but many (thousands of) requestors who are concurrent with the
first requestor.
The Sydney Olympics web site and hosting solution saw a lot of benefit in
avoiding
"miss storms".


   The delayed refresh directive would seem to put the onus of spreading
   delay on the server which must provide explicit scheduling to the
   clients.  It would seem better for the refresh directive to provide a
   time range which clients should select delays with uniform probability,
   at least as an option.  Assuming the server knows how many client it has
   an how many requests per second it can handle this time range should be
   easy to derive.  In the absence of such information the probabilistic
   mechanism would be much easier to configure than a static value.

Agreed. Actually, the "delayed refresh" directive is itself a secondary
option. WCDP
has in-built scalability features -- we distribute notifications in
"waves". A wave
corresponds to a set of clients, i.e., a client could belong to any wave.
Each wave
pulls content from a wave prior to it. By organizing the hierarchy
appropriately, this
allows for intermediate staging servers that could be used for content
distribution.
The notification carries with it a list of nodes that the client can pull
the content
from. The list is augmented at the beginning of each wave. I don't think
this was
described in detail in the document.


   I don't understand implicit subscriptions.  It sounds like the
   administrative step means that each fetch registers a subscription for
   the corresponding (individual) object, not a content group.


Subscriptions could be initiated in many ways: (1) server decides that a
client that
requests a certain object wll be notified about invalidations to that
object -- if we
need to keep this in sync with the content group notion, we could just say
that the
server will subscribe the client to all content groups to which the object
belongs.
(2) client can initiate the subscription itself -- 2(a) clients can run
heuristics on
access patterns, popularity etc., and come up with subscription requests
dynamically, or
2(b) administrators specify content groups that a client should subscribe
to, through a
(remote) administration API.


   In explicit subscriptions it says: "send invalidations to objects
   belonging" but you probably mean "for objects" not "to objects".

   There seems to be some confusion between the WCDP server and the Origin
   server.  The text seems inconsistent about which one sends invalidation.
   Since content groupings are part of metadata they are reported in the
   HTTP headers which presumably come from the origin server.  Clearly, it
   must know about group membership so these groups are not something the
   WCDP server can define on its own.  Maybe it would be good to clarify
   the distinction of roles more clearly or describe the practical limits
   to their separation.

Will describe this in more detail. In a nutshell, we imagine that these
groupings will be given as metadata by a content management/publishing
system,
which will also generate some kind of a notification to the invalidation
server. Therefore, the groupings are available to both the WCDP server and
the origin server.


   In section 4.4 the text says that the differences between "strong and
   weak consistencies are not significant...".  It seems that the key
   difference is that for weak consistency when sending invalidations the
   server must wait until all responses are received.  For weak it doesn't
   need to wait.  Maybe this sentence doesn't add anything and can be
   removed.

Bug. Maybe we intended to say something else :-)  I'll check.


   In node-level strong consistency, why do servers have to wait for
   clients to retrieve the new content before making it live.  This seems
   sure to delay the process (especially if the delayed refresh directive
   is used) but doesn't improve the consistency.  Also in this case, the
   client need a special way to get the not-yet-live content from the
   origin server.  How do they do this?

Let's say there is a large object that needs to be updated at two clients,
in
a strongly consistent manner. That is, we expect the clients to expose the
object to external requests at approximately the same time. However, since
the
object is large, and the speed of the network between the clients and the
origin
could be very different, one client could end up retrieving the object much
before
the other. Therefore, all clients should be synchronized as much as
possible. Doing
it in a two-phase manner achieves this. The same argument holds for why a
server
cannot make it live before that -- the end user experience will be
inconsistent.

To get at the data, clients can go to a special staging server that has
updated
content but is not exposed to public clients. Alternatively, they can go to
the origin,
but identify the request as a WCDP request, via a special header, so that
the origin
can serve out the not-yet-live content. Our product takes the first
approach; the
latter approach needs to be fleshed out in more detail.


   The force option in the invalidate request seems dubious.  What happens
   if a client declines to perform the invalidation in response response to
   a force, or an update perhaps due to a network outage.  What does it
   "mean" for the server to specify force?  How does force impact the
   semantics of the consistency guarantees?

I guess we haven't *clearly* defined the meaning of the "force" option. I
don't
understand what you mean by a "client declines" -- if we say that a force
is not
declinable, then the client cannot decline. For an update, we could say
that if
a client is not able to update the content, perhaps due to a network
outage, an
invalidate would preserve correctness. In any case, we need to specify the
exact
response codes and such.


   The wave description of propagating invalidations in section 4.6 is
   confusing.  How does the invalidation wave carry with it the list of
   notified clients and how does this influence the pulling of data?  Also
   we have clients fetching data from WCDP servers, not origin servers
   here, but I thought that was only the roll of origin servers.  Maybe the
   mapping of rolls across levels of the hierarchy need to be better
   explained.

Will try to clarify the description.


      Is the organization of an WCDP client/server hierarchy is outside the
      scope of this protocol?  It would seem that NTP-like stratum
      mechanism
      could be used to to self-organize CDN intermediates into a tree.
      This
      would allow automatic mechanisms to construct the hierarchy, which
      should be much easier to maintain and define than a static
      configuration.

Yes, it is out of scope of the protocol. I agree, there could be automatic
mechanisms for this.


      Explicit consistency is typically much weaker than strong
      consistency,
      unless the HTTP cache directives are all set appropriately (i.e. "do
      not
      cache").  Thus it would seem that falling from strong back to
      explicit
      consistency rules during a server outage would be unsafe.  Maybe
      reverting to uncached (as while waiting for a commit) would be safer
      for
      content groups that desire strong consistency?

Since there will always be caches that are not RUP-compliant,
administrators MUST
use HTTP headers in a responsible way, so that unsafe behavior is not seen
at caches.
Therefore, it is not a stretch to assume that administrators will continue
to use
these headers. Since a WCDP cache will overrule the cache-control headers
if the object
is being subscribed to, administrators can set a conservative (small) value
to the TTL,
so that safe behavior is exhibited by other caches, browsers etc.
Therefore, falling
back to explicit consistency is not unsafe.


      In section 4.7 it sounds like servers don't detect failed clients via
      heartbeat timeouts.  Is the assumption that the transport mechanism
      won't notify the server of a failure to deliver a heartbeat request
      too
      strong?  This means of failure detection would help reduce the
      latency
      of invalidations when some clients have failed.  For servers with
      many
      clients, or at the root of large hierarchies, this will likely be a
      crucial optimization.

Servers do detect failed clients when a message (including heartbeats) do
not make it
to the clients, or an acknowledgement is not received within a timeout
period.


      In handling failures of refresh directive, this section says that the
      client will try another WCDP server.  But that won't help if the
      origin
      server is down, because all WCDP server share the same Origin server.
      I
      guess just don't understand the Origin/WCDP server distinction.  Also
      I
      thought the refresh directive is basically a performance
      optimization,
      but it sounds from this paragraph (i.e. a failure to refresh caused
      the
      invalidation to fail) that there are semantic implications the affect
      the consistency guarantees.  I don't understand what those are.

   The refresh directive is indeed a performance optimization for a cache.
   But since
   we intend this to be an update protocol that is used to notify web
   servers (by that,
   I mean an authoritative source of content) in rehosting sense, or
   mirrored servers, an
   "update" can have a mandatory feel to it.

   Also, it would help to think of multiple sources of content, not just
   one "origin". If one
   fails, the client could go to any of the other nodes which has the
   updated content.


      I am sorry these comments are so late in coming, but I hope they are
      of
      some use.

      Ted Anderson