Re: LCUP and PSearch on Sync-And-Persist

Timothy Hahn wrote:

> Jeff and Rich,
>
> Thanks for your consideration on this topic.
>
> I went back and reviewed the draft to brush up on the general aspects of
> the protocol definition.
>
> I still have some questions regarding how the original intentions can be
> satisfied.
>
> I see two issues:
>
> 1) How are "intermediate updates" to be handled by implementations?  I
> suspect that this may be wound up in a server's implementation of the
> "state" represented by the "cookie" value and the draft may leave this as
> an "excercise for the implementer". If so, I think it would be to
> everyone's advantage that the draft indicate "beware - invention required
> here" on this topic.  For example: if the "cookie" value changes relatively
> often (so that client's presumably don't have to re-start from scratch all
> the time), and an update to an entry is received on one thread while a LCUP
> session is "in process" on another, it is problematic for the server
> implementation to merely send the "updated entry" in the middle of the
> "synchronize" data stream for the client.  If this occurs, and the "cookie"
> is updated ... depending on the "cookie format", lots of data could be
> missed on a re-driven synchronization request.

One possible implementation is to use an RUV or some other data containing one or more CSNs in the cookie.  The CSNs of course contain a
timestamp.  If the server sends LCUP responses to the client in CSN order, I think it would be relatively easy for the server to figure out
where in the outgoing data stream to insert the "updated entry" (assuming well behaved replication).  The cookie would contain the min and max
CSNs.  One possible problem would be that the LCUP server receives an update with a very old CSN - older than the min CSN in any cookie.  But
this should not happen in well behaved replication.

> 2) How is a client supposed to know when they're "close"?  I believe the
> algorithm you've listed below, using the final "cookie" from the
> "synchronizeOnly" session as the "starting point" for the
> "synchronizeAndPersist" session, can work (assuming the server understands
> this, of course, and the client is careful to re-use the last "cookie"
> value from the first LCUP session).  Again, the dangers in "intermediate
> updates" should be considered - for example, should server implementations
> send "intermediate updates" if "synchronizeOnly" is specified?

Yes, I believe.  Let's say a client establishes the initial LCUP session at time T0 (the T part of the CSN), and the session is for
synchronizeOnly.  This means that the server should send every update with a CSN which has a timestamp Tx <= T0.  So, if the server receives
an update after T0, but the CSN has a timestamp <= T0, it should be included in the initial synchronization.

I believe if a client wants to be "close", the client should simply do a synchronizeOnly session.  If the server is not experiencing a large
update rate, this should be sufficient.  If the server is experiencing a large update rate, how can the client ever be assured of being close?

> Since "cookies" are to be used "across" LCUP sessions, it seems to strongly
> suggest that they be related to some form of timestamp.  (I know it's not
> required, but establishing "what has changed since I sent this client this
> 'cookie'" seems to lead one to see that there has to be some time component
> to the "cookie" - if only there is a table in the server implementation
> which relates the cookie value to some time and space point.  But this gets
> into trouble depending on which cookie values are "remembered" by the
> client especially if "intermediate updates" flow in the middle of the LCUP
> session.
>
> One more point - since the protocol indicates that the response to the
> client is in the form of a searchResultEntry, this implies that for entries
> with attributes that have a large number of values, small changes to those
> attributes will require large data transfers over LCUP sessions.  It would
> be good to note this in the LCUP draft as a consideration for
> implementors/users of the protocol.

That's very good to note, and might cause big problems with people wishing to use LCUP to sync lots of changes to large static groups.

>
>
> Regards,
> Tim Hahn
>
> Internet: [email protected]
> Internal: Timothy Hahn/Endicott/IBM@IBMUS or IBMUSM00(HAHNT)
> phone: 607.752.6388     tie-line: 8/852.6388
> fax: 607.752.3681
>
>
>                       "Jeff Parham"
>                       <[email protected]        To:       "Kurt D. Zeilenga" <[email protected]>, "Rich Megginson" <[email protected]>
>                       rosoft.com>                  cc:       "Mark C Smith" <[email protected]>, Timothy Hahn/Endicott/IBM@IBMUS,
>                                                     <[email protected]>
>                       04/05/2002 11:39 AM          Subject:  RE: LCUP and PSearch on Sync-And-Persist
>
>
>
>
> What is the scenario in which a client needs to do something special on
> receipt of the "in sync" notification?
>
> The only one I can see is on an "initial sync," where some clients might
> want to know they have a reasonably complete set of search results as
> they existed in the recent past (excepting loose consistency due to
> replication latency between DSAs, etc.).  In such a case the client
> could initially use a synchronizeOnly LCUP exchange with the DSA, then
> switch to synchronizeAndPersist for future LCUP interactions.
>
> The example scenario cited by Mircea -- ensuring the client has a
> consistent view across many objects -- is not one that can or should be
> solved through LCUP.  The solution to this problem is the use of
> application versioning of objects.  Many strategies for this exist --
> e.g., those detailed at
> http://msdn.microsoft.com/library/en-us/netdir/ad/detecting_and_avoiding
> _replication_latency.asp.
>
> In short, I am not yet convinced that the currently specified behavior
> of LCUP in this area is deficient.
>
> -J
>
> -----Original Message-----
> From: Kurt D. Zeilenga [mailto:[email protected]]
> Sent: Thursday, April 04, 2002 10:12 AM
> To: Rich Megginson
> Cc: Mark C Smith; Timothy Hahn; [email protected]
> Subject: Re: LCUP and PSearch on Sync-And-Persist
>
> At 09:07 AM 2002-04-04, Rich Megginson wrote:
> >"Kurt D. Zeilenga" wrote:
> >
> >> There are many clients which need to know they are 'in sync'.
> >> I note that this is not just after the initial 'sync up' phase,
> >> but subsequently.  Also, there are cases where the server
> >> needs to update the cookie without sending an update (in the
> >> 'persist' phase.
> >>
> >> I suggest that we add a 'in sync' notification to LCUP.  Basically,
> >> just a (intermediate) response PDU which says "you're in sync"
> >> and, OPTIONALLY, provides a new cookie.
> >>
> >> The server should send one when it has sent all changes necessary
> >> to 'sync' the client.  The server may send them subsequently to
> >> again notify the client it is 'in sync'.
> >
> >But what exactly does 'sync' mean?
>
> It means that the server has sent all the messages necessary for
> the client to have a complete and accurate copy of the requested
> (and provided) information.
>
> >Does this mean 'all changes between the time I last connected and the
> time I connected for this
> >session'?
>
> It means all the changes necessary to have a complete and accurate
> copy of the requested (and provided) information.
>
> >Because changes may come in to the server during the sync phase.
>
> The server may send them or queue them or possible use other
> approaches.
>
> >What should happen to these changes?  Or should those
> >entries be included?  What if the server is under a heavy update load
> and never gets to the end of the sync phase?
>
> If it never reaches the 'in sync' condition, it never should sent
> the 'in sync' message.
>
> I think we should get away from this 'sync'/'persist' phase
> business and look at this simply as a sequence up messages which
> 'updates' entries of interest.  When the sequence of updates
> sent is known to reflect the server's current state (for all
> entries of interest), the server sends an 'in sync' message
> notifying the client of this.
>
> The initial 'in sync' message should be sent as soon as the
> all the changes necessary to be 'in sync' have be sent.
> Subsequent 'in sync' messages can be sent whenever the
> server desires to send them.
>
> Kurt
> Kurt