Re: [Imap-protocol] Re: Suspend/Restore feature proposal

Quoting Bron Gondwana <[email protected]>:

> The interesting part for QRESYNC is remembering "tombstones" for EXPUNGED
> messages for a while to keep it cheap.

[snip]

> If anyone else is interested, I can write up something about how Cyrus
> implements QRESYNC now, such that it can clean up old records, yet still
> be efficient for clients most of the time.

I have an interest in this logic, if just for the fact that this issue  
came up recently.

We provide ActiveSync mail syncing through our IMAP library.   
(Warning: I'm not an ActiveSync expert: this is my knowledge as  
provided to me by a guy who is).  To sync a remote ActiveSync client,  
it needs to be determined which messages have been expunged since the  
previous sync.  Fantastic if QRESYNC is available - we can just use  
VANISHED so there is no need to keep the UID list at the ActiveSync  
controller layer.

However, we were seeing transactions like the following (on a dovecot server):

- HIGHESTMODSEQ as known by activesync client: 53000
- HIGHESTMODSEQ on IMAP server: 54000

a uid fetch 1:* UID (VANISHED CHANGEDSINCE 53000)
* VANISHED (EARLIER) 1:37308,37310:40788,40791:41032,41034:41083
a OK Fetch completed.

Yikes!  That's over 40,000 UIDs returned.  Sure enough, there seemed  
to be a tipping point where the "expected" VANISHED return - only  
those UIDs actually removed between 53000 and 54000 - was achieved:

a uid fetch 1:* UID (VANISHED CHANGEDSINCE 53881)
* VANISHED (EARLIER) 1:37308,37310:40788,40791:41032,41034:41083
a OK Fetch completed.
b uid fetch 1:* UID (VANISHED CHANGEDSINCE 53882)
* VANISHED (EARLIER) 37309,41029:41030,41047:41083
b OK Fetch completed.

Turns out that dovecot purges old EXPUNGE records every so often from  
the cache.  Discussion was made of ways of possibly improving this  
behavior, and I believe the idea of tombstones/checkpoints came up.

Note: I understand the above VANISHED call is not the most efficient.   
I told the ActiveSync guy that it would be good practice to also  
include the lowest/highest UID known at a given MODSEQ to make the  
VANISHED call more efficient.  However, this still leaves open the  
possibility of a large UID return range, especially if a user has an  
old message in their mailbox.  e.g. the above example could still  
possibly look like:

a uid fetch 1000:41100 UID (VANISHED CHANGEDSINCE 53800)
* VANISHED (EARLIER) 1001:41032,41034:41083
a OK Fetch completed.

This could be a limitation of ActiveSync, but the problem comes that  
this entire list of UIDs needs to be sent to the remote client.  And  
if that remote client is a mobile device, you are causing a huge  
amount of traffic to be pushed across a wireless connection, and this  
huge amount of data needs to be processed on the remote device.  This  
is potentially an expensive action, both monetary (bandwidth cost),  
UI-wise (longer time to resync), and battery wise if the mobile client  
is using this.

As a client author I would settle knowing that at most only, say,  
1,000 spurious UIDs will ever be returned from a VANISHED command.   
The above behavior can be worked around if the list of UIDs is kept at  
the activesync connector level.  But that may not be desirable in any  
given implementation.

Wondering your theory behind this as a server author.  How much more  
storage does a tombstone regime require on the server?  Is there a way  
to optimize this - i.e. maybe tombstones aren't kept unless/until a  
client actually issues a VANISHED command in a mailbox.  What is a  
reasonable checkpoint/tombstone range?  Is this something that the  
CHECK command could potentially be useful for?

michael

_______________________________________________
Imap-protocol mailing list
[email protected]
http://mailman2.u.washington.edu/mailman/listinfo/imap-protocol