Re: [Imap-protocol] QRESYNC and new arrivals which get deleted immediately through VANISHED

Hi Jan,

On 8 Jun 2012, at 17:02, Jan Kundrát <[email protected]> wrote:

> Hi,
> a big part of this mail is probably answered in RFC5162's errata, but
> I'd still like someone to explicitly confirm that how I understand this
> thing is correct.
> 
> I'm afraid I have a possible race condition in my code which keeps
> mailbox up to date with server's responses. Right now, my code supports
> CONDSTORE but not QRESYNC. The following bits describe how it behaves
> after the mailbox has been "synced", ie. when my idea about UID -> seq
> mapping matches with the state on server.
> 
> In this situation, whenever the program needs to perform something (like
> send the LIST command, issue a FETCH for some body part, store some
> flags through UID STORE etc), the command will be sent to the IMAP
> server almost immediately, and this can in turn trigger many interesting
> responses like EXISTS or EXPUNGE. That's why I'm ready to process them
> at any point, no matter if there's a command in progress or not. I
> believe that I'm handling them correctly now:
> 
> - Whenever I receive EXISTS >= current number of messages, I simply add
> the correct number of "empty messages" with UID zero to the end of my
> UID -> seq mapping and to some internal data structure which is in turn
> shown to the user who will see a few "Loading..." placeholders. I also
> immediately queue a UID FETCH previous_uidnext:* FLAGS command. The goal
> is that the UID number will eventually arrive and when they arrive, I
> update my on-disk cache with the obtained data.
> 
> - When EXPUNGE arrives, I check the number for correctness (1 <= number
> < mailbox_size) and remove the message which is at that particular
> index. If its UID was not zero, I also remove any data associated with
> that UID from my on-disk cache. If the number was zero, well, the idea
> is that this shouldn't matter at all as I couldn't possibly have any
> useful data for that message in my cache (message parts, envelopes and
> body structure are fetched only after the UID is known).
> 
> I'm not sure how to deal with it if I enable QRESYNC, though. The spec
> says that I should be ready for both EXPUNGE and VANISHED. I'd like to
> leave my handling of EXPUNGE unchanged if possible, which means that I
> really have to create and insert the fake UID-0 messages immediately in
> response to increased EXISTS.
> 
> The problem is what shall I do when I get VANISHED with a UID which is
> bigger than the highest UID I already know? (Like if a message arrived
> and before I had a chance to ask for its UID, it gets removed through
> VANISHED.) At this point it looks like I can just pick any of the still
> unknown messages in my in-memory state and delete one of them for each
> high UID received in the VANISHED response. I'm a bit nervous about that
> approach, though -- consider the following situation (which is partially
> answered by the QRESYNC errata):
> 
> (Mailbox contains one message with UID 5; UIDNEXT is 11, everything is
> synced.)
> 
> S: * 12 EXISTS
> C: x UID FETCH 11:* (FLAGS)
> 
> (At this point, server sends unsolicited data like FLAGS or even
> BODYSTRUCTURE for the new arrivals. I believe that it's allowed to do so
> per RFC3501.)
> 
> S: * 2 FETCH (FLAGS (a) BODYSTRUCTURE ...)
> S: * 3 FETCH (FLAGS (b) ENVELOPE ...)
> 
> (I'm on a slow network, so even before my UID FETCH arrives to the
> server, one of these messages is gone and the server sends back VANISHED:)
> 
> S: * VANISHED 11
> 
> (The programmer has flipped a coin and hence the code always deletes the
> very last message with UID=0. That means that the message flagged "b" is
> gone and "a" remains.)
> 
> S: * 2 FETCH (FLAGS (b) UID 12)
> 
> (At this point, client finally gets the UID for the only remaining
> message with UID=0, along with flags. The bad thing is that user has
> already seen a stub message flagged as "a" while in fact there was just
> the "b" one.)

Right, guessing which of the yet unknown to your client messages got expunged doesn't look right.
> 
> After reading the errata, I realize that servers are discouraged from
> not sending UIDs in FETCH responses when QRESYNC has been enabled. Would
> it be reasonable for my client to always ignore FETCH updates which do
> not contain UID when QRESYNC is in effect?

Yes.

> (I can easily change my code
> to never issue a FETCH, always UID FETCH with QRESYNC.)

Good idea.

> 
> My biggest concern is that my code perform potentially intrusive actions
> (like allocating complex structures for BODYSTRUCTURE and actually
> showing them to the user) upon seeing the FETCH responses for a given
> message for the first time. When I receive interleaved VANISHED and
> FETCH-without-UID responses, the result could be confusing to the user
> (and given that I already have an application where the messages are
> pushed into an external DB as soon as they arrive, I can imagine a use
> case where people would want to store only messages with a particular
> flag -- again, the code would get confused in the situation I described
> above.)

It looks like your code should get smarter if you don't want to just ignore FETCH responses with no UIDs. 

> 
> It seems to me that if the QRESYNC extension introduced new * ARRIVED
> <uid-sequence-set> response to replace EXISTS in a similar way that
> VANISHED replaces EXPUNGE, the window where client knows about new
> arrivals but doesn't know about the UIDs would cease to exist. Is there
> a particular reason why QRESYNC is not modeled in that way?

The idea didn't' occur at the time of writing the QRESYNC spec :).
> 
> Finally, my last question is why does QRESYNC require me to issue an
> ENABLE QRESYNC at all? I feel like the SELECT ... QRESYNC should be
> enough to tell the server that I'm really expecting the VANISHED
> responses. My biggest issue with ENABLE is that (at least according to
> the last errata by Alfred Hoenes from 2008-03-11 which is "held for
> document update") I'm supposed to wait for untagged ENABLED before
> issuing SELECT ... QRESYNC which introduces another roundtrip to the
> connection process. I'm ignoring this requirement for now, so I'll
> happily pipeline ENABLE QRESYNC with SELECT ... QRESYNC and hope that I
> won't get BAD in response from some pedantic server. I'd love to get
> some comments on this as well.

ENABLE QRESYNC failing should be a rare event, so doing the optimization you do seems to be Ok, as long as you can handle the BAD response.

Best Regards,
Alexey


_______________________________________________
Imap-protocol mailing list
[email protected]
http://mailman2.u.washington.edu/mailman/listinfo/imap-protocol