[Imap-protocol] QRESYNC and new arrivals which get deleted immediately through VANISHED

Hi,
a big part of this mail is probably answered in RFC5162's errata, but
I'd still like someone to explicitly confirm that how I understand this
thing is correct.

I'm afraid I have a possible race condition in my code which keeps
mailbox up to date with server's responses. Right now, my code supports
CONDSTORE but not QRESYNC. The following bits describe how it behaves
after the mailbox has been "synced", ie. when my idea about UID -> seq
mapping matches with the state on server.

In this situation, whenever the program needs to perform something (like
send the LIST command, issue a FETCH for some body part, store some
flags through UID STORE etc), the command will be sent to the IMAP
server almost immediately, and this can in turn trigger many interesting
responses like EXISTS or EXPUNGE. That's why I'm ready to process them
at any point, no matter if there's a command in progress or not. I
believe that I'm handling them correctly now:

- Whenever I receive EXISTS >= current number of messages, I simply add
the correct number of "empty messages" with UID zero to the end of my
UID -> seq mapping and to some internal data structure which is in turn
shown to the user who will see a few "Loading..." placeholders. I also
immediately queue a UID FETCH previous_uidnext:* FLAGS command. The goal
is that the UID number will eventually arrive and when they arrive, I
update my on-disk cache with the obtained data.

- When EXPUNGE arrives, I check the number for correctness (1 <= number
< mailbox_size) and remove the message which is at that particular
index. If its UID was not zero, I also remove any data associated with
that UID from my on-disk cache. If the number was zero, well, the idea
is that this shouldn't matter at all as I couldn't possibly have any
useful data for that message in my cache (message parts, envelopes and
body structure are fetched only after the UID is known).

I'm not sure how to deal with it if I enable QRESYNC, though. The spec
says that I should be ready for both EXPUNGE and VANISHED. I'd like to
leave my handling of EXPUNGE unchanged if possible, which means that I
really have to create and insert the fake UID-0 messages immediately in
response to increased EXISTS.

The problem is what shall I do when I get VANISHED with a UID which is
bigger than the highest UID I already know? (Like if a message arrived
and before I had a chance to ask for its UID, it gets removed through
VANISHED.) At this point it looks like I can just pick any of the still
unknown messages in my in-memory state and delete one of them for each
high UID received in the VANISHED response. I'm a bit nervous about that
approach, though -- consider the following situation (which is partially
answered by the QRESYNC errata):

(Mailbox contains one message with UID 5; UIDNEXT is 11, everything is
synced.)

S: * 12 EXISTS
C: x UID FETCH 11:* (FLAGS)

(At this point, server sends unsolicited data like FLAGS or even
BODYSTRUCTURE for the new arrivals. I believe that it's allowed to do so
per RFC3501.)

S: * 2 FETCH (FLAGS (a) BODYSTRUCTURE ...)
S: * 3 FETCH (FLAGS (b) ENVELOPE ...)

(I'm on a slow network, so even before my UID FETCH arrives to the
server, one of these messages is gone and the server sends back VANISHED:)

S: * VANISHED 11

(The programmer has flipped a coin and hence the code always deletes the
very last message with UID=0. That means that the message flagged "b" is
gone and "a" remains.)

S: * 2 FETCH (FLAGS (b) UID 12)

(At this point, client finally gets the UID for the only remaining
message with UID=0, along with flags. The bad thing is that user has
already seen a stub message flagged as "a" while in fact there was just
the "b" one.)

After reading the errata, I realize that servers are discouraged from
not sending UIDs in FETCH responses when QRESYNC has been enabled. Would
it be reasonable for my client to always ignore FETCH updates which do
not contain UID when QRESYNC is in effect? (I can easily change my code
to never issue a FETCH, always UID FETCH with QRESYNC.)

My biggest concern is that my code perform potentially intrusive actions
(like allocating complex structures for BODYSTRUCTURE and actually
showing them to the user) upon seeing the FETCH responses for a given
message for the first time. When I receive interleaved VANISHED and
FETCH-without-UID responses, the result could be confusing to the user
(and given that I already have an application where the messages are
pushed into an external DB as soon as they arrive, I can imagine a use
case where people would want to store only messages with a particular
flag -- again, the code would get confused in the situation I described
above.)

It seems to me that if the QRESYNC extension introduced new * ARRIVED
<uid-sequence-set> response to replace EXISTS in a similar way that
VANISHED replaces EXPUNGE, the window where client knows about new
arrivals but doesn't know about the UIDs would cease to exist. Is there
a particular reason why QRESYNC is not modeled in that way?

Finally, my last question is why does QRESYNC require me to issue an
ENABLE QRESYNC at all? I feel like the SELECT ... QRESYNC should be
enough to tell the server that I'm really expecting the VANISHED
responses. My biggest issue with ENABLE is that (at least according to
the last errata by Alfred Hoenes from 2008-03-11 which is "held for
document update") I'm supposed to wait for untagged ENABLED before
issuing SELECT ... QRESYNC which introduces another roundtrip to the
connection process. I'm ignoring this requirement for now, so I'll
happily pipeline ENABLE QRESYNC with SELECT ... QRESYNC and hope that I
won't get BAD in response from some pedantic server. I'd love to get
some comments on this as well.

With kind regards,
Jan

-- 
Trojita, a fast e-mail client -- http://trojita.flaska.net/

_______________________________________________
Imap-protocol mailing list
[email protected]
http://mailman2.u.washington.edu/mailman/listinfo/imap-protocol

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/SIfoACgkQamXfqERyJRcLDQCeJ0JSSswtMT6l4fgpA/yYdoZj
XNYAniThj5SrPP7gmnIOayW48hTA7gBg
=lkga
-----END PGP SIGNATURE-----