[Imap-protocol] Re: Suspend/Restore feature proposal

Michael,
I've read your response a few times and I have to admit that most of the points you raise sounds very, very alien to me. I realize that you're comming from a different background than me (different programming language, environment, users and use cases), so I'll be happy to be proven wrong -- I like learning and challenging my own views.

On Wednesday, 21 November 2012 23:54:17 CEST, Michael M Slusarz wrote:
> I would strongly disagree with this statement.  As written, the 
> draft  is only minimally concerned with saving on network 
> round-trips.

That's quite different from what I've understood from your draft -- I'd suggest making the motivation clearer, then. But point understood, and I've now purged the "let's save roundtrips" from my understanding of the draft :). OK.

> $result = $imap->useCompression(true);
> // Check for success
> $imap->useQresync(true);
> // Check for success
> $imap->setLanguage([LANGUAGE]);
> // Check for success

It is pretty obvious that if you use synchronous primitives for enabling individual sub-features in a serialized fashion, your performance will be limited by the round trip times. To put it more bluntly, you cannot have code like the one shown above and expect a good performace.

Coming from that background, I see that it is tempting to replace this endless row of synchronous calls, each enabling a single optional feature, with a quick way to side-step this process by quickly jumping into a pre-negotiated state where everything which was enabled before is enabled now as well. However, my point is that clients already exist proving that the same efficiency can be achieved with the existing facilities. You're right that this requires abolishing the serial, synchronized code, but IMAP is not particularly friendly with synchronous APIs.

> In any OO IMAP interface the order of IMAP commands to allow 
> for  efficient pipelining, or the fact that pipelining even 
> exists, should  obviously not be a part of the API.  Thus 
> pipelining is fairly useless  in the real world as a way to 
> guarantee an increase in performance.

Wrong. I have a client (written in an object-oriented language using object-oriented paradigms) which uses pipelining, and uses it succesfully, I'd say. The trick was to have the library implemented in an asynchronous way. For example, in my client the API between the IMAP-specific parts and the GUI is built on the model-view-controller design pattern; appropriate callbacks are in place to update the attached views when the requested data arrive from the network. It works beautifully in the (native application, not HTML) GUI, and another party uses this IMAP library with the exactly same asynchronous API underneath in a batch tool for stuffing incoming messages into a CRM/ERP database. Yes, the programming is different than if the library provided synchronous calls.

> A client may, depending on the capabilities returned, need to 
> perform  various internal initialization tasks.  For example - 
> if  CONDSTORE/QRESYNC is listed, a client may have to then parse 
> a  separate configuration file to grab the details of the local 
> cache  where it is storing this information, and then connect to 
> this cache,  etc.

So you want to keep the cache information (among other things) inside some serialized client-side state storage. What prevents you from simply checking the capabilities against the previously recorded state and restoring the state when the capabilities match exactly? You can do that now, without waiting for this extension. Yes, it's ugly, but if your initialization is expensive...

> - Even when pipelining commands, they still need to be sent, 
> the  incoming command needs to be tokenized (server), the 
> command is  performed (server), the response sent back, any 
> untagged responses are  tokenized (client), the untagged 
> responses are interpreted (client),  the tagged response is 
> tokenized (client), and the tagged response is  processed 
> (client).  None of this is "free".  Pipelining eliminates  none 
> of this.

Using the numbers you posted later on, we're speaking about parsing roughly 600 bytes of a well-structured text. For me, it's hard to believe that this has any measurable impact.

> I would argue that the language of the RFC still controls 
> despite what  an e-mail on this list says.  A client shouldn't 
> be punished for  interpreting it that way either.

The RFC is a specification crafted by humans. It has errors, and all subsequent revisions will still have errors. (See the errata for a list of those which are known already.) If you choose to block and not pipeline ENABLE QRESYNC and SELECT ... QRESYNC, you hurt your users. (Also note that the clarification given on this list was by the original authors of the RFC.)

>> As of the LANGUAGE -- how often do you expect to hit an error  
>> condition which is not described by an appropriate response 
>> code? I  don't think that blocking for its result would be a 
>> good design  choice.
>
> That could be your decision as a client author.  I would 
> vehemently disagree.
>
>> And finally, what IMAP servers support the LANGUAGE extension?
>
> Why does this matter?  RFC 5255 is a Standards Track extension. 
>  A  year from now, every IMAP server and 200 new ones may 
> support it.

I stand by my reasoning. In order for the block to be actually usefull, you'll have to talk to a server which:

1) actually implements LANGUAGE,
2) executes all commands in parallel OR has the LANGUAGE command implemented in such a slow way that it enables parallel processing for it,
3) returns a failure for one of the first commands which you send *and* does not return an appropriate response code.

But it's your client, do whatever you want to do :). I'm merely saying that adding an extension driven by the desire to eliminate issues like this is not something I support.

> It would be impossible to determine benchmarks since there is 
> no  defined protocol yet.  And, as mentioned above, any given  
> client/server interaction may provide different results based on 
> their  own internal optimizations and extension support.

Right. Well, based on how my client works, I don't expect any significant performance gains obtained through this proposal.

I'm not the standards commitee, but having decent numbers saying "see, this RESUME extensions cuts 40% out of the 1300ms required to establish an IMAP session" is something which moves the discussion from the current, very vague stage of "this is good -- nope, this is worthless" into a stage where we can actually discuss what merits it really brings. As you're proposing the extension, you should IMHO provide these numbers.

> Additional network data required by RESUME commands: 61 bytes 
> (RESUME  command, RESUME response code)
> Additional round-trips required: 1
> Network data saved by RESUME: ~650 bytes
> Round-trips saved: 3
> Server parsed commands saved: 3
> Client issued commands saved: 3
> Untagged responses that do not need to be re-parsed: 4
>
> In this example, the one-time addition of 29 bytes of network 
> traffic  (1 additional untagged response parse) results in the 
> savings of 2  round-trips, ~600 bytes of network traffic, and 3 
> additional commands  that need to be parsed on the client/server 
> side.  And remember this  doesn't factor in any initialization 
> code that needs to be run within  the server/client to perform 
> these commands.
>
> To me, that is substantial savings, especially when the 
> connection may  be re-established every 10 seconds.

I disagree with your analysis for the following reasons:

1) You don't take the initial CAPABILITY into account, but you re-request CAPABILITY after login. (You need the initial capability to see whether the server supports RESUME at all.) This will change the numbers quite a lot.
2) The sample token which Timo showed on the other list was way longer than base64("state token") you use. Just saying.
3) Saving 600 bytes of transmitted data per connection is noise compared to what an actual session typically transfers.
4) You could save even more bytes by converting IMAP to a binary protocol. That possibility in itself is, however, no reason to do so.
5) You're taking an advantage of eliminating NAMESPACE, but so far have ignored LIST and STATUS, even though a typicall client will need them as well. When the LIST responses come into account, savings of 600 bytes starts looking more and more like noise -- not mentioning the mailbox synchronization or data transfers.

As usual, I'd love to be shown where my reasoning has flaws.

With kind regards,
Jan
_______________________________________________
Imap-protocol mailing list
[email protected]
http://mailman2.u.washington.edu/mailman/listinfo/imap-protocol