Re: POP3 "resume download" extension

--On Friday, December 17, 1999 1:22 -0500 Keith Moore <[email protected]> 
wrote:
> fwiw, a lot of folks are likely to push back on defining any new extension
> to pop. there is a feeling that two mail access protocols are too many,

Let me explain my theory for why we can't get rid of POP easily.  The 
primary customer for POP servers is large ISPs providing individual user 
service.  These ISPs need to put as many mail users on a server as possible 
in order to keep management costs to a minimum so the per-user margin is 
profitable.  Most of the server operating systems which can scale large 
enough with enough reliability and performance are based on the BSD-socket 
API.  When machines get that big, two major limiting factors on scalability 
are a small fixed limit on TCP/IP connections (max file descriptors) and 
the exponential performance cost of select and poll.  In this scenario, 
TCP/IP connections are expensive and need to be kept as short lived as 
possible.  IMAP is anethema, as would be any extension to POP which might 
result in longer-running connections.

The excessively ugly /dev/poll interface in Solaris 8 (pre-release) fixes 
half the problem but that OS still has a small hard-coded limit on the 
number of file descriptors instead of making RAM the only limiting factor 
as it should.  So we have to live with protocols that use short-lived 
TCP/IP connections like POP and HTTP 1.0 (with the shared Internet 
suffering as a result).

> if you do define an extension for partial message retrieval I think it 
> should allow partial message retrieval (e.g. byte count) rather than 
> just resume.  that way, you don't have to drop a connection to 
> get the server to stop sending you a message.

I happen to support the idea of a resume download extension, but I'm 
opposed to adding a partial message retrieval extension to POP.  My reasons 
are subtle (mostly gut reaction), but I'll attempt to explain:

First, I don't believe most clients would bother to do a pipelined chunking 
download of POP mail.  It simply adds too much complexity to the common 
case (download all mail and delete from server) in order to optimize an 
uncommon case (user cancel of large message).  Simply closing the socket 
and re-opening is much less error prone, and wasting a few round-trips 
simply isn't a performance concern when you're interacting with the user. 
In IMAP, connections are usually long-lived so a pipelined chunking 
download makes more sense, but even though the facility is present in the 
protocol it's largely unused.  Furthermore, the interactions between 
dot-stuffing and byte ranges would make the POP3 version tricker to 
implement than the IMAP version.  That's far too much complexity to save a 
few round trips.

Second, I don't think POP3 is the right protocol to start experimenting 
with new user cancel facilities.  Most of the times the IETF has tried an 
explicit user cancel facility in a protocol, the result has largely failed 
(e.g., telnet TCP urgent data & IMAP pipelined/chunked partial fetch).  A 
largely stable full standard protocol is the wrong place to experiment.

Finally, I think a byte-range fetch could be abused.  In particular, some 
clients will try to use it to fetch pieces of a large message (something 
which IMAP does far better).  This would require a series of round trips to 
see if enough data has been fetched.  So if you're going to add a 
byte-range fetch, you also need to add a separate facility to at least 
fetch to the end of the first text part (possibly with a size limit as 
well) at the same time to minimize the potential for abuse.  But now the 
extension adds one or two commands and a bit of server-side MIME parsing 
which is really pushing the complexity barrier for a POP extension.

A "resume download" extension is simpler and doesn't have the above issues. 
About all it can be used for is to reduce retransmission of data after a 
user cancel or network failure -- thus making a POP client signficantly 
more likely to succeed at downloading a large message over a flaky 
connection.  That has an acceptable complexity:benefit ratio in my book, 
byte-range fetch does not.

		- Chris