New and better email

Ok, let me dive right in:

I think it's important to recognize that email is many things to many 
people. For instance, until fairly recently I never used instant 
messaging and I still don't use it very much: most people I know see 
their email within minutes so emailing is easier. This kind of use 
needs a light-weight form of email: not too many gimmicks, just fast 
delivery. But there are also other ways to use email: to engage in 
transactions, to distribute documents that may be considered an 
integral part of an email message (ie, formatted text, images) or to 
distribute files for which the email system can only function as a 
transmission mechanism.

So there needs to be a lot of flexibility in a new email system: it 
should probably be heavier than today's where appropriate (provide 
non-repudiation for transactions) but at the same time light wait 
interactions should be possible. (I hate the fact that the headers are 
typically longer than the body for small messages these days.)

Next order of business: separation of data and signalling. SMTP suffers 
from a problem that's very similar to what we see in telephony: the 
data follows the path that depends on where the decisions are made. For 
email this makes sense historically because it is a store and forward 
network: first determining the path through the network and then 
actually following this path would slow down the flow of mail way too 
much and not lead to better or cheaper paths. Originally, I wanted to 
write "ditch store and forward", but in some situations store and 
forward is useful. However, even if a message has to travel through 
several hops, this doesn't have to take all that much time: think in 
the order of minutes. This means that for many messages it is 
advantageous to first send the header and maybe a kind of "cover sheet" 
and keep the acutal content at the source or the next hop after the 
source. Then when the destination decides that it want the content and 
how it wants it, it goes back to the source (or close) and do the 
actual content transfer. In many situations this could happen using 
direct communication, but it could also be done using store and 
forward. (Note that to some degree email already works somewhat like 
this as embeded images are often retrieved from an HTTP server.

(The cover sheet could be a small text message that indicates the 
content of the "real" message, which would be a (large) file or some 
such. Obviously doing all of this for a 1k text message would be 
overkill most of the time.)

Then there is the whole issue of large binary files, which alone would 
be worthy of a new protocol. It's ridiculous that we have to use a hack 
like base64.

Authentication is also important, but it's not obvious how this should 
work.

What we also need is support for automated interactions. For instance, 
my spam filtering is fairly ok but not good enough to trust so I have 
to scan my junk mailbox manually for false positives. It would be great 
if I could instruct my system that messages that look like spam aren't 
accepted until the sender performs some kind of action, think "type the 
number in this graphic". Another action would be "send through another 
server" (redirect through smarter anti-spam service) or "retry in 3 
hours" so I get to check with a few others whether they also see this 
sender with a similar message.

Note that the more specific actions are meant as examples of how a more 
general mechanism could be useful, wether each of these examples is 
useful in its own right isn't something I want to get into too deep.

Iljitsch van Beijnum