Re: IM2000 Won't Be Worth It

>James Craig Burley wrote:
>
>>     That means an IM2000 user reading his email will potentially be
>>     told about messages being available that he cannot, in fact, read.
>> 
>>     How bad is this problem?  Depends on how reliable you consider
>>     access to the WWW to be, since that's roughly a similar comparison
>>     and can be (and is) used similarly (though I could argue setting
>>     up IM2000 outgoing message stores and policing them will be
>>     harder, at least at first, than setting up a web server).
>> 
>>     Does every web page you have reason to believe exists come up
>>     quickly and easily, and as reliably as you'd expect your incoming
>>     message *contents* to come up along with their envelopes in your
>>     in-box?
>
>Ahh, but now we are down to a *performance* issue, easily solved with caching. 
>  One can easily imagine a mail intermeidary/proxy that pre-fetches the 
>content to a local cache, so that you can read it even in the face of an 
>itermittent and/or slow network interface.

Oh, sure, but this reminds me of the C/R debate I took part in a few
months ago on the qmail list.  There, C/R was being presented as "100%
the right solution", or some such wording, because it supposedly had
no false positives *or* negatives, except for Nigerian scam spam,
because that's actually sent and "babysat" by real live people.

But when I started asking questions about how it'd actually work in
real life, pointing out problems, all sorts of add-ons were being
proposed to deal with these real problems -- add-ons that were a)
reasonably useful in dealing with spam *without* adopting C/R, which
actually *does* lose legitimate email after all, and b) making a
supposedly simple system much more complicated than it at first
appeared or would be thought necessary based on the simplistic claims
of its adherents.

So now, for IM2000 and this simple "pull" concept, which revolves
around *recipients* controlling which messages, we now have to add a
cache that will, presumably, be a *system-wide* one (system ==
corporate LAN, ISP, or at least a daemon running on behalf of a user),
to which the recipients preferences must be uploaded, or propagated.

Well, that's also a useful idea for SMTP, and is being done already
via ad-hoc protocols and other such means.

And it raises the question, if IM2000 is such a slam-dunk improvement
over SMTP, why will we need all sorts of special cases to make it work
as expected/demanded by real users?

>This is however a *performance optimization on top of* the base IM2000 
>service, similarly to a squid-cache for WWW, but has some important 
>differences between both a web-cache and an IMAP/POP server:

You're right, certainly, that all sorts of useful optimizations can
and will be done on top of IM2000 in order to have it work somewhat as
well as people are accustomed to with SMTP.

>* vs a webcache, you know *beforehand* what remote resouces the user will
>   possibly want to look at, and you can *pre*-fetch them.

Only if you have the recipients' preferences propagated to you.  If
that's done for SMTP, which is not much harder, you get a lot of
benefit there as well.

Of course, if people insist on having their email scanned for spam and
vermin *before* they start reading their email, which is likely, then
about the only real difference between SMTP and IM2000 is that, with
IM2000, the message will be "pulled" shortly after notification
instead of "pushed" along with notification, and a third server (the
outgoing message server) will have to be available, in order for the
message to be retrieved and scanned.

So the main argument in favor of IM2000 seems to boil down to the idea
that scanning message content for spam and/or vermin will rarely, if
ever, be necessary, because the system will *inherently* resist them.

That's why I've raised these issues: if the system does *not*
inherently resist them, we'll end up with an IM2000 plus lots of
bodges designed to work around *its* inadequacies such that we can
send and receive email without difficulties and yet avoid wasting lots
of resources on UBM.

>* like a web-cache, such an intermediary  can be configured with preference
>   information about what senders/locations/content-types you want to pre-fetch
>   content for, and ones you don't.  So email from your boss could always be
>   pre-fetched, while those misspelled-prescription-drug-name spams might not
>   be.  Similarly messages from sites to which you have good network
>   connections could be uncached, while messages from difficult to reach sites
>   would be.

Again, the preferences have to be propagated to make this work, *or*
messages will likely get fetched multiple times by multiple recipients
on your LAN, but across more-expensive boundaries than with SMTP in
its ordinary state (assuming multi-RCPT, which qmail accepts but does
not normally deliver).

>* the cache is not actually neccesary if you have a suitably good network
>   connection.

But the *sender's originating message store* must *also* have a
suitably good network connection, for each sender whose email you want
to read, at that moment.

There are two implications here, the first I've stated before, the
second which is new (to me, anyway) and an important one to consider:

  -  If the sender's message store is unavailable to *you* at the time you
     want, you can't read any email he sent you

  -  If the sender's message store is unavailable to *him* at the time he
     wants to *send* a message, he can't send any messages to anyone!

That latter case is important for roaming users, because it means they
must always be able to access an outgoing message store, in addition
to a recipient notification agent, to send email to *anyone* via
IM2000.

Whereas, with SMTP, as long as they can reach an SMTP server that
accepts email for the recipient, they can send email to that person.

IM2000 thus introduces another point of failure to both sending and
reading emails, over and above vanilla SMTP.

>* vs IMAP/POP/sendmail, someone sending 300 (3000?) copies of the same message
>   yeilds only one copy in the cache.  While some IMAP/POP servers implement
>   this in some cases, this is much more reliable.
>
>Now you might say that an IM2000 system with a pre-fetch cache is right back 
>to SMTP/IMAP, only more complicated.  But the advantage is the fact that 
>caching any particular message is *optional*, and a cache needs much 
>weaker/cheaper resources:

It's not "optional" if people find that the inevitable delays they
experience while reading their incoming emails -- even if they're only
about half a second -- are intolerable.

And once you resort to caching on a sufficiently widespread basis, you
defeat a substantial argument for IM2000 that is getting repeated
during this discussion, namely, reduction of overall bandwidth used to
transport email.

>* you don't need to back up the cache (if it dies, refresh it, the content is
>   still out there...) whereas backing up SMTP/POP servers well is hard.

More precisely, until your cache or some other local entity accepts
*responsibility* for an IM2000 message, there's no need to back it up.

That being said, you are also depending on your *senders* to not only
reliably back up *your* incoming email, but to be willing, ready, and
able to *restore* it if it gets lost!

Now that's something I hadn't thought of before.  If "the network is
the in-box", which is a central tenet of IM2000, and a handful of
messages from a sender get lost or deleted because you haven't "saved"
them, even though you intended to, what procedures do you have to go
through to get them restored?

With a *local* filestore, one goes through one's *local* sysadmin,
which presumably, though it may be a PITA, at least has the benefit
that the sysadmin could conceivably have some reason to care about you
as a user.

With a *remote* filestore, you've got to somehow know how to
communicate with a sysadmin (who might not speak your language), tell
him you had some really important messages sent to you via IM2000 that
were sitting in his outgoing message store, giving him their IDs or
whatever, and hope he cares enough about *you* -- when you are *not*
in any way, shape, or form a paying customer -- to take time out of
his busy schedule (learning how to be a BOFH) in order to restore what
is essentially *your* email.

This is just another aspect of my argument against the whole "the
network is the in-box" idea, but it seems like an excellent
counterpoint to your (worthwhile) point that backing up a local cache
is easy or not even necessary compared to backing up a local
mailstore.

>* you can pick/choose what mail to cache and what not to

Again, only via propagation of criteria from the recipient to the
cache, which has advantages for SMTP as well.

>* you can multilevel-cache and have a local cache on your laptop (say) to
>   provide an "offline" mode.

Yes.

Note that all this message-content caching necessarily implies that
IM2000 *accommodate* it (and probably content relaying, since they're
similar enough) in its design and implementation, so any filtering
that isn't done by an upstream cache can be applied downstream, e.g.
by the recipient's own choices of filters.

So, information such as the original outgoing message store used to
hold the message must be easily obtainable for a message notification
(or envelope or whatever), without having to look at the message
*content*, via vanilla IM2000 protocols, as I've said before.

I think that'd be a nice improvement over SMTP right there, but at
this point it sure is becoming a lot like SMTP in that there are lots
of bodges being proposed to make it work as people will expect it to
-- and it hasn't even been deployed!

Are we having fun yet?

-- 
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>