Re: Inexpensive anti-spamware/anti-verminware tactic

>JCB> I'm still on the fence about im2000 though.
>
>You shouldn't be.  It has been separately re-invented in various places on
>Usenet no less than three times in the past fortnight alone.  (And those are
>just the places where someone else has replied saying "You've just re-invented
>IM2000.")  It's an idea whose time has come.

I agree with everything in that paragraph except the first sentence
and the implication that, just because lots of people think it's a
good idea, I should therefore accept it.

Let's say, hypothetically, I have substantial resources ($$, time,
expertise, willing and obedient slaves, whatever ;-) available to
throw at whatever project(s) I decide are most beneficial to the world
community -- the "Internet" being my main focus in this context.

Rather than throw substantial weight behind whatever is the latest,
greatest idea that "just about everyone" believes is what is needed to
solve certain problems, I'll investigate and decide for myself.

After all, the landscape is littered with the graves of all sorts of
ideas "whose time has come" in the minds of many who have separately
re-invented them.

Heck, your anti-anti-UBM page is a testament to exactly that
understanding: *many* people have independently come up with all sorts
of anti-UBM measures (C/R and SPF are two IMO noxious ones that come
to mind).  But you (IMO correctly) rebuke the *measures* without being
unduly influenced by the sheer weight and enthusiasm of their
proponents.

Please allow me the same opportunity to decide for myself regarding
IM2000.

>JCB> But djb, among others, already "admit" that it requires external
>JCB> efforts (law enforcement, mainly) to truly "work as advertised".
>
>I doubt that he's said that.  I haven't read anything where he has said that. 
>And, in any case, it isn't even true as far as I can see.

I googled and googled, and came up empty.  Then, at dinner, I realized
I was probably thinking of statements *other* people have made about
what it'd take to make (their cherished) Challenge/Response systems --
not IM2000 -- fulfill their promise.

Apologies for confusing the two issues.

>JCB> And when I imagine how it'd actually be deployed, over the same
>JCB> breadth and depth of audience as present SMTP email has, it 
>JCB> seems to me like it'd end up having pretty much the same
>JCB> weaknesses as SMTP has today, or will have as SMTP is
>JCB> incrementally improved.
>
>No, it won't.  The whole point of IM2000 is that by design it does not have
>one, particular, very basic flaw of SMTP-based Internet mail.

And that flaw is?  That is, put it into your own words, as clearly and
succinctly as possible.  (My best attempt at it would be "IM2000 by
design does not deliver the contents of a message along with the
envelope", but I'm concerned that if I just assumed that was the right
thing to say and then expressed concerns about that model's viability,
I'd be fairly accused of attacking a straw man.)

>The notion that
>you are buying into, that SMTP will improve in the future and will end up just
>as good as IM2000, is a false one.

Whatever I'm "buying into" is irrelevant in this thread, since I'm not
in any way claiming that my own beliefs should be considered in
weighing the viability of IM2000.

I'm simply asking questions at this point.  Yes, I happen to have a
substantial amount of technical expertise and, especially, a long
track record of thinking about and designing systems around issues
such as these; but right now my focus isn't on the nuts and bolts of
IM2000, rather, how might I advocate its deployment (and the
substantial costs of same) to someone like my wife, who is *not*
highly technical, but (at least in her past) runs an IT department and
therefore is likely to have some very insightful, incisive questions
regarding exactly what IM2000 buys us that SMTP not only *doesn't*,
but *can't*, regardless of how it's incrementally improved as an
ecosystem.

>The two have a very fundamental
>architectural difference, which happens to be the nub of the problem.  No
>amount of "improvement" to the SMTP-based system will change that fundamental
>difference.

Sampling the spam my system gets, I notice some that are nothing but one
of the following:

  -  URLs to web sites with text advertising a product

  -  URLs to web sites with only graphics (e.g. JPEGs) advertising a
     product

  -  MIME-encoded graphics (JPEGs) advertising a product

How does the fundamental architectural difference between IM2000 and
SMTP address UCE of those sorts?  In the third case, the payload is
embedded, so that seems to be more directly addressed by IM2000's
distinguishing qualities.

But in the first two cases, the sender is *already* paying for
resources to store and serve the commercial content: the web sites.

IM2000 (as would almost any new protocol, if well-thought-out) will
make *receipt* of such UCE much less expensive, which is a definite
win.

But it will also make *delivery* of such UCE much less expensive,
correct?  Because, in place of URLs to web sites, there'll just be
pointers into messages stores, with a mechanism to retrieve the
messages that I would think would be lower-overhead than vanilla HTTP
as its used today by spammers.

>Moreover, it is unlikely that SMTP-based Internet mail will improve.  Most of
>the changes to it in recent years have (in the long run) damaged, rather than
>improved, it.  I don't see that trend reversing.  Indeed, given how many
>people have recently expressed the conclusion that changing SMTP won't solve
>the problem, it seems probable that SMTP won't change much at all, let alone
>change for the better, in the future.

Well, that's partly due to coming to a conclusion, at which I have not
personally arrived.  If I believed IM2000 was the One True Way, I
wouldn't put much effort into improving SMTP either.

My concern is, what if IM2000 turns out to be a wasted opportunity to
replace Internet email protocols wholesale, in that it ends up not
really doing much to combat abuse, costs $$ to deploy and replace SMTP
on a sufficiently widespread basis, and also fails to offer sufficient
additional flexibility, features, etc. to have really been worth all
the work?

If enough people decide *against* IM2000 deployment, though, more
enthusiasm for incrementally improving SMTP as a protocol (which I'm
not enthusiastic about) or as an ecosystem (which I think still has
some promise, and is in fact happening anyway, as you point out,
though not *all* changes are necessarily in the direction of damaging
it, IMO) could result in SMTP actually being improved after all.

And, especially if enough people gain a clearer understanding of the
problem and how to solve it, come to a *useful* consensus, and work
together, they might decide that IM2000 is not really the One True
Way, rather, another form of message delivery that an improved
SMTP-like ecosystem should offer, in addition to many others.

>JCB> In particular, a "pull" paradigm is not, from a resource point 
>JCB> of view, all that much different from using Greylisting, Defer
>JCB> Hostility, or simply not having enough SMTP server resources to
>JCB> reliably and quickly accept all incoming email, with today's 
>JCB> SMTP infrastructure.
>
>You have a very restricted definition of "from a resource point of view".  You
>are taking it to mean only network service availability (because that's the
>only respect in which what you describe is actually similar to a "pull
>delivery" system).

I don't see how I'm doing that.  An SMTP server that defers
questionable materials N times forces the SMTP client to *store* that
message for a longer period of time, for example.  (Yet the receiving
system *could* still notify the recipient of the attempted delivery,
and even include the message contents with that delivery.  What it
could *not* do is directly notify the sender that it has accepted or
refused delivery; it'd have to wait for a subsequent delivery attempt.
I wonder how important that difference is, in practice?)

That storage is certainly within my concept of a "resource" that is
consumed by not being able to convince another server to accept
responsibility for a message, which is *roughly* shared with IM2000's
architecture.

>But there's far more to a resource point of view than
>that.  For example:  Storage is a resource, too, and a "pull delivery" system
>has significant effects on the distribution of storage costs, causing them to
>be _very_ different to the case of SMTP-plus-those-bodges.

Yes, and let me say right off the bat that I *am* contrasting IM2000,
which is a wonderfully (theoretically, anyway ;-) new, clean system,
with a hypothetically SMTP extended with -- frankly -- bodges
*designed* to shift some of the burdens and costs of message exchange
from the recipient system(s) to the sending system(s).

The key word here is "bodges".  I'm admitting SMTP would have to have
them, and some people already implement them.  (C/R is such a bodge,
as is Greylisting, as would be Defer Hostility, if I fleshed it out
and implemented it for anyone.)

But these bodges have a big advantage over IM2000: they can be (and
some are) implemented today, piecemeal, and they pretty much
automatically interoperate with the vast majority of *today's* email,
and the SMTP clients that send it, that the users of those bodges (via
their choice of ISP) *choose* to receive.

As to the issue of storage, which you properly raise: looking at the
big picture, I'm less enamoured with that as a *critical* resource.

After all, storage costs have been falling.  Think in terms of the
rate of growth of the pertinent population (world population, maybe
Internet user base, maybe just Internet user base with sufficient
disposable income to be worth reaching, whatever you want) versus the
constant-dollars cost of storing 1MiB for each person in that
population, multiplied by the population size.

This become "cost to store messages for target audience of [spammers,
vermin authors, whatever you think IM2000 fights better than can
SMTP]".

Now, is *that* storage cost increasing, declining, or remaining
steady?

My *assumption* is that, despite reasonably rapid growth in the
Internet-using population (compared to population growth itself), the
*rate of reduction* in storage costs is outstripping it.

If that's true, the amount of storage needed by spammers to reach
their target audience costs *less* over time.  (The target audience
also has an increasing amount of $$ to spend, as a whole, since it
increases.  Therefore UCE becomes, on the whole, more profitable to
send, though it has its own competitive pressures.)

So, if IM2000's main distinguishing quality is that it puts more of
the burden of storing messages onto senders than does SMTP, the
problem it has is that it might have been a great anti-UBM protocol in
1995, an adequate one in 2000, but, maybe by 2005, 2010, sometime in
the near future, it might have become hardly distinguishable from SMTP
in terms of making a useful distinction in costs to send bulk email to
an intended audience.

Of course, storage isn't the *only* burden IM2000 shifts onto senders.
They must run their *own* servers, which has a burden as well, or
intersperse their materials in with legitimate ones.

But that leads me into other territories that strike me as making it
*less* usefully distinct from SMTP, which I won't explore in this
longish email, pending some answers that tell me whether it's
worthwhile (mainly, whether I'm on the right track at all in
understanding IM2000).

>Moreover, you are falling into the trap, that people fall into time and again,
>of addressing the wrong problem.  The problem isn't denial of network
>service.  It's unsolicited bulk mail.  Denial of service is a completely
>different (and, as far as I can tell, unavoidable) problem.

Hold on a second!  *Today's* problem *includes* denial of network
service.  Lots of email is delayed or dropped because servers are
overwhelmed coping with UBE (mostly UCE plus vermin, I think) and the
various tactics (such as expensive scanning) they employ to combat
UBE.

If IM2000 addresses only UBE, then *sufficient* similarities between
IM2000 as actually widely deployed and SMTP as deployed, such that
IM2000 fails to *eliminate* UBE, would result in denial of network
service in a future IM2000 world, just as we have today.

Or am I missing something?

Put another way: if "IM2000 does not directly incorporate direct
recognition of unsolicited and bulk qualities", as your web page says,
then UBE continues to exist, possibly thrive, in an IM2000 world,
since IM2000 cannot, itself, *recognize* it.

Further, if "multiple copies of messages, even of messages sent to
many recipients, do not actually exist in the first place until the
recipients actually request the senders to make them", as your web
page says, under IM2000, then the *cost* to send UBE to bazillions of
people in the hopes that *enough* of those people read them and buy
products might actually be *lower* than it is today.

But that's (somewhat) besides my point (as you quoted above), which
was: is IM2000 different *enough* from employing Defer Hostility or
similar SMTP-based, incremental measures, on an ad-hoc basis, in terms
of resource utilization, to make replacement of SMTP with IM2000
worthwhile?

Looking over the IM2000 literature I've seen, and thinking through as
many possible implementations as I can, I *cannot* answer "yes" to
that question, even though I'd *love* to do so.

So, would I prefer we were running some form of IM2000 *today*, over
SMTP?  Definitely.  I think it's much better, ignoring deployment
issues, than what we have.  But I think we could do even *better* than
IM2000, without designing a system much more complicated than it; so,
ignoring deployment costs, why not design a better system, and, taking
into account deployment costs, given that IM2000 is not yet deployed,
why not deploy that better system instead of wasting time deploying
IM2000?  (We're not wasting time *discussing* IM2000, though.)

>JCB> So if we accept (as I have come to do) that anti-UBM tactics 
>JCB> that don't precisely target spam and vermin won't "work" in 
>JCB> the long run, despite my initial (and ongoing *theoretical*)
>JCB> enthusiasm over im2000, I don't quite understand how it is
>JCB> anything other than another anti-UBM tactic dressed up as a 
>JCB> whole new protocol.
>
>Then I suggest that you go back to the web page whose URL sparked this
>discussion, and read the whole of it, right down to the bottom.  

Done.  I'm still of the opinion that a) IM2000 is much better than
SMTP in most ways but b) not necessarily in enough ways to make it
worth deploying, given how it'll be used *in practice*.

>JCB> But im2000 would not necessarily be any more flexible [in
>JCB> coping with spam and vermin] than improved SMTP, though it 
>JCB> would offer lots less baggage, thankfully.
>
>Again, you are buying into the very dubious notion that an "improved SMTP"
>will ever exist.  No-one has yet come up with one, and as such you cannot say
>how flexible it might be relative to IM2000.

Again, that might be a sort of semantic quibble.  (Suppose we renamed
IM2000 to "improved SMTP" in order to convince upper-management types
to accept it.  Okay, not very convincing, but remember the days back
when the "programming language of the future...will be called FORTRAN"
and Digital Equipment Corp. supposedly didn't manufacture "computers",
rather, "programmed data processors", because the purchase of
"computers" by certain organizations required additional loops to jump
through?)

It's important to distinguish, as I do, between improving SMTP as a
*protocol*, and improving the SMTP *concept* as an *ecosystem*.  Most
anti-UBM tactics do the latter; the former is something different.

When I talk about improving SMTP, I usually mean as an *ecosystem*,
with potential for improving SMTP as a protocol, though strictly
speaking that could come in the form of offering new protocols (in
that sense, QMQP, or do I mean QMTP, is a new protocol that is really
part of the SMTP "family", in terms of the overall system into which
it plugs, which includes bounce messages, conjoined envelopes and
message content, and other things).

>Moreover, there are some very interesting possibilities that IM2000 yields in
>terms of behaviour modification through ostracism, as a matter of fact.

I'm sure that's true.  I'm still trying to understand the parameters
of such ostracism for IM2000, vis-a-vis SMTP and its potential
(however limited), so I can gauge how worthwhile an IM2000 rollout
might be.

(Or do you deny that ostracism happens in today's SMTP-based
ecosystem, or that at least some of it is valid and useful?)

>JCB> Surely if only a few "enlightened individuals" use im2000, [...]
>
>From following the news coverage second-hand, I speculate that if an ISP were
>to come out with IM2000 this year, popular sentiment is such that it will be
>overwhelmed - not with mail but with customers.  (-:

Hard to say.  ISP's might be more interested in SPF, for various
reasons, some of them probably short-sighted.

Meanwhile, those enlightened individuals could get a pretty big bang
for the buck by simply switching to Any Protocol Other Than SMTP
(which I'll call APOTSMTP for fun ;-) in order to communicate with
each other.  That is basically why I posted originally: to explore
*that* as the germ of an idea to combat UBM.

>It has some good selling points that can be condensed into marketing slogans. 
>"You pay for the size of your 'out' box, not for the size of your 'in' box."
>is one such, as I mentioned earlier, that would appeal to just about
>everybody.

I agree.  In *practice*, I'm not sure that the in-box issue will make
enough difference.  Well, for vanilla ISP end-users -- and my lack of
experience with that mind-set surely contributes to my reticence
vis-a-vis IM2000 -- I can imagine that being reassured that ignoring
their in-box for a week will *not* mean they're being "charged" ($$,
or against quota limits) for all the messages piling up will be *very*
persuasive.

But ISPs *could* offer something like that today, without IM2000, via
SMTP.  Though it'd be a bodge, given the relative popularity of the
two technologies, it'd be a *more* reliable way to continue receiving
and sending email while maintaining those promises.

(After all, ISP's can hardly promise anything *useful* regarding
IM2000 in-box sizes unless their users also promise to not sign up to
receive in-bound *SMTP* email...correct?  So, for the promises to be
worthwhile, promisees have to agree to stop receiving SMTP email.
Well, then what's wrong with accepting *some* SMTP email that's been
"filtered" through something like Defer Hostility?)

>"No more 'bounce' messages." is, of course, another, also with
>wide appeal.

That's the one that makes me happiest of all, I think.  Well, it's
close to a few others, but the SMTP bounce mechanism is a *huge* waste
of resources -- almost always an overreaction to any problem.  (Again,
that's not just a *protocol* problem, it's an *ecosystem* problem, in
that the whole concept of SMTP email presently requires bounces...but
it doesn't *have* to in future incarnations, which might be more
easily transitioned to than would be IM2000.)

>JCB> (And, let's face it, im2000 *would* be at least a partial 
>JCB> retreat against a victorious enemy.)
>
>No, it isn't.  Indeed, quite the converse.  Adopting IM2000 is the
>neutralisation of an enemy by completely eliminating the game that is biased
>in his/her favour.

Well, here's where I think we might disagree pretty strongly.

In my view, one of the big advantages of SMTP over IM2000, if I
understand IM2000 correctly, is that, upon waking up and commuting to
my office (down two flights of stairs) and firing up GNU Emacs "rmail"
to read my email, I *am* able to read and manipulate each and every
email, message included, in my in-box, regardless of whether any part
of the network (beyond my own little LAN) is down, because of the
design of SMTP.

With IM2000, I'm potentially offered the tantalizing "You Have IM2000
Mail From [someone spectacular]", and when I go to actually *read*
that email, I'm told "the network is down" or "the MSRAP is
unavailable".  (Which it might be, if it's the target of a dDOS by
spammers unhappy with messages stores that successfully ban their
garbage.)

Now, presumably I can configure my system to automatically retrieve
messages as notifications are received, e.g. overnight.

But that defeats some of the advantages of IM2000 as described on your
web page (e.g. I'm now paying for the size of my in-box, and some
daemon on my system has to run as root, in order to store message
contents in arbitrary users' in-boxes so they can be assured of those
message's availability).

(Further, that raises the question, for these
automatically-transmitted messages, would the MSRAP be notified that
it no longer has responsibility for such messages?  If so, we're
substantially back to SMTP; if not, even *legitimate* senders of email
end up storing outgoing messages that are already stored in the
recipients' in-boxes, until those recipients somehow think to tell
their MUAs that the email is legit and, so, its sender should be
"freed" from having to continue storing it.)

So, unless I'm wrong about the above scenario, especially the first
part of it, what we're retreating from, by switching to IM2000, is the
assurance of being able to *read* messages whose notifications are
sitting in our in-boxes.

In general, IM2000 seems a bit like implementing, for that subset of
computing activities known as "electronic mail", the enticing concept
of "network terminals".  So computers in households don't really
receive or store email, they just tie into a central mainframe (which
is now called the Internet, and has an arbitrary, fairly loose
arrangement of Message Stores) that takes care of that for them.

Network terminals have indeed been enticing, and they offer many
advantages, but without nearly-100% ubiquitous high-bandwidth
internetworking of the entire Internet, it'll be hard to convince
*lots* of people to leave their incoming email scattered all over the
Internet instead of in their own in-boxes, safe and warm.

Hence, network terminals haven't really caught on, AFAIK, in the
pertinent context (of connecting, and doing substantial portions of
one's work, including saving it, on a computer that is upstream via
the "hostile Internet").

People still generally insist on installing software on their local
computers, saving their data on local disks, and, at least for now,
storing their incoming email there as well.

I believe many will be more appreciative of IM2000's provision for
them to be able to store *outgoing* email, not just in the sense that
they already archive it, but such that its delivery progress is
monitored locally, and if something goes wrong, their local computer
can retransmit it, without the assumption being in place that they
have to wait for some MTA out there to decide to compose and
successfully deliver a bounce (a *broken* mechanism today, thanks
partly to anti-UBM tactics) and pray that it includes the entire
original message, in case they don't have it or can't easily identify
to which message it pertains.

And, certainly, that describes *me*; I want IM2000 for its *outgoing*
email delivery characteristics, but I want my in-box to be ready,
willing, and able, whenever *I* am.  So IM2000 doesn't really help me
all that much via its architectural differences from SMTP when it
comes to making my *incoming* email less "friendly" to spammers;
though, if enough people use it differently from me and the result is
less UBM overall, *that* might help me.  (But SMTP-ecosystem
improvements can have the same effects; they already do, and I can't
always tell which anti-UBM measures are being used elsewhere to help
me see less UBM and related bounces, but it's happening nonetheless.)

I could go on, but if I'm on the entirely wrong track, and somehow
missing something important, that'd be pointless.

Besides, I actually posted my first query to the qmail list for a
reason, specifically, I have a "completely different" idea that would
combine some of the immediate, obvious advantages of switching to
IM2000 (as you personally appear to have done, insofar as your web
page seems to suggest you don't generally read, or accept, arbitrary
incoming email via SMTP) with those of maintaining some degree of
compatability with SMTP as a deployed system (though not as a specific
protocol).

And it's be pretty easy to adapt qmail to implement my "different
idea" in its early forms, such that you'd be personally able, and
perhaps even willing, to receive email via the SMTP "ecosystem",
though not via the exact SMTP protocol.

But I'd rather not get into that without first coming to some
conclusion regarding the cost/benefit ratio of IM2000 vis-a-vis
today's SMTP.  And it's not really an IM2000 or even a qmail issue,
more of an overarching "email protocols" issue, and I don't know of
any mailing lists that discuss that as a topic.

-- 
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>