Re: Comments on IM2000

>James, thanks for those carefully thought-out comments.

You're welcome!

>I should say that the document I wrote was intended originally to consider
>IM2000 and how it would interact with spam; its scope has already crept to
>include other random thoughts on IM2000. It certainly wasn't intended to be
>a design for a FUSSP :-)

Heh..."FUSSP"?

>In reply to some of the points you raise:
>
>* The trouble with Artificial Intelligence is, it isn't. Today, AI basically
>means tree-searching and pattern-matching algorithms. Maybe in 50 or 100
>years time, there will be AI which actually *understands* documents (and
>people) at the highest semantic level; until then, I wouldn't trust it to
>make any value judgements on the content of my mail. And if it really *did*
>understand my mail, I probably wouldn't want it to be reading it anyway :-)

Oh, I agree, AI is (probably) a promise that might never be fulfilled.

But there are things we're doing with today's technology *now* that
would have been thought of as "requiring" AI 20 years ago.  Google
comes to mind, and when I think of how one of the ways it "works" is
by drawing conclusions partly based on connections among trusted
relationships, I think similar approaches can be brought to bear on
the UBE problem.

(We're already using such approaches on an ad-hoc basis, in the form
of RBLs, including arbitrarily blocking dynamic-IP-hosted MTAs like
mine.)

>* It's an interesting point that the real criteria as to whether spam is
>wanted or not is based on the content. However, I think you're imagining a
>future world where a significant portion of bulk E-mail is sent from
>legitimate businesses that you might actually want to deal with.

Not really, because, my "future world" includes a vast number of
smaller, lighter-weight businesses (and other organizations, as well
as people), from which I *might* indeed want to receive information on
deals, offers, alerts (e.g. from churches, charities), and so on.

The infrastructure allows that kind of thing, except *I* don't have
time to sort through it all myself, without some kind of technological
assistance.

(The way I've handled this in the past is to stop working on various
projects.  I don't think that approach scales well to the rest of the
Internet; besides, I've pretty much run out of projects to stop
working on.  ;-)

>From my own experience, the vast majority of spam is downright fraudulent.
>Even if I saw something in a spam that interested me - taking your example,
>it was talking about saving the Brazilian rainforest - if it asked for a
>donation of $20, there's no way I'd send it. That's because (a) the spam is
>almost certainly fraudulent, difficult to trace, and the $20 would just line
>somebody's pocket; and (b) conversely, I know that any legitimate
>organisation which was active in this area, would not dare to send spam.

Right.  Remember, the most "obvious" way to know whether something is
UBE, or at least BE, is the fact that it's sent in *bulk* to many
recipients.

Therefore, recipients sharing information that "profiles" sources
and/or topics of emails can discover, among themselves, that certain
sources and/or topics are so widely involved in email, within a given
period, that they indicate bulk.  Then it's up to each recipient to
decide, based on the source, the degree to which it's unsolicited.

My concern, here, is that we keep in mind that making email systems
work *better* means, generally, we'll get even more email, and that
the overall population of email users is rising and will likely
continue to rise.  (Given the low environmental costs and expense of
email versus other forms of communication, I think that's a *good*
thing.)

IM2000 may shift some of the burden of costs onto the sender, which
may reduce the overall % of UBE, and maybe it'll do this enough to
basically eliminate spam.

But it won't make the general problem of too much incoming email and
sorting through it efficiently and easy go away -- it might even make
that problem *worse*.

So, I find myself forced to imagine a more intelligent MUA that helps
users sift through their incoming email based on more personal
criteria.  Such an MUA would obviously appreciate having more
information upon which to draw -- including message content -- in
order to prepare its presentation to its end user.  Accordingly, it
would want itself, and other MUAs, to be able to share information by
propagating it upstream and to fellow MUAs, helping all of them detect
bulk email, etc.

As MUAs necessarily become more intelligent, I think the advantages of
IM2000 as a "pull" protocol become *smaller*, not greater, over SMTP,
and the efficiency advantages of IM2000 merely as a *new* protocol
(more precisely, of *any* new protocol that could replace SMTP but
still be an SMTP-like "push" protocol, perhaps with bounces) are not
all that great, even in today's environment, IMO.

>Therefore, the fact that it's received as UBE *automatically* classifies it
>as uninteresting to me, regardless of content. And so it should for everyone
>else; unfortunately, even with a hit rate of 1 in 1000 or less, there are
>still enough born every minute to make it worth the fraudulent spammer's
>time. The other 999 spams cause huge annoyance, but the law enforcement
>agencies at present seem completely disinterested in tracking down these
>fraudsters - even when the spams include postal or telephone contact points
>which would be easy to trace or sting.

When you say "received as UBE", I assume you mean from a known
*source* of UBE, that is, not based on content analysis.

A "razor" I apply to any system (like IM2000) or improvement (like
greylisting, Challenge/Response, etc.) designed for email is to
basically "cancel" components that are equal on each side of the
equation vis-a-vis today's SMTP.

So, since SMTP already provides means to block incoming content from
known UBE sources, IM2000's ability to do the same thing really
doesn't help matters much, in terms of justifying rolling out IM2000.

And since both allow a receiving system to "tarpit" a known sender of
UBE in some fashion, driving up her costs even just a bit, I cancel
*that* out.

What SMTP offers that IM2000 typically does not, and, based on your
and other proposals, cannot, is the abillity for a receiving system to
acquire message content without doing a DNS lookup and without
initiating a connection to any remote site.

That reduces the number of points of failure by at least two (DNS
doesn't need to be working to receive a message, which actually
mattered to *me* recently when Comcast screwed up its DHCP info
advertising nameservers but I still received incoming email; and a
remote sender's ability to accept incoming connections doesn't need to
be on-demand).

That, in turn, not only makes ordinary exchange of emails between
mutually trusting sites theoretically faster (SMTP's ancient protocol
introduces some unfavorably noise here -- HELO! ;-), but it makes it
easier and less expensive for an SMTP receiver to *tarpit* a known
sender of UBE.

>* There are I think a large proportion of people who would rather see no
>advertising at all in their inbox, even if some of it might be of interest
>(e.g. special offers or discounts from legitimate companies that they might
>want to deal with)

I don't see any way to achieve this without false positives, either in
an SMTP world or an IM2000 world.  (In any case, in my view that
merely pushes the responsibility of filtering upstream, turning the
final MTA prior to the recipient's MUA into a sort of "secretary" for
that MUA, if the MUA isn't itself able to funciton in that fashion.)

For example, some people send legitimate email to mailing lists, and
they do so from free email providers that add smallish banners to
outgoing email advertising those providers' services.

That *guarantees* that, for people receiving such emails, one of three
things must happen:

  1.  The recipient will see advertising in their inbox.  ("This email
      sent by AcmeCo's Free Email Provider -- sign up today at
      freeacmemail.com!")

  2.  The recipient will not see otherwise-legitimate email that
      contains such advertising.

  3.  The recipient's "agent" will somehow "know" (here we get into AI
      again, as it's basically an arms race -- see my jcb-sc.com site
      for details, and, yes, I just "plugged" it purely for purposes
      of illustration ;-) how to remove the advertising content from
      the legitimate email, "solving" problems 1 and 2 at the same
      time.

>There's a parallel with the real world. In the UK we have this thing called
>the "Mailing Preference Service". All marketers who follow the Direct
>Marketing Association code of practice are required to pre-filter their
>mailshots and remove all MPS-registered people from it.
>
>It's remarkably effective; the amount of junk paper mail I get is now very
>low (and the credit card offers and the like that I still receive go
>straight into the bin, as I know the company does not follow the DMA code of
>practice). And when I tell someone about the MPS, almost without exception
>they are completely delighted and register immediately.

But email (and all electronically-based communication generally) is
getting further and further away from the "comfort zone" we
historically have had with tangible goods.

Consider bounces.  They made sense in a tangible-delivery context,
because if you send Aunt Emma a package containing pictures of your
newborn, and it can't be delivered for some reason, you *really* need
those pictures to be shipped back to you by the delivery agent.  That
agent simply dropping them into the trash isn't really a legitimate
option.

So, it also made sense, to the designers of SMTP and some of its
predecessors, that "electronic" mail was simply just mail done
electronically.  As there was no tracking of tangible deliveries then,
they provided none in the electronic version.  As the responsibility
for delivering tangible mail naturally has to tag along with the
content of that mail (and, pretty much or closely at least, with the
notification that the mail is available), SMTP was endowed with the
notion that responsibility for message delivery accompanied message
content, and included responsibility for subsequent notification of
delivery failure.

Now, with IM, TXT messaging, and so on, people are used to, and I
think even prefer, to not be sent *back* a whole new message saying
"message delivery failed, here's what you sent", rather, just be told
what the status of each outgoing message is.

Along these lines, the electronic world is *much* more heterogenous,
and changes much more quickly, than the world of tangible delivery,
especially the narrow example of the UK.

(As an exercise, ask yourself how useful MPS would be if international
entities could send arbitrary deliveries of any size to even entirely
made-up addresses within the UK, for zero, or nearly zero, cost.)

>I think you're right that we suffer from information overload; and
>advertising is for most people the lowest form of information. Many people
>are happy to accept the risk of losing out on an advert that might actually
>interest them, for a reduction in the total amount of advertising that they
>are bombarded with.

Agreed.  The tricks include separating the advertising from the
useful, informational email, as well as distinguishing useful targeted
advertising from mass-marketed BS.

Source-based distinctions (RBLs and the like) do a lot of this for us
now.

I don't think IM2000 offers enough, in this regard, to improve on
today, and what I *do* think it offers is the ability for us to more
flexibly respond to a more complicated, more interconnected world that
will, therefore, include each person likely being sent much more
*legitimate* email anyway.

>I do occasionally receive chain letters and lottery frauds through the post.
>However it's rare because the cost involved is that much higher than
>spamming.

That is a key insight.  If true, it gets us back almost to square one
when discussing SMTP, IM2000, the UBE problem, etc.

Your document explains why IM2000 does not, "out of the box", really
make sending BE any more expensive, and in fact might make it
substantially *less* expensive, if it is not, in deployment, wrapped
up in all sorts of extra "baggage" a la today's RBLs.

So chain letters and lottery frauds (I get lots of the latter, by the
way) become, potentially, *less* expensive to send in an IM2000 world.

And your document explains how, because messages aren't unpinned by
recipients until, based on *post-notification analyses of sources*,
they can decide whether reading and unpinning a particular message is
a worthwhile.

That implies an MUA infrastructure that include cooperation among MUAs
(which probably includes cooperation with MTAs), or their IM2000
equivalents, in that, *somehow*, a person who reads a piece of spam
needs to be able to tell their MUA that it's spam *and* that MUA needs
to somehow communicate that claim (and it's only a *claim*) to other
MUAs, presumably via intermediaries.

Yes, the underlying MTAs can "infer" that a given source is a spammer
based on discovering the overall message-sending pattern.  But they
can, and probably do, do that *today* in the SMTP world.  (This gets
back to the MTA-as-upstream-secretary approach.)

What IM2000 allows -- but only by substantially changing the MUA
interface as seen by an appropriate constant % of its user base -- is
deferring accepting *responsibility* for a message until its recipient
is (nearly) ready to actually read it.

SMTP actually *can* provide that, in a less-elegant way, via
ecrulisting (a la greylisting, except "deferred", or greylisted,
messages are generally made available to an MUA via notification a la
IM2000 and the MUA has the IM2000-like ability to tell the notifying
agent, or MTA, how to deal with future attempts to deliver the
message).

What I'm thinking of doing is implementing ecrulisting for myself and
improving my MUA (in GNU Emacs ;-) to see how such a thing would work
in practice, and perhaps offering a free SMTP server that does
ecrulisting on my web site (in source form) for others to try as well.

It's not as clean as IM2000, but it has a generally-similar profile in
terms of potential points of failure, and it would give users an
opportunity to see how well an IM2000-like MUA would work in practice,
except there would (generally) be no inherent latency-based lags in
retrieving message contents (which I think would be considered
unacceptable and thus worked around, via prefetching, in most, if not
all, IM2000 implementations anyway).

>I think a similar kind of classification could work in the on-line world:
>
>  - Bulk mail, untrusted sender
>      => probably fraudulent (= chain letters)
>  - Bulk mail, classified as legitimate advertising by trusted third party,
>    classified by subject area
>      => can choose to reject based on the fact it's advertising (= MPS)
>      => can choose to accept or reject based on subject area (= PPS)
>
>Such systems would work because you are not having to assess a level of
>trust in potentially thousands of different organisations or individuals
>trying to contact you each day, down to a handful of agencies who sign and
>classify 'legitimate' advertising.

I'm still having trouble understanding how IM2000 makes this work
*that* much better than SMTP.

>If spam is not correctly labelled, then you still have
>all the existing difficulties with SMTP of tracing it back to source.

But IM2000 doesn't really make those difficulties go away, except by
*either* dispensing with store-and-forward (which I interpret as
"IM2000 won't be widely adopted, *ever*"), or providing an out-of-band
means to represent transmission/relay information.

The latter is practical and is how SMTP probably *should* have been
designed -- that is, make message contents basically opaque as far as
the MTA is concerned, and put all the "Received:" and "Delivered-to:"
stuff into the envelope in some fashion.  (In addition to MAIL FROM
and RCPT TO, it'd include RCVD BY, DLVRD TO, whatever, corresponding
to today's mucking with the message headers.)

That way, a receiver doesn't need to snarf down an entire 10MB email
to conclude that, while the *immediate* sender might be a generally
trusted relay, it got *this* particular message from a known, or
likely, source of spam.

Even here, this becomes a tradeoff in which IM2000, or a "proper"
SMTP, helps *spammers* reduce their costs, in that *they* wouldn't
have to transmit entire messages before a receiver decides to not
accept them based on originating or intermediate IP address.

The salient "feature" of IM2000, from the point of view of blocking
based on message stores that are no longer trusted, fades once the
store-and-forward capabilities of SMTP are implemented in an IM2000
world, which they *will* be, for a variety of practical reasons.

>* While we are in a situation where on-line identities can be created at
>will, and assuming you would like to be able to receive mail from people who
>have not dealt with you before, whitelisting is not very useful for
>controlling spam. It would require a network of trust which would prevent
>individuals creating large numbers of identities at will; the identities
>would have to be bound to a personal or corporate identity (and even
>corporate identities would be weak, since companies can be created without
>too much difficulty of expense).

Yup.  When the originator is new to the receiver, the receiver
therefore *must* rely substantially on content analysis to determine
utility of email.

Therefore, as incoming email load goes up, content analysis becomes
more important to do via software assistance.

Today, we accomplish much of it by having an email infrastructure
that's so overloaded by UBE that we arbitrarily block substantial
portions of the Internet from accessing our MTAs, so our MUAs never
even see the messages.  And we do other things, including limiting
email-transmission resources overall, that make sending email more
expensive than it *needs* to be.  (All of this probably conspires to
*reduce* the amount of UBE sent, since much of it is sent based on its
attractive bang-for-buck ratio.)

That definitely has collateral damage.  I don't use any traditional
RBLs, though I do block email with envelope senders of known spammers
based on a central repository run by someone who uses whois-style
lookups to confirm that domain names and/or email addresses are indeed
"owned" by known spammers (so joe jobs do not result in innocents
being listed).  I haven't set up my SMTP server to *report* such
blocks on my logs yet, so I don't know how efficacious they are (silly
of me, but I've been busy with other things).

But I really don't end up *seeing* lots of spam, because my primitive
MTA (funnily patched qmail) dispenses with most of it via a
combination of very trivial tactics -- giving a multi-line greeting in
most cases, tarpitting multiple RCPTs, and a few other very simple
content-based things done on the fly -- which somehow conspire to
rebuke most older spam software (which can't tolerate multiline
greetings and HELO responses) and cause newer spam software to give up
sending to my site (as a result of the RCPT TO tarpitting).

(It "helps" that my qmail-smtpd is "vanilla" in that it doesn't
reject, out of hand, envelope recipients with unrecognized usernames.
Between that and the longtime "attractiveness" of my jcb-sc.com domain
to spammers, almost all incoming UBE has four or more RCPT TOs, and,
since only my wife and I are legitimate "human" recipients of email
here, almost all incoming email with more than two RCPT TOs is UBE.)

I'm considering switching to an ecrulisting-style system in order to
get a better grip on how many false positives and negatives I'm
*really* getting, since there's no way to be *sure* that, say, clients
giving up due to a multiline SMTP greeting or a minute-long delay
before each RCPT TO after the second are in fact trying to send UBE.

If I get around to this, my system might include using a catchall
address that runs a "no such user" MUA -- a concept that would likely
be useful in an IM2000 world -- that analyzes incoming (misdirected or
"evil") messages, shares information on likely spam with other MUAs
(like the one *I* use to read email), and so on, in order to elegantly
help me and my wife (or, more properly, the mail forwarder we use for
her, since she's mostly Blackberry based through her work ISP) sift
through all our incoming email.

My experiment would be designed to answer these questions: if I set up
my MTA to allow pretty much *all* email into my system, rather than
blocking it, and I relied more on my MUA to sort things out, would I
be just as happy with the results?  Have fewer false negatives and
false positives?  Feel better about forcing spammers to keep sending
me the same emails over and over again, knowing all they were doing
was wasting *their* resources?

>I don't think many people will accept that they can't send E-mail unless
>they've first had their identity checked.

Yup.  It's ultimately up to a *recipient* to decide on what basis and
priority they want to be shown any given email.  Everything else in
between involves conserving resources, avoiding resource starvation,
thus propagating recipients' selection and prioritization criteria
upstream, and so on.

>However, it might be reasonable to receive mail only from domain X only if
>the owner of domain X has got a certificate, and the existing X509
>infrastructure would be fine for that.

[...really out of my domain of expertise at that point, though I think
I get the basic picture...]

Again, I'm having difficulty seeing how a practically-deployable
IM2000 really improves upon SMTP in these areas.

>Using domain certificates is not going to help spam much anyway, if:
>- spammers send out mail from 0wned machines with upstream ISP accounts; or

This is a *big* issue.  As end-user (& computer) mobility increases,
it becomes increasingly obvious that we wouldn't count on having
IM2000 message stores sitting on originating machines any more than
today's email infrastructure assumes home PCs have IMAP mailboxes for
all potential recipients of email they send (and, after all, many of
those home PCs are blocked from sending SMTP email directly to sites
like aol.com, so they need to relay email anyway).

So, there *has* to be a widely supported means for any software
running on a home PC to send an arbitrary email "upstream" to an MTA
(SMTP server, IM2000 message store, whatever) that is more trusted and
more assuredly connected (that is, a real server, not a home PC that
can and might well be turned off as soon as a message is "sent") than
the home PC or a Blackberry.

That means 0wned machines will send such emails via that same method
anyway.  IM2000 really can't improve on this at all, that I can see,
in terms of the volume of such messages reaching a recipient's inbox.

>- spammers get lots of domains and certificates (although then new CAs
>  may spring into life, whose remit is to sign only non-spammers domains)
>
>However, it might stop spammers setting up mailstores on 0wned machines,
>which would definitely be a good thing. If you want to run your own
>mailstore, then not only do you need your own domain, but you need a
>certificate. That's quite a high hurdle to jump; it might mean we have
>fewer, larger mailstores in practice.

Again, we can, and, in some cases in practice, we do, this already
with SMTP.  I don't think assuming a trend towards fewer, larger
mailstores is wise, nor do I think *going* in that direction is wise,
because centralization of resources, or even trust, breaks down in all
sorts of ways.  Also, as we centralize trust, SMTP with all today's
add-ons (such as AUTH) becomes sufficient for the task, AFAICT.

>>   a *sender* should be able to control just how
>>   "persistent" his outgoing MUA is in terms of notifying the recipient
>>   about the message being available, checking on the status of
>>   delivery, and so on.
>> 
>>   IM2000, as presently formulated, really doesn't offer much more
>>   flexibility in this regard than does SMTP.
>
>I think in principle it does; a message in a message store is labelled with
>how long you want to keep it there, and it could also be labelled with an
>indication of how aggressively you wish to retry. There's a little of this
>sort of stuff in ESMTP DSN, but I don't think it's that widely implemented
>in MTAs, and less so in MUAs.

What I'm saying is that it isn't an *inherent* advantage of IM2000.
To take advantage of its potential requires doing a bunch of things,
maybe 95% or more of which would accomplish it for SMTP as well.

Sysadmins are always asking questions (on the qmail list anyway) about
how to get finer-grained control over lifetime of emails in their
outgoing queues, as well as over incoming message sizes, and the like.

There's little in SMTP that flat-out prevents such things, and little
in IM2000 that *magically* enables them.

IM2000, being a "new" protocol, can enable such things out of the box
with less muss and fuss.  The question is, does that offset the
*penalties* that go along with IM2000 being a "new" protocol (in terms
of adoption expense, etc.)?

>>   An advantage to giving senders a wider range of options when it
>>   comes to notifying and inquiring about outgoing emails is that it
>>   allows *recipient* systems to actually "bias" their prioritization
>>   of incoming emails based on the apparent persistence of the sender.
>
>But I don't think that any precedence or priority flag assigned by the
>sender has any value in absolute terms.
>
>I think it may have value as a [sender,priority] tuple. That is, if I
>receive a mail from [my boss,Important] then I may treat it as important.
>But mail from an unknown third party labelled as Important almost certainly
>isn't. In fact, I would probably apply a negative weighting; the more
>important the message screams that it is, the less important that it is
>likely to be. Consider counting the exclamation marks in spam subject lines
>:-)

You're absolutely correct about that.

What I mean is that, in a greylisting-type system, it is assumed (and,
often, correctly so) that a second or third attempt to send an email
lowers the probability that it's UBE.  So the first and maybe second
attempts are deferred by temporary rejection.

In an ecrulisting-type system, the first delivery attempt is deferred,
the message is delivered to the user's inbox anyway (with an
indication that it was deferred -- in fact, with a dynamic "feed"
concerning delivery attempts and other "discoveries" about the
message), and the user's MUA decides, potentially based on how many
delivery attempts were in fact made, as well as source IP of client,
other clients in the "Received:" headers, etc., how to treat the
message.

A typical spammer doesn't retry delivery of deferred messages, so
single-delivery messages could be ignored, or given very low priority
for display, by a given user's MUA.

IM2000 offers a similar potential, in that message notifications can
be used in determining how "interested" a sender really is in making
sure the message arrived, was delivered, was read, etc.

Message notifications could in fact trigger a series of increases in
priority for things like prefetching a message for an MUA -- as long
as the daemon seeing the notifications can also tell when such
notifications are being sent in bulk to a large population of
recipients (including spamtraps), in which case the daemon presumably
would decide to just chuckle and let the spammer expend resources
sending notifications, etc.

>> After all, UBE is really just a subset of UBI (Unsolicited Bulk
>> Information), in the sense that UBI includes spam posted to blog and
>> other web sites allowing posting of arbitrary anonymous content, as
>> well as to USENET.  So the fight is really not *just* in the email
>> arena, and many of the techniques used in that arena have plenty of
>> applicability elsewhere (and, presumably, vice versa).
>
>True. I think E-mail does have a special place, primarily because it's an
>on-line analogy to a physical service we're all familiar with, and because
>private one-to-one store-and-forward communication is an extremely useful
>tool.
>
>If you read someone's website or blog, then you've made an active decision
>to go there and participate. If there's advertising there, then so be it. It
>may be because the blogger put it there, or because spammers put it there
>and the blogger does not control them. If it annoys you enough, you may
>choose not to participate there. However, one's inbox is considered
>'personal space' and not to be infiltrated by advertisers, and especially
>fraudsters.

I see email as representing a subset of a continuum here, though.  If
I'm maintaining a software package, incoming patches and bug reports
are really a lot like email messages (historically, as well as today,
that's how they were transmitted).  Similarly, if I'm maintaining a
web site (not really a blog, more like my actual sites), containing
technical information that might be found to be buggy or out of date
by arbitrary readers, incoming reports are also a lot like email
messages in the sense you mean, even though they don't *have* to be
sent in the form of email messages per se.

So, I want my incoming patch/bug server to have similar capabilities,
in terms of resisting UBI, validating content, and so on, to any email
system I might be using.

That is, I don't want to give up the many add-ons a widely used system
like email offers when I get into areas involving more-specific
content such as patches, bug reports, and so on.

Though I have yet to delve into the world of wikis, I *think* they are
kinda like what I mean here.  It's the maintainer of the wiki'd site
who, theoretically, wants to receive arbitrary updates regarding the
content of the site from arbitrary people, and that's an email-like
model -- she's not going out to other peoples' sites looking for bug
reports on her own, necessarily -- but presumably the wiki system
provides communications that are tailored to allow fully automated
updates, which email, by itself, does not (unless we get into MIME
encodings for such things).

>It would be interesting to see a list of desirable features or
>characteristics of the GUIXP system you describe.

Your web page's proposal concerning folders and such really begins to
hit on such things.  But I've written so much already, and am mainly
trying to figure it all out, so my tentative "design" for such a
system is really hard to express in any useful way at this point.

The only way I can really describe the system I'm imagining is to
suggest a Unixy/shell-like system designed, from the ground up, around
the assumption that *all* resources are dynamically located in
arbitrary places on an unreliable heterogenous network.

The basic building blocks for such a system would be, accordingly,
determined by studying how existing systems and proposals work and
figuring out what the underlying requirements really are, in an
approach not too dissimilar from how, nearly 40 years ago, a few
bright people looked at huge monolithic systems and proposed designs
like IBM MVS, JCL, Multics, and PL/I, and came up with Unix and C,
which somehow managed to survive and ultimately prosper despite their
comparative "feebleness".

So, just as I explained to my manager at an IBM shop, back in 1982,
that the one full *week* we both spent studying and experimenting with
JCL in order to come up with what, in PRIMOS, was the equivalent of
"pl1 *.pl1" (that is, compile all PL/1 source files in this directory
-- in Unix, think "gcc *.c"), I'm trying to imagine what kind of
low-level networking OS would make implementing IM2000 or SMTP
*faithfully* a matter of little more than some shell-fu.

An essential component (perhaps the main component) of the system I'm
imagining would be what I call a "bidirectional shell", or "bidish",
which is kinda like a combination of low-level features of SMTP, HTTP,
FTP, and similar protocols.

It would allow client/server communications regarding data spaces on
*each* end of the communication channel.  A client might say "I have a
data stream to send you" (SMTP DATA, FTP PUT, etc.); but the server,
now knowing the stream was available on the client side, might be able
to ask questions about it or specify client-side processing ("give me
the first 100 lines" -- aka "pipe it through head -100l"; "send no
more than 1MB, feel free to compress via gzip or bzip2 if you like to
make that happen"; etc.).

So I always try to think, when I study protocols like SMTP, about just
what *semantics* underly each interaction, in order to try to discover
a "core set" of semantics that, if implemented, would enable just
about "everything", with a lot less work.

Some of this I've gotten ironed out pretty firmly, since I've been
thinking about those aspects for over 20 years.  Real-world
equivalents of other aspects are still fairly new to me, and evolving
besides, so I am not ready to design the thing yet, just to start
experimenting, even assuming I had the time!

-- 
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>