Re: Requirements for a filtering header

I thought I'd try to summarize the points discussed in various threads
so far. Obviously, there will be things people don't agree with, so
jump in at that point. I'm obviously trying to present this in a
unified manner.

> 1. If there is a rough consensus that a standard filtering header would
> be useful.

We're reading this list because that's our belief. A better question
I think is what kind of information does a standard filter header(s)
need to convey?

Putting aside the actual syntax to be decided later, the following
list of tasks is required from headers, as gleaned from the comments
on this list:

1) filter identification (name, version, network location)
2) one or more scores per filter (the scores must therefore be labeled)
   Moreover, scores have no standard syntactical form or meaning, so
   are really just strings.
3) external network query results
4) header authentication and message markings (such as a hash used to
   keep meta data about the message for internal filter use)
5) user convenience


There are some rough relationships between the different tasks. I can
think of the following:

5) putting tags in a subject line is a user convenience, as is putting
   a link to a web interface for training (popfile etc)
4) marking messages is used by some statistical filters so they don't
   learn a message twice, or can verify that a message has already been
   learned/scored in case the user wants to undo an operation.

Both the above types of header tags are better characterized as
user/filter communication, whereas the remaining types of tags 1)-3) are
better characterized as summary of filtering operations.

However, score results 2) can sometimes be intended for human eyes,
when presented via ascii art. So scoring straddles the
characterizations.

The summary headers 1)-3) can contain an extremely broad variety of
things. For scoring, there's not just the actual point value, but
possibly a confidence level. There are two directions we can go in:

- We can agree that scores are freeform strings, which poses the least
  problems but makes interoperability a pain. How does the MDA or MUA
  know how to parse the score, and use the results?

- We can propose a set of standard scoring ranges, which MDA and MUA
  can in future be programmed to interpret, but we all have to agree
  that every score can be represented in those ranges.
  For a basic score, there's three obvious ranges:

  a) an integer between 1 and n
  b) a real number between 0.0 and 1.0
  c) an enumerated list element ("junk", "virus", etc)

  Beyond the three basic types, a confidence can also have one of
  these types, which unfortunately gives 3x3 = 9 combinations.

The external network query results 3) pose a novel issue, I think.
Unlike all the other things a filter may do, those are not repeatable,
nor are they consistent over time. Rather, they depend on the exact
state of the external service (eg RBL) at the moment the filter is
invoked. This is why I split this class off the general scoring
headers in 2). I feel these types of scores must be treated specially,
but we haven't discussed this on the list, so I'd like to hear some
views. Alternatively, 3) and 2) can be considered more or less the same.

The class 1) contains meta data, useful for diagnostic and trust
purposes. Trust is always an underlying issue, and there are several
approaches we've discussed, which I'll summarize later. I just want to
note one thing that seems important: the information in 1) cannot be
split off from the other information in a header. What I mean is that
something like this:

Processed-By: superfilter, v.78.9 with extra power
Processed-Score: 0.65, unsure

is a bad idea. The strongest rule in mail processing is that header
contents are treated as a unit, but individual headers can be mixed,
matched or removed by unscrupulous agents. If we have a separate
header for class 1), we're inviting unnecessary trouble.

> 2. If there is a rough consensus as to what issues need to be solved for
> such header to be useful.

An important one would be a discussion of the form of the scores in
2). I can't recall this being discussed on the list, yet.

I'd like to spend the rest of this email summarizing the trust issues
we've discussed. Obviously, my bias will show through but it'll be
easier for people to jump in here rather than in past threads.

First of all, why does trust matter?

Trust here is that thing which enables an agent to look at the
contents of a message and perform an action consistent with what is
seen within the message.

This is not at all the same kind of trust that the SMTP verification
people are talking about, I think, although there can be overlap.

Maybe a better word is "content trust". A filter deals with all messages
presented to it, and if such messages contain meta information (ie
header tags), then this meta information must be dealt with. It can be
ignored, but ignoring is just another way of dealing with it.

The typical MUA has to "content trust" the messages in its inbox. When
such a message is marked spam, the MUA has to decide if the tag is
genuine or not, then act on it. Even if it ignores the tag, why does
it ignore this tag? Would it ignore all tags, or are some special?
If we dig deep enough, the MUA has to justify the way it handles the
tag, and that's where the "content trust" issue appears.

Of course, the reason the MUA content trust issue is hard is because
spammers have learned to fake tags. I'll talk in terms of MUAs, but
other downstream agents have similar issues.

From the discussions on this list, we've come up with several partial
solutions, which I'll try to list exhaustively with pros and cons.


1) The simplest solution is to not content trust a single tag. The MUA simply
   ignores all header tags on principle.

pros: trivial to implement, there are no content trust issues
cons: all spam messages are treated like legitimate mail

2) Another solution is to trust only "bad" tags. A "bad" tag is a tag
   which says to the MUA, "don't display me".

pros: all bad tags are legitimate, and can be content trusted. Why?
      If spammers insert fake bad tags, they might as well not send
      the message at all.
cons: all non-bad tags must be ignored, by definition. This limits the
      options for filter scores. Moreover, bad tags must be absolutely
      bad. If a filter marks a message "bad, but I could be wrong",
      then a spammer can do the same, and the MUA will present the
      message to the user for visual verification. Pretty soon,
      spammers will mark all their messages "bad, but I could be wrong".

3) The MUA keeps an internal list of content trustable filters. Each
   such filter is guaranteed to look at the message before final
   delivery, and each such filter makes sure to only leave within
   the message those tag instances it personally wrote.

pros: when the MUA sees a tag from a filter on its list, it knows the
      tag is genuine. Unlike 2), there is no constraint on what such a
      tag can contain.
cons: the internal list of filters must be kept in sync with reality.
      When reality changes, the user or administrator must personally
      update the internal list, rather than some automatic system
      which can distinguish good filters from bad. ("personally" could
      mean a script written with the organization's specific
      architecture in mind, but the script must also be kept up to
      date). This doesn't scale very well.

4) A general form of 3) replaces the list of content trustable filters
   with an authentication mechanism. Filters write their tags together
   with an authentication code (I'll refer to this as a "password" for
   simplicity). When the MUA sees the message, it looks at each tag's
   supplied password, and if the password checks out, the tag is
   trusted.

pros: this scales better than 3), and is a general and extensible solution.
cons: the password must be managed, which involves a complex
      authentication infrastructure behind the scenes. There could be
      a central authentication server in the organization, which would
      verify passwords on the fly, etc. This is a heavyweight
      solution, and like 3) it requires maintenance over time.

5) Another solution is to ask filters to prove their location in the
   delivery chain. If the filter can prove it lives within a trusted
   organization, then its tags are trusted by the MUA. It proves this
   by showing a piece of the email which can only exist *after* the mail
   passed the organization's SMTP boundary (e.g. time stamp on the
   Received: header)

pros: this scales extremely well, and requires no authentication
      infrastructure because the last Received: header cannot be
      forged. A MUA can simply look at the message and decide
      immediately what it can trust.
cons: only the last step in the MTA chain  (after last Received:
      header) is fully trustable. A filter which lives between the
      second to last and last MTA cannot be trusted
      automatically. Instead, a list such as 3) would have to be
      configured to imply trust of the second to last MTA, etc.

6) MUAs trust tag insertion order. All filters in the organization
agree to insert their tags at the top of the mail header.

pros: easy to implement, no authentication required
cons: only the very last filter tag can be trusted, provided it is certain
      that the organization employs at least one filter. To trust more than
      one filter tag, a system such as 3) or 4) is still needed.


I think I've covered all the current proposals, though perhaps not
minor variations.

A couple of further points: solutions 1)-4) are robust against header
removal, eg if some agent removes or rewrites some headers. 5) and 6)
depend on tags being left semantically unchanged.

Every "actor" (ie the MUA, or a MDA which sorts into folders, or a
filter which blocks messages) must deal with the "content trust"
issue. It just cannot be ignored (which would be solution 1).

This issue is a direct consequence of the RFC2822 message format,
because the format doesn't separate meta information (ie headers) from
content (ie body) in a secure way. Thus anybody can and does spoof meta
information, and if successful can control actors which act on the meta
information, like the MUA.

Ok, that's quite enough for now. Hopefully, I'm not too far off the
mark.

--
Laird Breyer.