Re: Start Again ((was Re: Filtering Header Draft Discussion

David Nicol wrote:
> On Wed, 20 Oct 2004 17:29:55 +1000, Laird Breyer <[email protected]> wrote:
> 
>>On Oct 16 2004, J.J.Bailey wrote:
>>
>>>Does anyone have a feel for how much effort is being put into client-
>>>side filtering?  I'm curious to know what's driving the standard and
>>>who needs what where.
> 
> Several MUA systems use stastical filtering and is-junk/is-not-junk buttons. 
> 
> Larger unified webmail providers (yahoo, gmail) have shared systems.
> 
> There are shared systems available for DIY e-mail servers.  The best are
> integrated with IMAP servers in such a way that moving messages into/outof
> the JUNK folder is equivalent to an is-junk/is-not-junk button.
> 
> My feeling WRT the general purpose informational header is that the scope
> of the problem has been defined far too broadly.
> 
>   My feeling WRT this feeling
> is that I maybe I sit too close to the the problem and a result that is a set of
> reasonable reccomendations regarding key/value pair syntax in extended headers
> and creation of an authoritative registry for known extended heaers and their
> meanings -- a "best practices" document that refers to a created central name
> registry -- would be a good thing, even though it ignores what I see
> as the problem.
> 
> I see the problem as facilitating shared is-junk/is-not-junk interoperabilty.

I believe that shared designation interoperability should be one ultimate goal of a
document that this group produces. I also believe that integrating shared designation
systems into MUAs depends on the MUA being able to read the designation that the system
produced. Currently, there is no interoperable way for MUAs to do this, and I therefore
believe that we cannot proceed with an interoperability standard for MUA feedback to the
designation system unless and until we have a standard way for the MUA to read the result.

I will admit that the current draft does not address MUA feedback to designation systems.
However, the only necessary changes would be the addition of keys and their value
specification that describe what to do to provide feedback.

> The filtering WG does not IIRC have regular contribution from anyone who
> is involved with, for instance, the Thunderbird project, or development of any
> of the various commercial MUA products that currently have built-in
> junk/not-junk buttons that train internal statistical filters.

Then we need to find such developers and figure out how we can work with them.
Alternately, if we can convince filter authors to use a standardized header, MUA
developers will follow naturally, because they will see that they can provide better
functionality for their users by reading the new standardized header.
Note that this provides a nice transition plan: During the transitional period, filters
will still output the same headers they always have, so current MUA-based message handling
still works. However, MUAs will then have nice, standard headers that they can provide a
UI to work with when creating new message handling rules. Thus, for users and MUA
developers, we've made it easier to handle the simple case of "put the message where the
upstream filter recommended" from "parse arbitrary header that can be different for every
user (thus requiring the user to teach the MUA how to parse)" to "parse this one type of
standard header, the same way for every user". Once all of the major MUAs have added an
interface to handle the standard header, the old headers can slowly be deprecated.

>>Well this group is somewhat research oriented, so we might not end up
>>with an actual standard;-)
>>
>>If you read the past discussion on the list, there's a wide variety of
>>headers used in the wild, andthe list is growing. We did a quick
>>survey on one of the threads on this list, but if anything it showed
>>lots of variation.
>>
>>I believe the main interest for the filtering standard is to identify
>>whether we can formalise a common extensible format which would
>>
>>1) allow machines to robustly read header contents without needing
>>separate custom parsers for each header.
> 
> We chose to recognize key=value pairs separated by semicolons as a
> best practice.
> 
>>2) identify and solve(?) issues such as authentication, overwriting,
>>modification, removal, coherence which are otherwise decided by each filter on an
>>ad-hoc basis.
> 
> "decided by each filter on an ad-hoc basis" is not a problem. Not all the blades on a
> Swiss army knife are sharp. By trying to "solve" these isues, we are looking for the
> Swiss army blade that does it all, rather than defining the meta-issues that map in the
> pocket-knife idiom to, "there are axles in the tool on which the blades fold in and
> out" and "when all blades are folded in, the tool will not cut a hole in your pocket."

All of those issues being decided on a filter-by-filter basis is a problem. If different
filters do these things differently, then MUAs cannot behave consistently without sentient
consideration of every filter in the path of messages coming in. That leaves us in exactly
the position we're in now.

> Defining a set of criteria that will be maintained in the central registry for
> interoperability of the different blades, I see that as within the greater mission, not
>  micromanaging solutions to problems that are not facing us immediately.

Are you then suggesting that we create a central registry of the output of existing 
filters, with a complete description of how to parse and interpret each of their outputs, 
and expect MUAs to apply that information sensibly?

> How does each compliant extended header behave WRT authentication, overwriting,
> modification, removal, and coherence?  These are issues that can be deferred to
> definitions of the particular compliant extended headers.

Most of these are issues that have to be dealt with in every such header, and should 
therefore be handled by the general standard. If some variant header needs to modify that, 
such modification can be specified, and such variant headers will comply with the variant 
specification.

>>3) be simple and easy enough for all interested parties to implement.
>>
>>For example, the key/value pair format helps with 1).
>>
>>In the case of 2), consider the following header:
>>
>>X-Filtered-by: Plusnet (hmail v1.01)
>>
>>This is practically useless.
> 
> 
> but it will work for an example.  Maybe it would become
> Tag-Plusnet: agent=hmail; ver=1.01; status=filtered

You're basically suggesting putting data in the name of a header. The reason X- headers 
are named as they are is so that 2 different types of headers, output by different 
software, don't end up with the same name. That's not a worry for a standard header.

> I favor a short prefix ("Tag") to allow for lots of length following it. X- are
> experimental and are already well defined, so if we declare that compliant new headers
> are like
> 
> Tag-<name>: <keys and values>
> 
> and declare what information is to be kept in the registry of such
> things, we could call our mission accomplished, without driving ourselves crazy.
> 
> The big question becomes, "What is accomplished? instead of a plethora of
> X- headers, now there are Tag- headers too" and the answer is, X- headers are
> experimental and Tag- headers are well defined and the definitions are
> available in a well known place.

You're again suggesting that every MUA should apply information in a registry of headers, 
except that they now have registered names that are not directly contradictory to the 
nature of the X- semantics. This is not a major improvement.

> i envision, for instance, the SpamAssassin project would submit a definition for the
> Spam name, currently in their X-Spam headers, and would deprecate X-Spam in favor of
> Tag-Spam. the SPF people would deprecate Received-SPF in favor of Tag-SPF. The first to
> publish a definition of a general purpose junk status reporting instructions header
> would submit a definition for it, perhaps something like the SJMDP Simple Junk Mail
> Declaration Protocol described in 
> http://article.gmane.org/gmane.ietf.asrg.filtering/270 except that the sections on
> compliance, rewriting, and so on would follow a rigid format demanded by the general
> purpose Tag- header framework.
> 
> Changes to SJMDP would include changing the conclusive tags to
> 
> Tag-Junk: <keys and values>
> Tag-NotJunk: <keys and values>
> 
> changing the "If" tags to keys that would appear in the others,
> and changing the proposed renaming syntax to comply with a renaming
> convention described in the general proposal.
> 
> Perhaps the general proposal would declare that tags that are renamed have a 
> reserved name of "Renamed" and the reserved Renamed name description would
> include a mechanism for readably escaping the keys and values from the old
> one that has been rewritten. Perhaps using bencoding.  
> 
> (for bencoding purposes, all white space sequences can be presumed to compress
> to a single space.)
> 
> "Old" is shorter than "Renamed" and better carries the meaning of inclusion for
> forensic purposes. (That the header line was renamed is true, but meaningless to
> everyone except the developers of the software that is doing the renaming.)

Why do all that work? When specifying a new header, it is easier to standardize how they 
must be inserted so that it is eminently clear to a reader when and where each one was 
inserted. Forgeries will always appear on the wrong side of the trust boundary.
On the other hand, if we're just going to rubber-stamp current practice, despite the known 
problems (which are described in the draft), then what real benefit have we produced?

> [snip rewriting/renaming examples]
> 
> 
> One design constraint I see, in which I disagree with Phillip Miller, unless he's come 
> around, is:
> 
> legacy MUA software that can prefile based on header appearance and content but which
> is not able to compare bits from different headers in the same message MUST be able to
> get full use out of compliant informational headers

That is the one constraint that I don't think we can fulfill while providing full 
functionality. I think that rewriting and renaming headers is a fundamentally incorrect 
approach which will only cause pain and suffering for filter developers everywhere.

On the other hand, if we specify where newly inserted headers go and that existing headers 
are not to be touched, such limited MUAs can still match header contents, and will behave 
predictably. They still get the benefit of implicitly trusting the first or last header 
match. If someone wants to examine Outlook Express (which seems to be the single example 
we have been using of braindead MUA software) to see whether it acts on the first or last 
match, we could change the insertion order to say that each new header goes at the bottom 
of existing headers depending on the result.

> Using the third renaming syntax option above, someone who trusts hmail more than imail,
> assuming that the imail plusnet software would reject a forgery using some
> plusnet-specific mechanism, could declare a recipient prefiling rule about Old-Plusnet
> headers containing "agent=hmail" and "status=filtered" and 
> "replhost=cheeseburger.example.com" in preference to the Tag-Plusnet header provided by
> the later relay.

I don't know of any MUA that will do multiple conditions on a single header, but cannot do 
arbitrary processing. Perhaps you could do multiple OR-ed conditions with wildcards, to 
account for the different possible orderings of the keys (note that you would otherwise 
have to specify order in a standard). However, that seems rather silly.

Philip Miller