Start Again ((was Re: Filtering Header Draft Discussion

On Wed, 20 Oct 2004 17:29:55 +1000, Laird Breyer <[email protected]> wrote:
> On Oct 16 2004, J.J.Bailey wrote:
> >
> > Does anyone have a feel for how much effort is being put into client-
> > side filtering?  I'm curious to know what's driving the standard and
> > who needs what where.

Several MUA systems use stastical filtering and is-junk/is-not-junk buttons. 

Larger unified webmail providers (yahoo, gmail) have shared systems.

There are shared systems available for DIY e-mail servers.  The best are
integrated with IMAP servers in such a way that moving messages into/outof
the JUNK folder is equivalent to an is-junk/is-not-junk button.

My feeling WRT the general purpose informational header is that the scope
of the problem has been defined far too broadly.

  My feeling WRT this feeling
is that I maybe I sit too close to the the problem and a result that is a set of
reasonable reccomendations regarding key/value pair syntax in extended headers
and creation of an authoritative registry for known extended heaers and their
meanings -- a "best practices" document that refers to a created central name
registry -- would be a good thing, even though it ignores what I see
as the problem.

I see the problem as facilitating shared is-junk/is-not-junk interoperabilty.

The filtering WG does not IIRC have regular contribution from anyone who
is involved with, for instance, the Thunderbird project, or development of any
of the various commercial MUA products that currently have built-in
junk/not-junk
buttons that train internal statistical filters.

> Well this group is somewhat research oriented, so we might not end up
> with an actual standard;-)
> 
> If you read the past discussion on the list, there's a wide variety of
> headers used in the wild, andthe list is growing. We did a quick
> survey on one of the threads on this list, but if anything it showed
> lots of variation.
> 
> I believe the main interest for the filtering standard is to identify
> whether we can formalise a common extensible format which would
> 
> 1) allow machines to robustly read header contents without needing
> separate custom parsers for each header.

We chose to recognize key=value pairs separated by semicolons as a
best practice.

> 2) identify and solve(?) issues such as authentication, overwriting,
> modification, removal, coherence which are otherwise decided by each filter on an
> ad-hoc basis.

"decided by each filter on an ad-hoc basis" is not a problem.  Not all
the blades on
a Swiss army knife are sharp.  By trying to "solve" these isues, we are looking
for the Swiss army blade that does it all, rather than defining the
meta-issues that
map in the pocket-knife idiom to, "there are axles in the tool on
which the blades
fold in and out"  and "when all blades are folded in, the tool will
not cut a hole in
your pocket."  

Defining a set of criteria that will be maintained in the central
registry for interoperability
of the different blades, I see that as within the greater mission, not
micromanaging
solutions to problems that are not facing us immediately.

How does each compliant extended header behave WRT authentication, overwriting,
modification, removal, and coherence?  These are issues that can be deferred to
definitions of the particular compliant extended headers.

> 3) be simple and easy enough for all interested parties to implement.
> 
> For example, the key/value pair format helps with 1).
> 
> In the case of 2), consider the following header:
> 
> X-Filtered-by: Plusnet (hmail v1.01)
> 
> This is practically useless.

but it will work for an example.  Maybe it would become
Tag-Plusnet: agent=hmail; ver=1.01; status=filtered

I favor a short prefix ("Tag") to allow for lots of length following
it.  X- are experimental
and are already well defined, so if we declare that compliant new
headers are like

Tag-<name>: <keys and values>

and declare what information is to be kept in the registry of such
things, we could
call our mission accomplished, without driving ourselves crazy.

The big question becomes, "What is accomplished? instead of a plethora of
X- headers, now there are Tag- headers too" and the answer is, X- headers are
experimental and Tag- headers are well defined and the definitions are
available in
a well known place.

i envision, for instance, the SpamAssassin project would submit a definition for
the Spam name, currently in their X-Spam headers, and would deprecate X-Spam in
favor of Tag-Spam.  the SPF people would deprecate Received-SPF in favor of
Tag-SPF.  The first to publish a definition of a general purpose junk
status reporting
instructions header would submit a definition for it, perhaps something like the
SJMDP Simple Junk Mail Declaration Protocol described in 
http://article.gmane.org/gmane.ietf.asrg.filtering/270 except that
the sections on compliance, rewriting, and so on would follow a rigid format
demanded by the general purpose Tag- header framework.

Changes to SJMDP would include changing the conclusive tags to

Tag-Junk: <keys and values>
Tag-NotJunk: <keys and values>

changing the "If" tags to keys that would appear in the others,

and changing the proposed renaming syntax to comply with a renaming
convention described in the general proposal.

Perhaps the general proposal would declare that tags that are renamed have a 
reserved name of "Renamed" and the reserved Renamed name description would
include a mechanism for readably escaping the keys and values from the old
one that has been rewritten. Perhaps using bencoding.  

(for bencoding purposes, all white space sequences can be presumed to compress
to a single space.)

"Old" is shorter than "Renamed" and better carries the meaning of
inclusion for forensic
purposes.  (That the header line was renamed is true, but meaningless
to everyone
except the developers of the software that is doing the renaming.)

Tag-Plusnet: agent=imail; ver=0.5; status=filtered;
host=cheeseburger.example.com
Tag-Old: name=Plusnet; replhost=cheeseburger.example.com;
 orig=39:agent=hmail; ver=1.01; status=filtered

We could more fully embrace the concept of rewriting by declaring Old to
be another new header prefix rather than a reserved name.

Tag-Plusnet: agent=imail; ver=0.5; status=filtered;
host=cheeseburger.example.com
Old: name=Plusnet; replhost=cheeseburger.example.com; orig=39:agent=hmail;
    ver=1.01; status=filtered

or even better,

Tag-Plusnet: agent=imail; ver=0.5; status=filtered;
host=cheeseburger.example.com
Old-Plusnet; replhost=cheeseburger.example.com; orig=39:agent=hmail; ver=1.01; 
   status=filtered

I have no idea what "status" might mean to the Plusnet system, but the final
recipient (or their software) would know that the Plusnet-enabled software at 
cheeseburger.example.com and the Plusnet-enabled software that gave
cheeseburger the
message both agreed that the message in question warrants a status of
"filtered."


One design constraint I see, in which I disagree with Phillip Miller,
unless he's come
around, is:

   legacy MUA software that can prefile based on header appearance and content
   but which is not able to compare bits from different headers in the
same message
   MUST be able to get full use out of compliant informational headers

Using the third renaming syntax option above, someone who trusts hmail more
than imail, assuming that the imail plusnet software would reject a
forgery using
some plusnet-specific mechanism, could declare a recipient prefiling rule
about Old-Plusnet headers containing "agent=hmail" and "status=filtered"  and
"replhost=cheeseburger.example.com" in preference to the Tag-Plusnet header
provided by the later relay.


-- 
David L Nicol
transportation infrastructure technology contracting since 2002