Re: Start Again ((was Re: Filtering Header Draft Discussion

On Oct 20 2004, David Nicol wrote:

> I see the problem as facilitating shared is-junk/is-not-junk interoperabilty.

> My feeling WRT the general purpose informational header is that the scope
> of the problem has been defined far too broadly.

Can you explain how this problem is being ignored or not addressed?
(We haven't a full list of keys yet, so I'm making up a few)

Processed: name="popfile"; result-tag="work"; result-confidence="78%";
	auth-received="7280B11DCC";
Processed: name="SpamAssassin"; result-tag="spam"; result-score="0.1";
	auth-received="7280B11DCC";
Processed: name="bogofilter"; result-tag="ham"; result-spamicity="0.500000";

etc.
We do need to clean up the list of key/values, that's true. Would
you like to go through the list we have and put in what you think will
address your concerns? I admit I've been concerned with other integrity issues,
so I may not be the best person to build such a list.

But there are other integrity issues we can't skip, such as not splitting up a
single processing result into multiple headers, which e.g. will simply result
in broken filtering results.

For example if the second line above was given as

Processed-ID: name="SpamAssassin"; version="1.23";
Processed-Result: result-tag="spam"; result-score="0.1";

then if the message is changed along the way to the MUA, maybe Processed-ID
is dropped or moved further away in the headers, 
and the MUA only sees the Processed-Result: header without knowing
which software wrote it. If some other software adds a Processed-ID: header of
its own, to which one does Processed-Result: refer to then? 

This particular issue is already a problem today. SpamAssassin can add several
headers, and how do we know which header goes with which other header? 

> > 2) identify and solve(?) issues such as authentication, overwriting,
> > modification, removal, coherence which are otherwise decided by each filter on an
> > ad-hoc basis.
> 

> "decided by each filter on an ad-hoc basis" is not a problem.  Not
> all the blades on a Swiss army knife are sharp.  By trying to
> "solve" these isues, we are looking for the Swiss army blade that
> does it all, rather than defining the meta-issues that

Agreed, I was being overly general in scope. It's not up to us to decide
what gets written in a header, but we do have things to say about how
things ought to be written to fix obvious interop difficulties today.
These are recomendations such as

1) don't write several headers with complementary information
2) provide an identification/authentication mechanism in each header that you write
3) use common key/value pairs if possible with standard meanings
4) don't change subject lines
5) don't modify mail bodies by embedding a hash value say, use a header instead

I can give you examples of filters which break each of those things, today.
Beyond those types of things, which can be forced by syntax, a standard wouldn't
have anything to say, really.

> 
> I favor a short prefix ("Tag") to allow for lots of length following
> it.  X- are experimental
> and are already well defined, so if we declare that compliant new
> headers are like
> 
> Tag-<name>: <keys and values>
> 
> and declare what information is to be kept in the registry of such
> things, we could
> call our mission accomplished, without driving ourselves crazy.
> 

Didn't we already discuss this? The above is equivalent to 

Tagged: name=<name>; <keys and values>

The perceived benefit of Tag-<name> is that you can run a simple regex
to find the header line, but pretty soon people will want to write
Tag-<name>-<ip>:, Tag-<ip>-<datetime>:, Tag-<name>-<version>-<ip>:
etc, and at that point you've actually made parsing more difficult,
because how do you know the exact format
Tag-<variable1>-<variable2>-<variable3>: and what the variables are?
That's the benefit of key value pairs.


> The big question becomes, "What is accomplished? instead of a plethora of
> X- headers, now there are Tag- headers too" and the answer is, X- headers are
> experimental and Tag- headers are well defined and the definitions are
> available in
> a well known place.

In the case of a registry listing all valid Tag-<variable1>, 
Tag-<variable1>-<variable2>, Tag-<variable1>-<variable2>-<variable3> 
sequences say, you're missing type information. What is the type of <variable1>, is
it a name, an IP address? Do you try each one in turn to see which better fits?

> SJMDP Simple Junk Mail Declaration Protocol described in 
> http://article.gmane.org/gmane.ietf.asrg.filtering/270 except that

> Changes to SJMDP would include changing the conclusive tags to
> 
> Tag-Junk: <keys and values>
> Tag-NotJunk: <keys and values>
> 
> changing the "If" tags to keys that would appear in the others,

That would be doable. Would you like to make the modifications to the document
we have (either the SJMDP or miller docs) so that we can discuss it concretely
on the list?

> or even better,
> 
> Tag-Plusnet: agent=imail; ver=0.5; status=filtered;
> host=cheeseburger.example.com
> Old-Plusnet; replhost=cheeseburger.example.com; orig=39:agent=hmail; ver=1.01; 
>    status=filtered
> 
> I have no idea what "status" might mean to the Plusnet system, but the final
> recipient (or their software) would know that the Plusnet-enabled software at 
> cheeseburger.example.com and the Plusnet-enabled software that gave
> cheeseburger the
> message both agreed that the message in question warrants a status of
> "filtered."

That's fine. If "status" is one of the common keywords, then "filtered" should
have a well defined, universal meaning. If "status" is a local keyword only
used by Plusnet, then "filtered" need only mean something to Plusnet.

> 
> 
> One design constraint I see, in which I disagree with Phillip Miller,
> unless he's come
> around, is:
> 
>    legacy MUA software that can prefile based on header appearance and content
>    but which is not able to compare bits from different headers in the
> same message
>    MUST be able to get full use out of compliant informational headers
> 
> Using the third renaming syntax option above, someone who trusts hmail more
> than imail, assuming that the imail plusnet software would reject a
> forgery using
> some plusnet-specific mechanism, could declare a recipient prefiling rule
> about Old-Plusnet headers containing "agent=hmail" and "status=filtered"  and
> "replhost=cheeseburger.example.com" in preference to the Tag-Plusnet header
> provided by the later relay.
> 

I'm not sure I understand what you're saying, please correct me if I'm
interpreting incorrectly. Nothing precludes the destination MUA from doing
any kind of filing it likes, based on any heuristics and trust relationships
it likes. That's simply not an issue for a standard. What matters I think is
that a standard, when its rules are followed, guarantees a certain behaviour.

The current situation with filtering headers in use today is that some filters
add a single header, and other filters add several headers representing a single
processing step. If a MUA wants to take full advantage of filtering performed by 
the kind of software which nowadays writes several headers, then such a MUA 
has to look at all the headers written, that is all. 

In my personal opinion, each filter should write a single header, with
as many key/value pairs in it as it likes. Splitting a result over
several headers is just not robust enough for interoperability.

-- 
Laird Breyer.