Summary of Issues Raised to date

Jesse was working on something but seems to have disappeared 
temporarily. Meanwhile, I put together his original message plus your 
comments into a Wiki page 
(http://www.shaftek.org/wiki/wiki.pl?Filtering_Standards/Issues_Raised). 
I am including a copy here, feel free to comment here and edit the Wiki 
directly:

=== This page outlines some of the issues raised during the discussions 
on the mailing list. This is based on 
[http://article.gmane.org/gmane.ietf.asrg.filtering/64 a message] by 
Jesse Dougherty. ===

=== Multiple Filter Types Possible ===
* spam filtering isn't the only potential filter running.  A filter 
header specification should be extensible to other filtering types. 
Possible three types of filters:

1) Protection - MTA/MDA - implemented by administrator
   - malware
   - UCE/spam filtering

2) Policy - enforce communications policies (MTA or MDA)
   - content policy
   - archiving
   - ...

3) Preference (MDA or MUA)
   - personal rules
   - personal bayes filtering

Each level may remove items from the mail stream, or add data to the 
messages for consumption later down the line.

* More types of filters are possible such as IDS systems, etc.

* Filters should generally NOT modify the body of the message directly 
(e.g. add a URL for the filter), since such standard description  can 
quickly become an indicator for the filter. Anything that makes it so 
the downstream filter cannot see exactly the same message as the 
upstream filter is unstable (ie if you remove a filter in the chain, 
you've sabotaged training for all filters downstream).  If the filter's 
behavior is changed (e.g. removing the URL), repercussions could be 
dramatic for a while.

* Anti-virus software and other agent may sometimes need to modify the 
body of the message but in those cases they will likely to elect to 
remove an email, or say "replace" it with a notice that the email is 
suspect and can be picked up in a quarantine zone. In this sort of case, 
filtering software is not likely to see  tampered messages.

* An upstream filter may know about something that breaks a downstream 
filter (in possibly damaging ways), and *need* to be able to stop that 
payload from getting to your filter. One possibility may include a blend 
of "don't modify unless you have to" with  "if you have to modify, 
here's how you signal what you did, and why".

* Sometimes upstream filters such as IDS systems may need to pass some 
sort of indicator that this message had some strange characteristics 
which may not be apparent when simply analysing the message. (like it 
was delivered in a connection which also attempted to deliver to a lot 
of other addresses, all starting with 'r').

* Filtering headers add to the information processed downstream. To 
prevent unwanted feedback (ie a procmail filter catching on to a Spam 
Assassin recommendation), the spec needs to be so that added headers can 
be recognized by all and skipped. This suggests a common prefix such as 
"X-SPAM-", so that the programs downstream don't need extensive logic to 
identify and skip headers.

=== No Standard Meaning for Labels ===

* an administrator may opt to have their filtering framework assign a 
label with the outcome of a specific filter.  These labels have no 
standard meaning, requiring that configuration of the clients requires 
more offline work. Moreover, the labelling may not even be consistent 
over several versions of the same filter.

* The definition of spam isn't universal and filtering technologies may 
have subtle differences in their implementation of spam detection.  To 
this end, labelling a message as spam may not be consistent. (unwanted 
vs. UCE or some combination thereof)

* filters all have different methods of labelling messages, resulting in 
no common way to configure MUAs to identify filter labels in incoming 
messages

* some filters develop a confidence as well as a label, allowing later 
processing to be tuned to reduce errors.  Some filters are unable to 
parse these confidence levels.  A common format for confidence is also 
required.

* MUAs and MDAs have varying rules capabilities from simplistic to 
complex and not all labelling methods respect this.  It takes work for a 
recipient to configure their filters to deal with labels added by a 
trusted spam filter.

* Users want a label and a confidence to allow the recipients to set 
their own spam thresholds.

* One of the ways we can do this is to be able to provide for multiple 
rule systems in the headers. For example: "Filter-Results: 
rule-system-type=sieve; results=xxx". We can specify an IANA registry 
for rules, plus some commons rules as part of the standard.

* The labelling spec needs to consider that MUAs and MDAs have a vast 
array of parsing capabilities and we need to consider that when dealing 
with a multiple filter chain. We need to consider:
1) multiple filters attempting similar classifications (ie: dnsbls at 
the gateway edge, followed by content filtering at
a second tier.)
2) multiple filters attempting different classifications (ie: virus, 
spam, other content filters)

* Actions should not be determined by the filters.  They simply need to 
label a message and actions would be determined downstream.

=== There is a Need for More Metadata ===

* filtering decisions are dependant on the state of the filtering (virus 
definitions, software version) at the time of message scanning resulting 
in different results for the same message at different times.  It is 
important for filters to be able to add meta data to the label.

* Recipients may believe that a message was labelled in error and want 
to provide as a defect to the administrator. This administrator (or the 
filter vendor) will need more meta data about the filtering than a 
single judgement label.  More state data will be required to diagnose 
and adjust/reject the feedback.

* This suggests a different type of labelling is already used by some 
filters: instead of advisory labels only, the message also carries state 
information. I think that means we need to discuss two different types 
of headers separately:
1) headers which advise the downstream programs.
2) headers which are meant to be private to the filter.

=== Trust Is Needed  ===

* Labels may be added to the message prior to gateway filtering (on the 
outgoing MTA?) and may not be trusted.  How to tell if a label was added 
by the trusted MTA?  What should an MTA do if it receives an incoming 
labelled message? "- this message was virus scanned by ..."

Yakov