Re: draft of Simple Junk Mail Declaration Protocol

David Nicol wrote:
> 
> Point-by-point response.

I'll take the end of your message first, since it's really the most 
important point:

 > Like I've said before, I believe the reason we're here, working at
 > this project, is to accommodate the shared designation engine scenario,
 > and not anything else.

I don't entirely agree with that. I think there is significant benefit to 
MUAs developing to recognize a single classification header as opposed to 
users having to program in regular expressions and more complicated systems.
Basically, I want to start pushing the implementation complexity of spam 
filtering with cooperation between the MTA, MDA, and MUA from the user to 
the software developers.

 > The Filtered header, by itself, is only half of the SDE scenario.

That's not true. The fact that there is no key for 'upstream 
reclassification' doesn't mean it's not supported or out of scope. My 
intention was for the Filtered header to be comprehensive, supporting all 
manner of systems, and trying to provide as much of a stable base as 
possible for developers to work with in order to make life easier for their 
users and customers.

I'd like to adapt what you have written for upstream communication in your 
draft. Does "standard internet draft front matter goes here" include the 
'derivative works allowed' clause?

 > It seems like implementing recognition of the Miller Filtered Header
 > would imply embedding a considerable programming language into
 > filing rule parsing engines.

I don't think it would need to be nearly that complex. Simplest would be 
picking out the classification provided by the last MTA in the chain or the 
MDA and matching it to the user-designated folder. Obviously, it's also 
possible to create a Bayesian system based on meta-features (i.e. the 
classifications and their meta-data, rather than the actual content).

> Philip Miller wrote:
> 
>> David Nicol wrote:
>>
>>> I have written a description of a simple protocol.  The front matter,
>>> back matter, abstract are missing.  It is available at
>>>
>>> http://zarquon.advenge.com/RFCwiki/view?doc=AI
>>>
>>> as well as appearing attached to this message.
>>
>> I have one issue with this system:
>> The vast majority of users have not configured spam filters for 
>> themselves. If/when they upgrade MUAs to a version that supports this 
>> standard, there will not necessarily be a SJMDP-compliant filter 
>> working on their behalf, or even to their knowledge. Thus, spammers 
>> could add Is-Not-Junk headers to all outgoing messages, knowing that 
>> it will be treated as 'first class mail' by the MUAs of people who 
>> aren't filtering for themselves.
>> On the other hand, these users are probably signed up with large ISPs 
>> that do filter aggressively, so it may not be such an issue.
> 
> I take this to mean that the SJMDP document could contain a section
> in the bit on recommended MUA design that says the junk/not-junk button
> should fall back to an internal bayesian system until configured.

That's an intriguing approach, but doesn't that defy the simplicity, even a 
little? How many users are going to go to the effort of enabling it, unless 
their ISP does it for them (human installer or software autoconfiguration tool)?

I'll admit, the bulkier Filtered header design doesn't make this easier in 
the short term. However, I believe that in the long term, it will be more 
useful in the software users will have.

> Also a possibility that POP3 and IMAP could be extended to have a
> mechanism in them that says "This system is SJMDP-conformant" to keep the
> configuration level low.

This doesn't work in the case where the user has an address that goes 
through a tagging service that then forwards to the non-tagging ISP. The ISP 
mail box server will not indicate that, and the user would have to override 
such a mechanism.

> Or we could recommend that MUA software that notices a consistent
> mismatch between SJMDP-report and internal bayesian score merely use
> SJMDP value as just another piece of information.

Again, that assumes a non-apathetic user. Will users really take notice?

On the other hand, assuming somewhat interested users, take a look at a 
not-so-theoretical possibility that MUAs implement CRM114-like matching as 
an advance over token-Bayesian sorting. Classification phrases that name a 
given host would acquire a very high weight due to correspondence to the 
users expressed classification.

For example, if my MUA integrated such, it would very quickly weight 
"host=fda.zemos.net class=[not-]spam" Filtered headers because they would 
almost always match what I say, because I control all filtering on 
fda.zemos.net.
> I had to turn the junk/not-junk button in Thunderbird on before it
> worked. Adding a "respect SJMDP headers" button in the junk button
> configuration is pretty much what I had in mind, and exactly
> what degrees of full/partial/none would be made available are
> outside the current reach of my crystal ball.

It seems that full/partial/none are all that SJMDP enables in MUAs. You 
could do things like vector manipulation on classification, confidence, and 
host trust in Filtered headers. The trust could be automated based on how 
users react to messages with various classifications.

I guess I'm going to have to implement the Filtered header in an 
experimental system and try it out. Seems the only way to get a truly 
valuable result from it.

>> Also, many ISPs will not facilitate users 'opting out' of filtering at 
>> the ISP in favor of an upstream forwarding address. If a formerly 
>> non-using ISP installed a SJMDP system one day, there would be no 
>> indication to end users that the results are coming from a different 
>> source.
> 
> I expect the ISP would announce the upgrade through their normal
> channels through which they communicate with their customers.  The
> admins of Shared Designation Engines currently installed in offices
> certainly have channels available to them.

What if it's a cross-grade, say from SpamAssassin X- headers to SJMDP 
headers? There's no reason any sensible residential ISP would announce that, 
because it would be meaningless to many users and ignored by most, but it 
could have a big impact on how their mail is handled in the MUA.

>> Finally, there's also the problem of (potential lack of) 
>> authentication. It's trivial to authenticate that the headers were 
>> written by the last compliant MTA the message went through. However, 
>> this isn't particularly helpful, because there doesn't seem to be 
>> anything indicating which MTA that was.
> 
> We don't care.  A conformant system clobbers preexisting declarations
> and replaces with its own.  I suppose I may have silently presumed that
> SJMDP would be appropriate where the MTA and the final hop MDA are under
> the same management.  Does this need to be spelled out further?

But if the MTA/MDA are not conformant and the MUA is, then the user has the 
problem that the MUA is acting on designations that they have no trust in. I 
assert that this will very frequently be the case, until deployment is 
widespread.

>> Here's how it compares to a more general Filtered header:
>> 1. Programming generation/insertion
>> SJMDP - strip existing, insert fixed string (maybe generate part)
>> Filtered - Insert generated/composed string in trace field position
>> So the cost tradeoff is finding and stripping the existing headers 
>> versus building the new header.
>> 2. Programmatically interpreting results
>> SJMDP - search for Is-[Not-]Junk headers
>> Filtered - read all such headers (or stop after certain distance from 
>> top), decide which ones to trust (whatever desired algorithm, from 
>> presence to crypto-signature), parse for desired information
>> Tradeoff is more time processing for more information gotten
>> 3. Range of possible results
>> SJMDP - yes or no
>> Filtered - # possible classifications * range of confidence values
>> Do we want restricted, single use, or do we want open-ended, 
>> extensible use?
>> 4. Information conveyed to end recipient
>> SJMDP - does some system think this is or isn't junk?
> 
> also, SJMDP includes facility for indicating back to that system
> whether the final recipient agrees with the designation.

As noted above, Filtered header does not exclude this. It was omitted 
because I hadn't gotten that far through the list archives yet, as I've been 
cramped on time.

>> Filtered - At time T, software S (of version V) on host H determined 
>> this to match classification C with confidence X
> 
> This information is currently available by extended headers, through
> which vendors of external message analyzers communicate them currently.

It's *sometimes* available, in varying levels of quantity, quality, and 
completeness.

> What advantage does *having a standard* give?  The disadvantage -- a
> standard way to spoof -- is clear. 

All of it can be spoofed, except the where-when pair that authenticates it 
and lets the MUA attach a high trust value to it.

> It seems to me that the information that at time T, software S (of
> version V) on host H determined a message to match classification C with
> confidence X is better hidden in an obscure and site-specific encoding
> and the advantage of standardizing this is not clear.

Creating MUA software that can present the output of any filter in a 
reasonable manner without frivolous and possibly repetitive configuration by 
the user is an unclear advantage?

> Unless the scenario the Filtered header is supposed to support is
> internal between trusted advisors and a junk decision system.

It is intended to supersede every existing vendor-specific header for all 
purposes they're used for. That includes downstream communication, internal 
tuning at the recipient end, and upstream communication.

> If e-mail system E uses a shared designation engine and gets
> opinions from several different content analyzers before arriving
> at a designation, an architecture other than SMTP and addition of
> headers would work just as well, or even better.

Unfortunately, unless you're running a system like MS Exchange that can 
communicate effectively outside the RFC 2822 content, adding headers does 
appear to be the best way. I'll note that the only current approaches are 
addition of headers or integration into the software the user interacts with 
directly. We can get the advantages of each of these with a standardized 
Filtered header for MUAs to handle.

> Or is the point of the Filtered standard an encoding for C and X
> that can be incorporated into message filing rules using sliding
> scales in new MUA software and recognized as sliding scales
> by cognizant future Beyesian systems?

That's one good way to use it.

>> What do users really want/need?
> 
> Maybe we could issue two reccomendations for the two scenarios.

I don't think these two scenarios have any conflicting demands. It should be 
quite possible to make a solution that serves both.

>> 5. Authenticity of information
>> SJMDP - if user has knowledge of the last compliant system in the MTS, 
>> guaranteed. Otherwise, basically no guarantee
> 
> when receiving mail from a compliant system, it works, otherwise it
> doesn't.  <frown> My MUA doesn't suddenly randomly download incoming
> e-mail from unknown POP servers.  Does yours? </frown>  The SJMDP
> configuration would be a per-account setting on a multiple-account MUA.

If the MUA is upgraded to SJMDP compliance before the server, what happens?
Filtered suffers the same issue, except there's no implication that it would 
be trusted 100% by default, and it can be handled more usefully in the long run.

>> Filtered - guaranteed insertion after fixed points in time/network. 
>> Can easily be extended to strong cryptographic protection
>> I can see someone saying "I don't care who said this was spam, but I 
>> know someone did". However, that doesn't mean such an attitude should 
>> be written into a standard.
> 
> SJMDP also has some focus on operability with current MUA software.
> There is no option for comparing time stamps between two headers in
> the message filing rules creation editing interface on any GUI MUA
> I've ever used, although I am sure adding such a rule to procmail
> would be relatively easy.

That much is true. SJMDP can operate on the simple pattern matching/text 
inclusion that MUAs universally provide, where Filtered cannot. I think it's 
time to move beyond the simple text-matching in MUAs. External spam filters 
left it behind years ago.

Simply stated, I think a modern MUA can and should implement a complete 
email environment. Users should not have to interact with their MUA, and 
filter control panel, their ISP's web-based configuration interface, and a 
million other things to control their email. Their MUA can and should handle 
all of this.

> My question is, if we're not trying to facilitate shared designation
> engines, what are we doing? And I think I answered that above, at
> least I gave a guess.

I guess a single user controlling a filtering engine is just a degenerate 
case of shared designation, but it seems silly to call it that. I think 
facilitating communication between filtering software, MUAs, and users in 
general is a more useful but still quite reachable goal.

> There are several points to list in a MUA Implementation Considerations
> section, I'll try to add that.

OK, it is definitely something I haven't taken the time to start writing out 
yet.

 > Thanks for your comments.

Very welcome, and I think having two contrasting proposals could be a very 
good thing. If you don't mind, I'd like to use text from your proposal in 
the Filtered header draft.

Philip Miller