are we talking past each other?

Philip Miller wrote:
> David Nicol wrote:
> 
>>
>> Point-by-point response.
> 
> 
> I'll take the end of your message first, since it's really the most 
> important point:
> 
>  > Like I've said before, I believe the reason we're here, working at
>  > this project, is to accommodate the shared designation engine scenario,
>  > and not anything else.
> 
> I don't entirely agree with that. I think there is significant benefit 
> to MUAs developing to recognize a single classification header as 
> opposed to users having to program in regular expressions and more 
> complicated systems.
> Basically, I want to start pushing the implementation complexity of spam 
> filtering with cooperation between the MTA, MDA, and MUA from the user 
> to the software developers.
> 
>  > The Filtered header, by itself, is only half of the SDE scenario.
> 
> That's not true. The fact that there is no key for 'upstream 
> reclassification' doesn't mean it's not supported or out of scope. My 
> intention was for the Filtered header to be comprehensive, supporting 
> all manner of systems, and trying to provide as much of a stable base as 
> possible for developers to work with in order to make life easier for 
> their users and customers.
> 
> I'd like to adapt what you have written for upstream communication in 
> your draft. Does "standard internet draft front matter goes here" 
> include the 'derivative works allowed' clause?

i think i agreed to that when I subscribed to this list :)



>  > It seems like implementing recognition of the Miller Filtered Header
>  > would imply embedding a considerable programming language into
>  > filing rule parsing engines.
> 
> I don't think it would need to be nearly that complex. Simplest would be 
> picking out the classification provided by the last MTA in the chain or 
> the MDA and matching it to the user-designated folder. Obviously, it's 
> also possible to create a Bayesian system based on meta-features (i.e. 
> the classifications and their meta-data, rather than the actual content).

GUI MUAs, including netscape,evolution,thunderbird, provide the
following tests on e-mail headers.

	contains
	does not contain
	begins with
	ends with

I agree that a standard way to declare the results of an analysis
is a good thing.  SpamAssassin's X-Spam headers currently carry
a lot of semantic weight.



...
>> I take this to mean that the SJMDP document could contain a section
>> in the bit on recommended MUA design that says the junk/not-junk button
>> should fall back to an internal bayesian system until configured.
> 
> 
> That's an intriguing approach, but doesn't that defy the simplicity, 
> even a little? How many users are going to go to the effort of enabling 
> it, unless their ISP does it for them (human installer or software 
> autoconfiguration tool)?

this is not our problem, marketing available services to users of ISPs.
Some clear instructions in the welcome package should do it I'd think.


> I'll admit, the bulkier Filtered header design doesn't make this easier 
> in the short term. However, I believe that in the long term, it will be 
> more useful in the software users will have.

I included a way to show a variety of different spectra.


>> Also a possibility that POP3 and IMAP could be extended to have a
>> mechanism in them that says "This system is SJMDP-conformant" to keep the
>> configuration level low.
> 
> This doesn't work in the case where the user has an address that goes 
> through a tagging service that then forwards to the non-tagging ISP. The 
> ISP mail box server will not indicate that, and the user would have to 
> override such a mechanism.


the power user with the forwarding service would of course disable
the autoconfiguration mechanism.

I'm basing the approach on proxy.pac files.  I set up a web proxy for
university of missouri and published a proxy.pac file.  Those who
used it were pleased with the performance improvements.  It was not
widely promoted and was not widely used.  A proxy.pac file makes
configuration of web browser proxy settings trivial, once it is
set up.  Someone needs to set tell the browser about the .pac file
though.


>> Or we could recommend that MUA software that notices a consistent
>> mismatch between SJMDP-report and internal bayesian score merely use
>> SJMDP value as just another piece of information.
> 
> Again, that assumes a non-apathetic user. Will users really take notice?

I don't have any statistics about how many "users" (rather than
power-users) are currently using beysian filtering built into MUAs, or
are using shared designation engines (or what the SDE vendors call it,
I coined "SDE" for the draft and will gladly switch)

We're sort of at the point where this is turning into work...

> On the other hand, assuming somewhat interested users, take a look at a 
> not-so-theoretical possibility that MUAs implement CRM114-like matching 
> as an advance over token-Bayesian sorting. Classification phrases that 
> name a given host would acquire a very high weight due to correspondence 
> to the users expressed classification.
> 
> For example, if my MUA integrated such, it would very quickly weight 
> "host=fda.zemos.net class=[not-]spam" Filtered headers because they 
> would almost always match what I say, because I control all filtering on 
> fda.zemos.net.

Right.  So defining a language for analysis results appearing in headers
is a good thing.  It may be partially or entirely separate from
defining a language for communicating the result of an automated
"put this in the junk folder!" decision.


>> I had to turn the junk/not-junk button in Thunderbird on before it
>> worked. Adding a "respect SJMDP headers" button in the junk button
>> configuration is pretty much what I had in mind, and exactly
>> what degrees of full/partial/none would be made available are
>> outside the current reach of my crystal ball.
> 
> It seems that full/partial/none are all that SJMDP enables in MUAs. You 
> could do things like vector manipulation on classification, confidence, 
> and host trust in Filtered headers. The trust could be automated based 
> on how users react to messages with various classifications.

Currently the beyesian in thunderbird (bayesian?) has to learn new
spampaigns each time one is released.  The SDE vision is, the first user
in the group who receives the spam in question reports it, then the
later ones don't have to.

It seems like a feature that thunderbird or evolution hackers could
tack on to the bleading-edge releases fairly quickly.

It could even default to sending junk to [email protected] or
something.



> I guess I'm going to have to implement the Filtered header in an 
> experimental system and try it out. Seems the only way to get a truly 
> valuable result from it.
> 
>>> Also, many ISPs will not facilitate users 'opting out' of filtering 
>>> at the ISP in favor of an upstream forwarding address. If a formerly 
>>> non-using ISP installed a SJMDP system one day, there would be no 
>>> indication to end users that the results are coming from a different 
>>> source.
>>
>>
>> I expect the ISP would announce the upgrade through their normal
>> channels through which they communicate with their customers.  The
>> admins of Shared Designation Engines currently installed in offices
>> certainly have channels available to them.
> 
> 
> What if it's a cross-grade, say from SpamAssassin X- headers to SJMDP 
> headers? There's no reason any sensible residential ISP would announce 
> that, because it would be meaningless to many users and ignored by most, 
> but it could have a big impact on how their mail is handled in the MUA.

Why would they drop the spamassassin headers?  They would continue
to include them, at least during a transitional period.  They would
announce that the spamassassin could now be trained and tuned, and
offer the list of affinities that are being offered, and the
procedure for defining a new affinity.

>>> Finally, there's also the problem of (potential lack of) 
>>> authentication. It's trivial to authenticate that the headers were 
>>> written by the last compliant MTA the message went through. However, 
>>> this isn't particularly helpful, because there doesn't seem to be 
>>> anything indicating which MTA that was.
>>
>>
>> We don't care.  A conformant system clobbers preexisting declarations
>> and replaces with its own.  I suppose I may have silently presumed that
>> SJMDP would be appropriate where the MTA and the final hop MDA are under
>> the same management.  Does this need to be spelled out further?
> 
> 
> But if the MTA/MDA are not conformant and the MUA is, then the user has 
> the problem that the MUA is acting on designations that they have no 
> trust in. I assert that this will very frequently be the case, until 
> deployment is widespread.

then the user has not turned on the feature.  Just like with web 
browsers that support configuration to use proxies, but have no proxies
available to them. The feature is not turned on.  Hopefully all the
machine code relating to the feature is close together so it can all
be cleanly swapped out.

>> also, SJMDP includes facility for indicating back to that system
>> whether the final recipient agrees with the designation.
> 
> 
> As noted above, Filtered header does not exclude this. It was omitted 
> because I hadn't gotten that far through the list archives yet, as I've 
> been cramped on time.

works for me


>>> Filtered - At time T, software S (of version V) on host H determined 
>>> this to match classification C with confidence X
>>
>>
>> This information is currently available by extended headers, through
>> which vendors of external message analyzers communicate them currently.
> 
> 
> It's *sometimes* available, in varying levels of quantity, quality, and 
> completeness.
> 
>> What advantage does *having a standard* give?  The disadvantage -- a
>> standard way to spoof -- is clear. 
> 
> 
> All of it can be spoofed, except the where-when pair that authenticates 
> it and lets the MUA attach a high trust value to it.

spamassassin currently clobbers spurious spamassassin headers that are
there already.

>> It seems to me that the information that at time T, software S (of
>> version V) on host H determined a message to match classification C with
>> confidence X is better hidden in an obscure and site-specific encoding
>> and the advantage of standardizing this is not clear.
> 
> Creating MUA software that can present the output of any filter in a 
> reasonable manner without frivolous and possibly repetitive 
> configuration by the user is an unclear advantage?

if its diffirent it is less vulnerable to spoofing, but if it is
spoofproof then it can be standardized.

>> Unless the scenario the Filtered header is supposed to support is
>> internal between trusted advisors and a junk decision system.
> 
> It is intended to supersede every existing vendor-specific header for 
> all purposes they're used for. That includes downstream communication, 
> internal tuning at the recipient end, and upstream communication.

We don't want to presume a particular analysis architecture, or
do we?

>> If e-mail system E uses a shared designation engine and gets
>> opinions from several different content analyzers before arriving
>> at a designation, an architecture other than SMTP and addition of
>> headers would work just as well, or even better.
> 
> 
> Unfortunately, unless you're running a system like MS Exchange that can 
> communicate effectively outside the RFC 2822 content, adding headers 
> does appear to be the best way.

I had in mind a cluster where messages get put into shared storage
and could then be analyzed by various spam daemons before being filed
directly into IMAP folders.

Pushing my filing instructions from my MUA onto my MDA would be
an improvement; maybe its time to start using procmail after all these
years of playing with GUI tools.

> I'll note that the only current 
> approaches are addition of headers or integration into the software the 
> user interacts with directly. We can get the advantages of each of these 
> with a standardized Filtered header for MUAs to handle.

Are you meaning MTA here?

"Compatiblity with existing MUA software does not matter to Philip Miller."


>> Or is the point of the Filtered standard an encoding for C and X
>> that can be incorporated into message filing rules using sliding
>> scales in new MUA software and recognized as sliding scales
>> by cognizant future Beyesian systems?
> 
> That's one good way to use it.

:)

>>> What do users really want/need?
>>
>>
>> Maybe we could issue two reccomendations for the two scenarios.
> 
> 
> I don't think these two scenarios have any conflicting demands. It 
> should be quite possible to make a solution that serves both.

So we're drawing the line at describing the details of upstream
tuning communication with any granularity smaller than whole messages?


>>> 5. Authenticity of information
>>> SJMDP - if user has knowledge of the last compliant system in the 
>>> MTS, guaranteed. Otherwise, basically no guarantee
>>
>> when receiving mail from a compliant system, it works, otherwise it
>> doesn't.  <frown> My MUA doesn't suddenly randomly download incoming
>> e-mail from unknown POP servers.  Does yours? </frown>  The SJMDP
>> configuration would be a per-account setting on a multiple-account MUA.
> 
> If the MUA is upgraded to SJMDP compliance before the server, what happens?
> Filtered suffers the same issue, except there's no implication that it 
> would be trusted 100% by default, and it can be handled more usefully in 
> the long run.

the user finds that the SJMDP feature is not doing anything for them
and they turn it off, like Javascript before the infinite recursion bugs
in the javascript parsers were worked out.

>>> Filtered - guaranteed insertion after fixed points in time/network. 
>>> Can easily be extended to strong cryptographic protection
>>> I can see someone saying "I don't care who said this was spam, but I 
>>> know someone did". However, that doesn't mean such an attitude should 
>>> be written into a standard.
>>
>>
>> SJMDP also has some focus on operability with current MUA software.
>> There is no option for comparing time stamps between two headers in
>> the message filing rules creation editing interface on any GUI MUA
>> I've ever used, although I am sure adding such a rule to procmail
>> would be relatively easy.
> 
> That much is true. SJMDP can operate on the simple pattern matching/text 
> inclusion that MUAs universally provide, where Filtered cannot. I think 
> it's time to move beyond the simple text-matching in MUAs. External spam 
> filters left it behind years ago.
> 
> Simply stated, I think a modern MUA can and should implement a complete 
> email environment. Users should not have to interact with their MUA, and 
> filter control panel, their ISP's web-based configuration interface, and 
> a million other things to control their email. Their MUA can and should 
> handle all of this.

So is the account configuration language XML?  I thought it best to
leave that out for now.


>> My question is, if we're not trying to facilitate shared designation
>> engines, what are we doing? And I think I answered that above, at
>> least I gave a guess.
> 
> I guess a single user controlling a filtering engine is just a 
> degenerate case of shared designation, but it seems silly to call it 
> that. I think facilitating communication between filtering software, 
> MUAs, and users in general is a more useful but still quite reachable goal.

I coined up "shared designation engine" and "designation engine" while
writing the draft.  I do not know what the developers of current 
real-world systems that perform this function call it, or even if they
have taken a view large enough to bother assigning a name there.


>> There are several points to list in a MUA Implementation Considerations
>> section, I'll try to add that.
> 
> 
> OK, it is definitely something I haven't taken the time to start writing 
> out yet.
> 
>  > Thanks for your comments.
> 
> Very welcome, and I think having two contrasting proposals could be a 
> very good thing. If you don't mind, I'd like to use text from your 
> proposal in the Filtered header draft.


No, I don't mind: your RFC style seems more conformant than mine. please
list me as an author.


> Philip Miller


David Nicol

-- 
[email protected]
"There's a fine line between participation and mockery" -- Scott Adams