daily response 18 mar 2004

Jesse Dougherty at Sophos.com (have you seen their ads in SysAdmin?) wrote:

>IMHO, it is necessary to distinguish between simple headers and data
>added by email filters which may be consumed downstream.  Perhaps we
>should distinguish between "classifications" and "recommendations for
>action".
>

I am opposed to "recommending" anything. Classifying as "reccommend 
deletion"
is completely equivalent. We also run the risk of getting into "should" 
language
which experience has shown is to be vigorously eschewed.
What would the operational distinction between c. and r.f.a. be?



Laird Breyer wrote:

>The most popular systems will be gone in 5 to 10 years
>
that is a marketing-driven product cycle. With e-mail it is not
true. Witness the number of people quite content to be answering their
e-mail with pine since they know it already. Extended to embrace
new developments (launching handlers from pine) but not replaced.


>>>> >... the filter hacks the MUA, ...
>>>
>>>
>>> I do not understand this point.  There appears to be confusion
>>> between MTA filtering and MUA filtering.
>>>
>>    
>>
>
>Maybe I am confused, but of course I wouldn't know, I'll have to take
>your word for it 
>
>It's true that I don't really make a distinction between MTA filtering
>and MUA filtering. Perhaps I should, but my current thinking is that
>the MUA is only a step along the lifetime of a message, not a final
>destination (from the pov of filtering).
>
>Messages can be forwarded, replied to, used for learning material,
>moved around folders, kept in archives and reimported into another
>MUA, retagged.
>
Hmm. we're trying to discuss e-mail filtering here, not exhaustive
library indexing.

>Unlike transport information, filtering information impacts these
>actions. That's because filtering doesn't stop after transport.
>Filtering could go on for years after the message is received, and
>every piece of information a filter adds could be used by the next filter.
>
>By contrast with transport information, the MUA is officially the final
>destination, ie there is no official use for the transport info after
>it's in the MUA. If the message is moved or forwarded, the original
>transport information is discarded, not used by the next MTA.
>
okay, so you understand the differences between MUA/MTA

>Here's the simplest comparative example I can think of, which repeats
>what I just said:
>
>A message is sent from A to B, who forwards it to C. There are two
>transport actions, and two filtering actions.
>
>>From A to B, MTA1 adds transport information. From A to B, filter1 adds
>filtering information.
>
>>From B to C, MTA2 adds transport info, ignoring old transport info
>(it's really a completely new message from transport pov).
>>From B to C, filter2 adds filtering info, taking into account contents
>of message and therefore previous filter1 info. (it's basically the
>same message from filtering pov)
>
>So after this long explanation, I don't see a distinction between MTA
>filtering and MUA filtering.
>

I am working on an aggressively filtering SMTP server. It refuses messages
that have not been approved in advance by the recipients. This means that
it does not accept them and the onus of attempting redelivery is on the
sender. I call this MTA filtering. I regularly use both thunderbird and
evolution to answer e-mail. Thunderbird has bayesian filtering and evolution
(at least the version I have upgraded to) does not. Both allow me to set up
delivery rules so that messages meeting certain criteria are placed in 
certain
folders.

MTA filtering: messages are delivered or blocked (or delivered and 
tagged) (et cetera)
MUA filtering: messages are filed (perhaps respecting tags placed by an MTA)

That is real, today, current practice. Big distinction.


>>> We desparately need named scenarios.
>>
>
>I agree. But they must be simple, so everyone can understand them.
>
http://www.shaftek.org/wiki/wiki.pl?Filtering_Standards now contains
/Scenarios and /Filter_Types ready for your enumerating.



Mark E. Mallett wrote:

 > I believe that some kind of location marker is needed [in the 
filter-added
 > header]: whether it be a made-up ID or a sequence number or an 
address-- a
 > sequence number (such as suggested by Laird earlier) does seem neat 
and tidy,

the point of this is to better identify, later, the filtering event, for 
purposes of responsibility tracking?

>
>
> Does that makes sense?

it makes enough sense that I nominate you to paste it all in to a 
"format of filter headers (draft)"
page on Yakov's wiki.

>>> Which brings another requirement. We must decide what MUA we want to
>>> support. Reccomending munging the Subject line seems to be a good 
>> practice
>>> for things like list servers.
>
>
> Maybe it's the time of night, but I don't follow. Support an MUA
> in what way? I would think we would have a goal of not breaking
> any existing operation, while providing better data for future software
> to use. But why pick a MUA ?

someone raised "but that won't work with current releases of outlook 
express" as an objection to
a suggestion.

>>> Maybe we can agree on and produce a BCP on subject line alterations.
>
>
> I think I'll give that, er, subject, a rest for a bit

add a page to the wiki -- it's easy! it's fun!

>
> mm


Y.S. wrote, in an introductory message:

> Since this is a research group, the longer term stuff is important 
> since that's where ideas for standards come from. The slight friction 
> between the longer term research ideas like MTA/MUA communications and 
> the short term stuff like filtering headers is natural here. What the 
> group needs to do at certain intervals is to determine whether 
> specific ideas are mature enough and have enough support in the 
> community to be developed further as standards. This is what we the 
> goal is for the filtering header. 

for instance, numbered "spam score" is difficult to work with as number 
compare is not
unversally provided in MUA filing systems.

In general, now, an e-mail provider can announce to their customers "we 
have foobarbaz filter
in place so configure your MUA to sort anything with an 
"X-Foobarbaz-Is-Spam: Yes" into
your probably junk folder.  No further standardization is required, 
besides enshrining this practice
as a BCP.  Where standardization would really add something is in 
returning feedback to the system.
Yahoo webmail aggregates the spam reports of millions of users, within 
their webmail.  Mechanisms
exist whereby filter tuning data may be added, such as the ISP 
maintaining web pages or designated
e-mail addresses and a system for tuning the tagger (I guess I 
distinguish between "filters" that block,
"taggers" that advise, and "filers" that deliver here or there, 
depending) .  

Bayesian tagging is ....

In order to implement a framework where I could configure the "this is 
[not] junk" button in
Thunderbird to pass the reccomendation to a shared filtering (tagging?) 
engine instead of adjusting
the internal engine, with off-the-shelf software, what are the 
requirements for that? 

* the MTA tagging engine must re-accept the message somehow, either in 
full text or by an expiring reference

* the MUA must be configurable to present the user reccomenation to the 
MTA in the way in which the
MTA is prepared to accept it

* the communication (at some layer) needs to authenticate the user

* the system as a whole needs to support multiple filtering instances 
within the same MTA  because
rarely do all recipients associated with a given MTA share the same 
filtering preferences in terms
of what unsolicited yet legally protected advertising they care to see

(I'm going to paste that into a page on the wiki ...
http://www.shaftek.org/wiki/wiki.pl?Filtering_Standards/Requirements_For_Off_The_Shelf_Upstream_Bayesian_Tuning_Reccomendation_Interoperability
it's extended already: enjoy)



> As per my message regarding the scope of IRTF work, the research 
> topics are actually very appropriate. The key question is whether a 
> larger, more general MTA/MUA protocol will encompass the filtering 
> header. If not, then we should probably concentrate on the filtering 
> header first.
>
> Thoughts?
>
> Yakov

My thought on this is that the MUA->MTA communication itself does not 
"encompass the filtering header" as
the filtering header is the MTA->MUA communication concerning the aspect 
which is to be tuned.  Therefore,
we need to know what the upstream communication needs to know in order 
to decide what to put into
downstream communication.

If all we care about is downstream communication there is no need for 
further standardization: good MUAs
will let you declare filing rules on arbitrary headers and that is all 
that is needed.

The existence of this group implies that we need to look at both sides, 
because otherwise there is no
point to the group.  I don't want to have to consider the last three 
hours a waste so therefore both sides
of the conversation (upstream as well as downstream) must be considered, 
as the Point Of A Stanard Should
One Appear would be to facilitate upstream as well as downstream 
communication.  Downstream communication
is working quite well as is. (for definitions of quite well that ignore 
popular MUAs that cannot be
configured to file based on arbitrary headers and that ignore the 
failure of other popular MUAs to offer
filing rules to offer number-comparitive criteria. These are political 
issues, not technical.)

So.  If the MTA contains multiple filtering engines,  the upstream 
communication needs to
know which engine is to be tuned, and how far to tune it.  Maybe 
is/is-not is too crude.

Within the scenario of upstream communication to tune a shared bayesian 
tagger.


> The approach taken by similar standards has been along the lines of 
> "key=value; key1=value1; key2=value;" with the ";" separating tokens. 
> The actual "key" values are stored in an IANA registry and people have 
> freedom to define their own "x-key" extensions which do not interfere. 
> At the same time it remains extensible by adding new "key" extensions 
> to the IANA registry.
>
> However, one particular point to keep in mind here is the limit of the 
> header length (something like 900 bytes?, can't find the RFC 
> reference). I know that some spam filters like SpamAssassin can 
> generate a large large output, so we might want to consider an ability 
> to split headers into multiples to get around the limit.

Data I am aware of as desirous to include in a downstream filter at this 
time would be
a score heuristic btn 0 and 100, a not-junk score heuristic btn 0 and 
100, a filter name,
a filter-adjustment is-junk e-mail address, a filter-adjustment 
is-not-junk e-mail address
(the last two are highly optional and might be best dclared as 
configuration options rather
than inviting further chaos -- in fact they only matter if upstream 
communication has to
fall back to sending the whole e-mail somewhere), an MTA identifier, a 
message ID

Informative details (such as SA's results of each test) still make sense 
to leave them
exactly as is, but summarize them with somethign formally similar to

    Filter-Tag: name=K-5-Age-Appropriate; P=80; Q=4;  
FEARI=http://example.k12/mailfilter/adjust

Authenticating the respondents who attempt to tune the filters, so 
vandals and outliers can
be ignored, is an issue to list for resolution in the upstream portion.

FEARI = Filter Engine Adjustment Resource Identifier

Multiple tagging engines in the same MTA could each add their own lines.


>
> The current Wiki link is here:
>
> http://www.shaftek.org/wiki/wiki.pl?Filtering_Standards
>
> I only put this into a Wiki because I was too lazy to type up an 
> Internet draft. The subgroup can decide as whole on any tool they want 
> - if you want to have one person do a document that's fine also. The 
> decision is yours. 


You give us permission

> The Wiki is pretty open as of now. That may change depending on how 
> this subgroup develops.

And you take it away

:-)



The wiki is excellent.  I had not used one before but I figured out
a little bit of how it works from examining the editable text of pages
that are there already.  I summarized what I determined without reading
any instructions here 
http://www.shaftek.org/wiki/wiki.pl?Filtering_Standards/QuickStartGuide

The "Issues Raised" document is too large.  It needs to be broken into
many smaller pages, one for each issue. In my opinion.



David Nicol


-- 
[email protected].
cat and buttered toast