Re: daily response 18 mar 2004

On Mar 18 2004, David Nicol wrote:

> Hmm. we're trying to discuss e-mail filtering here, not exhaustive
> library indexing.

> MTA filtering: messages are delivered or blocked (or delivered and 
> tagged) (et cetera)
> MUA filtering: messages are filed (perhaps respecting tags placed by an MTA)
> 
> That is real, today, current practice. Big distinction.

Ok, I'll try again. Some filters have components that act during MTA
filtering (your definition above), and also components that act after 
MUA filtering (your def). This is true for any statistical filter
whose learning material is read directly from filed messages. Learning
occurs after MUA filtering, while tagging (etc.) occurs during MTA
filtering.

We all have most experience with the schemes we've come up with
ourselves, so it's natural for the discussion to polarize along those
lines.  There's no bestest solution. If what I say below seems biased,
that's because it is.

In your case, you do not filter or tag based on content at all:

> I am working on an aggressively filtering SMTP server. It refuses messages
> that have not been approved in advance by the recipients. This means that

It follows that you have no need to worry about modifications to the
content of messages, and it's probably counterproductive to discuss 
complex message modification protocols from you pov.

The statistical filters are totally content based. Some are installed
inside or just before the MUA, but others can be installed within an
SMTP server. 

Statistical filters need a lot of data (typically stored in a
database). This data comes from messages, which are either learned
during filtering, or learned after MUA filtering. Here are two
scenarios, many filters can handle both of these. bogofilter is the
canonical example.

Scenario 1: messages arrive, are filtered and learned at the MTA or
MDA level. What happens when the MUA boundary is crossed is
irrelevant. The statistics database is stored separately from the MUA.

This design has problems with feedback from users. Incorrect
classifications must be awkwardly communicated to the filter by
resubmitting the *exact same* message, character for
character. Alternatively, the filter must keep its own duplicate
message cache, separate from the MUA, so that the user need only refer
to a previous message somehow. In that case, the filter is evolving into a
message repository just like the MUA already is.

Scenario 2: messages arrive, are filtered only at the MTA/MDA level.
Learning is performed on messages scavenged from the MUA repository
(mbox files, maildirs whatever).

This design is nice for users, because the learning/feedback can be
tightly integrated with their user interface. It suffers from the
problem that by the time messages are learned, they may have been
modified, mangled, and otherwise become slightly "wrong".

I mentioned bogofilter because it can do both these scenarios, and
people really do use both. For scenario 1, there's a couple of
switches which update the database on the fly. For scenario 2, some
people relearn their bogo database from scratch every night, through a
cron job. 

Why do people use both scenarios? Because databases can get out of
whack, hence needing a rebuild, and because some people report better
results through complete relearning from scratch, and because
databases can fill up and become stale. Scenario 1 is nearly
maintenance free, but untended misclassifications will slowly sabotage
the database.

> okay, so you understand the differences between MUA/MTA

In summary, I still maintain that the difference between MUA/MTA is 
close to irrelevant for statistical filtering. It's like an overlay
which doesn't fit properly. Note that I've given you reasons which
have nothing to do with library management this time.


-- 
Laird Breyer.