Re: Realisticness of header rearrangement

david nicol wrote:
> On Sat, 2004-04-03 at 21:47, Philip Miller wrote:
>>[ ... potential] problems if some
>>provider not directly controlled by the final recipient uses MTA software
>>that rearranges headers.
>>
>>>ieee, doing the tagging, will have a non-rearranging outbound MTA that
>>                                        ^^^^^^^^^^^^^^^
>>That's the statement I don't agree with. Because the standards don't require
>>non-rearrangement, we should not depend on that behavior.
> 
> We are proposing a standard, therefore we can include "these do not
> rearrange" as part of the standard.

We can require that MTAs with their associated filters implementing this 
standard not rearrange headers, but there's nothing we can do about 
intermediate servers that don't conform in any way to the new spec.

>>I am not opposed to signing or checksumming mechanisms as an optional
>>extension. However, as the sole method of authenticating a tag header, I am
>>opposed to them. I am opposed to them as the sole authentication method
>>because they are much more expensive to compute than copying the date stamp, 
>>and that could needlessly hinder adoption by large service providers.
>>
>>>I think a key included in both Received and Tagged is superior
>>>to reusing the timestamp.  Tagged would have two timestamps, if
>>>there was a delay, and that would be confusing.  Received usually names
>>>the receiving agent and domain, so Tagged could use the same identifying
>>>information.

Forgot to say something about this earlier: If we use the Received timestamp 
in the tag header, we only need to change the filters adding that header. On 
the other hand, if we require a unique key to match in Received and a tag, 
then we need to modify MTAs to include that tag. That could greatly increase 
the cost of implementation.

>>As far as I can tell, we are trying to design this header so that the vast
>>majority of its use will be in a parsed form by a GUI MUA that presents the
>>contained information to the user in a useful format or processes the mail
>>as a result of the tags attached. MUA programmers are quite capable of using
>>two separate variables independently.
>>Anyone reading headers, who we can assume is a fairly advanced user, will
>>learn pretty quickly that this one has 2 time stamps, and they are used for
>>different things.
> 
> And the second use isn't a time stamp any more, therefore its confusing. 

It may be confusing, but the people who will be seeing it are not the type 
that are easily confused.

> An explicit event-ID would be clearer (not confusing as a timestamp being
> used for something other than marking the time something happened)
> A filter-event-id, re-use the messageID or make up a new code, could be
> protected by vendor-specific checksumming, could be listed in an
> independently verifiable database accessible via another channel,
> etcetera,  Yes you can do all this with a timestamp but if processing
> more than one message per second a timestamp/messageID tuple would be
> required anyway.

If some way to individually identify the specific message and the processing 
it underwent were desired, that can easily be an extension. However, simple 
authentication that the processing was done after a certain point in the 
transport process does not require something so complex. I'm looking for the 
least-cost method to provide that authentication.

>>I don't think we need any particular way to sign headers. It's an expensive,
>>silly proposition with limited use.
> 
> I want to add an "out of scope" section on the wiki, unless someone
> beats me to it

That someone probably won't be me.

>>And I think they should, but for debugging and trust reasons. The tuple
>>(hostname, received-timestamp) in a tagged header authenticates that said 
>>tag was added with or after the Received header it matches. The user can 
>>choose to trust or not trust the judgement made at that point in the system.
>>Note: when I say 'user' here, I mean 'the user or the software under the
>>user's direct control'. That would be the MUA, and in my case, my private
>>email server, fda.zemos.net.
>>
>>>Yes, I trust that Philip Miller uses a secure connection to upload
>>>e-mail to zemos.net; that zemos.net uses amavis anti-virus software;
>>>that zemos.net then passes its e-mail to comcast.net, which forwarded
>>>the message to the asrg mailing list on iecc.com, where qmail copied it
>>>a couple times before sending me a copy from a VERP return address.
>>
>>The first half of that trust is misplaced. For all you know, I connected
>>directly to the iecc.com MX and forged all the prior headers.
> 
>>As long as I had the right IP address,
> 
> But you didn't, and the mail2.iecc.com server looked up your IP
> address.  There are a lot of subtle interactons that can make
> forged headers look fishy.  Other working groups are trying to
> formalize an expert system about fishiness of received headers,
> unless I have gist of the the charter of the path verification working
> group wrong.

That's fine, but unless we expect every filter in the world to implement 
such a (pie-in-the-sky) expert system, we need to provide the simplest 
possible trust-worthy authentication in the base standard.

>>Basically, it's impossible to trust headers that were added on 'my side' of
>>the mailing list server.
> 
> If relays on the receiving side don't rearrange headers, and I am
> expecting to see a tag, I can rely on the relays closer to me to not
> rearrange the headers and the last tag added will be first, so I won't
> even see your tag, or a forged tag, because it will be hidden by the
> earlier tag.

And if someone on your side of the transaction switched software (we'll say 
the mailing list server) to a system that does rearrange, what then?
Since it's cheap and easy for us to make rearrangement a non-issue with no 
change to non-implementing systems and minimal complexity for implementors, 
we should.

> Just an off-topic note, I have received the first piece of junk mail
> in my pay2send.com address ever, today, it seems that a junk mailer
> has amavis-perl tag all outgoing messages with a X-Virus-Scanned:
> header, which matched my far-too-trivial magic phrase "perl."

That brings to mind one of the few spams that got through to Paul Graham as 
he was writing about his experiences with Bayesian filters. It was from a 
company offering contract programming services, and named all sorts of 
languages, some of them obscure enough to appear regularly in his 'ham' corpus.

Back on topic: However, you haven't considered the case of this actually 
being a virus, which could have been marked 'safe' by some obscure feature 
of your MUA because it had this header. If you opened it and got infected, 
that's much worse than a simple spam getting through and getting deleted.
With even the simplest authentication of where a tag was added, MUAs with 
such features could make it easy for users to designate the 'trust cut-off 
point'. ISPs could tell users what to answer in their setup instructions, or 
this could even be integrated with the mailbox server, e.g. as an IMAP 
extension specifying where mail comes into the ISP.

> So the issue of tags added at sending of junk is very much real.  Were
> I looking for an X-Virus-Scanned header with software that escaped other
> X-Virus-Scanned headers that might be there already, perhaps turning
> them into X-Mycompany-Hidden-Virus-Scanned, the one added on sending
> would be ignored due to being rewritten; if my software that read for
> that header only took the first matching header (which would require
> altering current MUAs, that do not differentiate between different
> instances of the same header AFAIK), a matching header placed by my
> software would hide a matching header placed by the sender.

Escaping or rewriting them is one thought, but that requires much heavier 
processing, is error-prone, and needlessly complicated. If we can make it 
simple to distinguish them, why shouldn't we?

>> Making sure that tag headers are uniquely
>>associated with specific Received headers means that you can choose exactly
>>which step to start trusting.
> 
> no argument with this, except that a relay or processing step
> administration is what we want to associate with.  Associating with a 
> Received: header seems to me to be a classic case of confusing the name
> of a thing for the thing.

I agree with you on this point, but I don't think there's much we can do 
about that without specifying something much larger, such as a way to 
authenticate the administration cited in the tag. I think this association, 
in this context, is perfectly acceptable.
It would make sense to acknowledge that point in our standard, and show that 
it's not an issue because any such administration would have some way of 
communicating to users what associates them with specific Received headers.

Side thought: the most secure way I can see for an ISP to do this would be 
to setup one or more front-end MTAs to act as the MX, and have those forward 
to internal hosts who would do that actual processing. With that design, the 
internal hosts would be the first trusted hop, and forgery would be 
completely impossible.

>>>Tagged:  Host=(fda[127.0.0.1]); Tagger=amavisd-new; Tue, 30 Mar 2004
>>>20:24:38 -0500 (EST); Result=CLEAN
>>
>>I would write it more as follows:
>>
>>Tagged: Received-from=zemos.net (phil [10.0.0.2]); Received-time=Tue, 30 Mar
>>    2004 20:24:38 -0500 (EST); Host=fda.zemos.net; Tagger-name=amavisd-new;
>>    Tagger-version=xxxxxx; Virus=no; Tagger-time=Tue, 30 Mar 2004 20:25:42
>>    -0500;
>>
>>Note the following:
>>1. Everything is a key-value pair, timestamps and all.
> 
> Received lines tend not to follow this format.  They tend to
> use barewords (by, from, with) separated with whitespace to
> identify the parts of the header.  I would like to issue
> a reccomendation to update SMTP to make received line syntax
> conform to something too.  Too much coffee?

There's no way we can hope to change how SMTP does things at this point. 
However, the trace fields were designed in an era in which the protocols 
were experimental, and the headers were just as likely to be examined by 
human eyes as machines. We are designing a header to be efficiently produced 
and processed by computer, and key-value pairs are probably the best 
compromise between that and readability.

>>2. This tag was provably added after a particular receipt in the mail
>>transport system. It's ambiguous
> 
> I'd like an explicit Received-ID or Action-ID or tagging-ID to use
> instead of timestamp.  Using timestamp for ID seems to me to be an
> ugly kluge instead of an elegant reuse of existing information.  This
> may be a purely aesthetic opinion.

As a matter of aesthetics, I agree with you.
As a matter of engineering, I think it's quite elegant to pull cheap 
authentication of a newly added header practically from thin air, without 
harming backward compatibility in any way.

There is another reason to have an explicit counter.
Consider an ideal network, transporting messages instantaneously between 
MTAs that process them in 0 time. All of the timestamps will be the same 
(assuming that they all synchronize to the same clock). If things get 
reordered in this peachy-looking future, we have a problem which such an 
explicit counter would solve.
However, adding this counter requires a lot more than copying the Received 
timestamp. Using the Received timestamp means looking at the top Received 
header, which will be at the top, as required by RFC 2821. Finding a counter 
value greater than all previous values requires parsing every previous tag 
header, some of which could have been rearranged downward by non-conformant 
MTAs.

>>The timestamp in yours matches the timestamp in the Received lines, but I
>>think it would be useful, for debugging, if nothing else, for tags to
>>include the time they were added, as it may differ by quite a while from the
>>time of receipt.
> I agree
>> In the case of resource-intensive scans, such as virus and
>>statistical or DNS-lookup-based scanners, being able to attribute a delay to
>>a specific scan is a good thing.
> I agree

I'm glad we agree on these two points.

If someone comes up with a more aesthetically pleasing way to match tag to 
Received header, with similar or better functional characteristics than the 
Received-timestamp, I'm all ears.

Philip Miller