Re: [apps-discuss] Comments on Malformed Message BCP draft

Tony Finch wrote:
> Dave Cridland <[email protected]> wrote:
>> It may be that the discussion suggests rejecting, in which case I suggest the
>> document should clearly explain why, and what the implications of not doing so
>> are, beyond "it makes some problems harder to diagnose".
> 
> It does not make sense to have a uniform policy for dealing with corrupt
> messages. Some kinds of corruption are caused by common legitimate
> software, in which case you will want to treat it leniently; others are
> caused by malware or rare kinds of incompetence, in which case it makes
> more sense to reject. You can only determine which is which based on
> operational experience, and the least-worst response changes from time to
> time.

+1.

My view is that since only Good Guys complain, Bad Guys do not, there 
are some well established areas for strong compliancy checks that has 
proven to have a high efficiency bad mail filtering rate with little 
false positives for good guys. This is because most good guys are 
compliant, especially for these particular areas of compliancy. 
Others are more subjective, so it generally it becomes a decision to 
how its enabled - on/off by default.

Traditionally, the transport is a passive payload transfer system but 
has have options for decisions than it does in applying subjective 
decisions related to the payload.

I think good examples are:

SMTP LEVEL:

   - anal strong syntax checks, i.e, "MAIL FROM:space"

   - Checking for explicit EHLO [ip literal] matching connection IP
     requirement. Exceptions used for growing NATS related issue
     in SOHO market with more than 1 PC.  PORT 587 can be used
     to skip this check.

   - Return path verification - methods in debate, but it is
     increasingly recognized that the validity requirement
     has become more important in the area of bounce avoidance.

822/2822/5322 PAYLOAD LEVEL

The first thing here is making sure we have a valid message per RFC 
822/2822/5322.  I listed all three because it depends on the receiver 
on how it  determine a valid message.

First, many systems don't need any RFC headers at all.  Just the body 
and will automatically fill in required RFC fields:

    From:          <--- SMTP Return Path
    To:            <--- SMTP Forwarding path
    Date:          <--- Reception Timestamp

822 systems require From:, To/Cc:, Date:, 2822 systems relaxed it to 
From:, Date and it remains this way with 5322 systems.  I recall 
reading long ago an I-D by Keith Moore to even further relax it to 
remove the From: requirement.  That would of been terrible.

Anyway, when some of these are missing or none, the design question 
becomes whether the missing headers should be filled in or rejected, 
always or sometimes.  For some, the answer was straight forward, for 
others it was not.

I will note that with RFC2821bis/RFC2822bis, we explored enabling 
stronger compliance (default on) and almost immediately, we got a few, 
not a lot, reports and that was enable the default to a FILL IN to 
avoid any future support overhead.   I believe it was 2-3 reports 
only, and I recall one report because the issue here was scenarios 
where two identical list messages (by content and context) for two 
users at a host and one fail and the other did not was found to be 
because both took different mail processing distribution paths.

I honestly don't recall the details of the incorrect headers, but the 
main thing was how the differences occurred.  Following the trace of 
both, it was clear the sender was using two path distributions, where 
one path was using  the original non-compliance software path and the 
the second was a new beta software path (I recall the "Beta" term in 
Received lines).  So it appeared the sender was aware of the 
distribution problem and was migrating to more RFC5322 compliant 
software.  Of course, while this is all interesting to the customer, 
he could care less and wanted a solution now.  I forget what was done 
here.

Anything beyond this has not really been a big issue for us. Some of 
the examples listed in the report are interesting to note, but I can't 
see a real justification to add time and energy to automate 
corrections when the most practical reality by far is majority of the 
good guys are compliant.  If an issue does occur, and the issue is 
related to broken legacy software, then you have no choice by to help 
solve the problem.  But if the offending software is active, then you 
have to weigh all the factors on how to feasibly serve the customer. 
Severity and frequency of problem. It is more than one, is the 
offending active vendor aware and have solution you can research. Does 
the customer have access to the offending senders?  Can you provide a 
work around, etc, etc, etc.

Just consider the report malformed message scenarios with the space 
for colon, this will require you to change YOUR software, not trivia 
for everyone, when parsing RFC headers to one that checks for two 
conditions:

     "field-name:"
     "field-name :"

I am not sure I really want to do that.  It will fall into the same 
engineering compliance debates with SMTP space issues:

     Mail From:<address>
     Mail From: <address>

The difference here is that it was well known that the #1 MUA Outlook 
was doing this and most systems could not ignore that fact.   Today, I 
believe these MUAs were corrected and you don't expect it, so you can 
probably get away (to a higher degree) with enforcing the syntax.

But do we have such a high level of the malformed 5322 header with a 
space in it, in particular from any big brand vendor?  I'm not seeing 
this.

-- 
HLS