Re: message sent by detector has duplicate Message-Id header, rejected by SMTP server

Hi Dan:

In message <[email protected]>,
Dan Tenenbaum writes:
>
>(First of all, is this list still up? I saw some discussion of
>moving it, and very little traffic. If the list is elsewhere,
>please let me know where to go.)

I got your email at least so I claim the list is up.

>OK, I am trying to switch to a new SMTP email server, one provided
>by Amazon Simple Email Service (SES).
>
>Messages sent by roundup to this server bounce back to me with the error:
>
>DetectorError: Error: couldn't send email: (554, "Transaction failed:
>Duplicate header 'Message-ID'.")
>
>If I instead arrange to have the same email sent to a mailcatcher app
>so I can inspect the email, it does indeed have two (identical)
>Message-Id headers (note the difference in case between the actual
>headers and the error message (Message-Id vs. Message-ID); not sure
>if it is significant).

I claim not significant.

> [...]
>You can see that the Message-Id header, with identical value, appears twice.
>
>This message is sent by the detector called nosyreaction.py which I
>guess is a detector that is supplied with roundup. It may have been
>modified slightly since we started using it.
>
>I notice that the detector code has references to 'msgid' so I wonder
>if this gets translated to a Message-Id header at some point.

It doesn't.  msgid is the internal message id for the message (think
msgid as the handle in the database (msg1, msg200 ...) not the smtp
message id header.)

I have email logs from my development setup and I think I see the same
duplicate headers. However, my development setup isn't working at the
moment so I can't debug this issue.

Note, if you change the debug variable in the [mail] section of the
trackers config.ini:

  # Setting this option makes Roundup to write all outgoing email
  # messages to this file *instead* of sending them.
  # This option has the same effect as environment variable SENDMAILDEBUG.
  # Environment variable takes precedence.
  # The path may be either absolute or relative
  # to the directory containig this config file.
  # Default: 
  debug = /tmp/tracker_mail.log

it will write the email to /tmp/tracker.email_log, so you don't have
to use an email catcher.

>Any ideas for suppressing these duplicate headers?

Note that both Message-Id and Reply-To are duplicated. Message-Id has
the same value in both cases, but Reply-To is different.

It looks like the headers are added at roundupdb.py in the function
send_message. I see three occurrances of Message-Id in that
function. The first is around line 529 that reads:

            # message ids
            if messageid:
                message['Message-Id'] = messageid
            if inreplyto:
                message['In-Reply-To'] = inreplyto

Can you change that to read:

                message['MESSAGE-ID'] = messageid

and see if one of the headers changes case. If so that will tell us
that we are in the correct codepath. As a second test, change back the
capitalization at line 529. and change the case of one of the other
matches for Message-Id in that function. Hopefully this will tell us
where the first and second copies of the header are coming from.

Since the X-Roundup-issue* headers are added at line 554, I am
guessing the first Message-Id header comes from line 529.

However, unless crypt is in use, I don't see how the additional
headers are getting in there. The Message-Id header values are the
same in both cases, but Reply-To is different so even more confusion
on my part.

Hopefully changing the headers in the code (remember to restart the
server process between code changes) will narrow this down.

Sample message
>MIME-Version: 1.0
>Content-Type: text/plain; charset="utf-8"
>Reply-To: Someone at Somewhere <[email protected]>
>Message-Id: <[email protected]>
>X-Roundup-issue-files: spbtest_0.99.0.tar.gz
>X-Roundup-issue-priority: software package
>Content-Transfer-Encoding: quoted-printable
>Subject: [issue1] spbtest
>To: [email protected]
>From: a sender <[email protected]>
>Date: Fri, 21 Aug 2015 05:10:06 +0700
>Precedence: bulk
>X-Roundup-Name: the name of this tracker
>X-Roundup-Loop: hello
>X-Roundup-Version: 1.5.0
>Message-Id: <[email protected]>
>Reply-To: Our file submit tracker <[email protected]>
>
>here is the message body

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

------------------------------------------------------------------------------