Re: Some parsing/generation issues of email in Python 3

This is a multi-part message in MIME format.

--nextPart1804458.c4oWNNuu3I
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="iso-8859-1"

[This mail is intentionally hand wrapped..]

On Freitag, 10. Juni 2016 17:54:34 Hans-Peter Jansen wrote:
> On Donnerstag, 9. Juni 2016 18:37:54 Hans-Peter Jansen wrote:
> >=20
> > Let's see, how this goes.
>=20
> Hmm, compat32 and Python3 start to get in my way in no funny ways.

Okay, got it working now. It turned out to be a problem in the logging =
module.=20
Unfortunately, it's not reproducible outside postfix mail filter setup.=

=09
=09http://www.postfix.org/FILTER_README.html

I will try to explain this in short words, but you will not believe me,=
 as I=20
have a hard time to believe this myself..

When a python3 process is used in a simple mail filter setup, as descri=
bed in=20
the FILTER_README.html document, it is executed in a stripped down envi=
ronment.=20
As relevant parts, just LANG=3DC and PATH=3D/bin;/usr/bin is set.

The filter reads the mail from stdin, and calls sendmail for passing it=
 on,=20
again, with the mail on stdin..

Now, it takes a mail with some "higher" encoding (I'm using a utf-8 enc=
oded=20
subject containing german umlauts), and an attempt to log the subject l=
ine=20
(which has to be a log file for obvious reasons). Now take a save seat,=
 this=20
attempt of unicode logging results in a manipulation of the execution f=
rame,=20
the execution precedes a few instructions "below". Yes, I'm not kidding=
, no=20
escaped surrogates output, no error message, just no logging of the off=
ending=20
line, and this "esoteric" behavior. Be assured, that if I would still h=
ave=20
enough hair on my head, I would have teared it off completely by now.

This is reproducible for Python 3.4.4 on openSUSE 13.2/x86_64 here.=20

For the brave, who want to reproduce/investigate this issue, I'm attach=
ing=20
everything necessary. All others should stop reading now. Thank you.

Still with me, here we go: [$: root prompt]

A working postfix setup is implied. Stop all usual processing (fetchmai=
l, ...)

$ useradd --gid mail mfilter
$ cat >> /etc/postfix/master.cf << EOF
mfilter   unix  -       n       n       -       1       pipe
  flags=3DRq user=3Dmfilter argv=3D/path/to/mail_filter_test.py -f ${se=
nder} -- ${recipient}
EOF

$ systemctl restart postfix
$ sendmail -f [email protected] [email protected] < umlaut-subject-2.=
mail
$ less +F /tmp/mail_filter_test.log

Defective output:
2016-06-12 15:36:28,540 [mail_filter_test] DEBUG: parse message
2016-06-12 15:36:28,543 [mail_filter_test] DEBUG: call ['/usr/sbin/send=
mail', '-G', '-i', '-f', '[email protected]', '--', '[email protected]=
s']

Output with SILLY_BEHAVIOR =3D 0:
2016-06-12 15:37:50,887 [mail_filter_test] DEBUG: parse message
2016-06-12 15:37:50,889 [mail_filter_test] DEBUG: subject: Wie deaktivi=
ere oder l=F6sche ich meine SprachBox IP der Telekom?
2016-06-12 15:37:50,890 [mail_filter_test] DEBUG: call ['/usr/sbin/send=
mail', '-G', '-i', '-f', '[email protected]', '--', '[email protected]=
s']

Note, that the subject line is missing. In my real filter, it left the=20=

current execution frame, and execution continued one or two level up th=
e
stack.

This reminds me at my assembler times (680x0 power, long ago!), where I=
 used
such tricks like modifying the program counter in "very special arrange=
ments".

I don't think, this is an adequate outcome of the attached code, do you=
?

I'm directing this here, while I know, this is quite off-topic in the r=
esult.
OTOH, Stephen and David discussed the LANG=3DC issues with Victor, and =
this=20
is one example, where this is very relevant to Python3 fitness to act a=
s=20
such a filter. Apart from fixing the logging, I'm encoding all paths an=
d file=20
names with a configurable encoding before calling the OS. Butt ugly, bu=
t=20
feasible.

I hope, that at least one of you is able to reproduce this, before we d=
ecide,
how to precede. I hope, I don't offend anybody here with that approach.=


Please speak up, if I should go away and search for another tree to bar=
k at.

Thanks,
Pete
--nextPart1804458.c4oWNNuu3I
Content-Disposition: attachment; filename="mail_filter_test.py"
Content-Transfer-Encoding: 7Bit
Content-Type: text/x-python; charset="UTF-8"; name="mail_filter_test.py"

#!/usr/bin/env python3

import sys
import getopt
import logging
import subprocess

import email
import email.policy
import email.header
import email.generator

logfile = '/tmp/mail_filter_test.log'
logformat = '%(asctime)s [%(name)s] %(levelname)5s: %(message)s'
encoding = 'utf-8'

# setup logging
log = logging.getLogger('mail_filter_test')

EX_TEMPFAIL = 75        # queue and retry
SILLY_BEHAVIOR = 1

if SILLY_BEHAVIOR:
    logging.basicConfig(
        level = logging.DEBUG,
        format = logformat,
        filename = logfile,
    )
else:
    log.setLevel(logging.DEBUG)
    filelog = logging.FileHandler(logfile, encoding = encoding)
    filelog.setLevel(logging.DEBUG)
    filelog.setFormatter(logging.Formatter(logformat))
    log.addHandler(filelog)


def decode_header(value):
    if value is not None:
        value = str(email.header.make_header(email.header.decode_header(value)))
    return value


def mail_filter(sender, recipients):
    log.debug('parse message')
    msg = email.message_from_binary_file(sys.stdin.buffer, policy = email.policy.compat32)
    if not sender:
        sender = msg.get('from')
    if not recipients:
        recipients = msg.get_all('to')
    subject = decode_header(msg.get('subject'))
    log.debug('subject: %s', subject)
    sendmail = ['/usr/sbin/sendmail', '-G', '-i', '-f', sender, '--'] + recipients
    log.debug('call %s', sendmail)
    p = subprocess.Popen(sendmail, stdin = subprocess.PIPE)
    email.generator.BytesGenerator(p.stdin).flatten(msg)
    p.stdin.close()
    return p.wait()


if __name__ == '__main__':
    try:
        optlist, recipients = getopt.getopt(sys.argv[1:], 'f:')
    except getopt.error as msg:
        print(msg, file = sys.stderr, flush = True)
        sys.exit(EX_TEMPFAIL)

    sender = None
    for opt, par in optlist:
        if opt in ('-f', '--from'):
            sender = par

    sys.exit(mail_filter(sender, recipients))


--nextPart1804458.c4oWNNuu3I
Return-Path: <[email protected]>
From: Hans-Peter Jansen <[email protected]>
To: Hans-Peter Jansen <[email protected]>
Subject: Wie deaktiviere oder =?UTF-8?B?bMO2c2NoZQ==?= ich meine SprachBox IP
 der Telekom?
Date: Sat, 11 Jun 2016 13:23:09 +0200
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain; charset="us-ascii"

https://www.telekom.de/hilfe/festnetz-internet-tv/telefonieren-einstellungen/sprachbox/sprachbox-ip/sprachbox-ip-deaktivieren-oder-loeschen?samChecked=true

--nextPart1804458.c4oWNNuu3I
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline