Inconsistent determination of TMDA config dir in various utililies

I have been happily using TMDA for several years, but only for incoming
mail.  I have wanted to use its outgoing mail features for a long time,
but I keep getting discouraged every time I try, because the
tmda-sendmail and tmda-ofmipd programs do not determine the user's TMDA
config directory in the same way that this is done for incoming
messages, and the mechanisms they use do not always fit well with my own
email set-up.  Furthermore, tmda-sendmail and tmda-ofmipd don't even
behave consistently with each other, in this regard.

The purpose of this message is to discuss this issue and to perhaps
encourage a dialog about how we can fix these inconsistencies in order
to make TMDA more useful.

First of all, I want to explain my own email setup.

I host a number of domains, and I use postfix as my MTA.  Email for all
users is kept under a central directory tree, as follows:

  /var/vmail/DOMAIN/USER

For example, for [email protected], the directory tree in which all
incoming email is stored would be this:

  /var/vmail/domain.tld/someone

All of these directories are owned by the "vmail" user and the "vmail"
group.  In other words, they all are associated with the same unix user
ID.  The HOME directory for the "vmail" user is "/var/vmail".  This is
one of the issues that causes me to have problems tmda-sendmail and
tmda-ofmipd.  I'll explain further, below.

But first, I want to finish the description of my email set-up.  I only
store email in the /var/vmail/DOMAIN/USER tree.  Any ancillary
user-specific files and directories (such as the .tmda directory)
are kept in a parallel tree named as follows:

  /var/vmusers/DOMAIN/USER

For example, if [email protected] has TMDA set up, the path to the
.tmda directory would be this:

  /var/vmusers/domain.tld/someone/.tmda

I am able to manage this by means of a few small changes in the
/etc/tmdarc file.

The reason I separate these trees is twofold: (1) I don't want meta-data
to be viewable or trashable via software that accesses the email itself;
(2) it makes maintenance and email administration much cleaner for me if
the data and meta-data are separated.

This structure is maintained within a MySQL database.

My set-up has been working well for me for years.

So now, we come to the issues I have with the TMDA software.  First,
let's look at tmda-ofmipd.

While incoming tmda software uses /etc/tmdarc to determine where to look
for a user's .tmda directory, tmda-ofmipd seems to not use the data in
this file in the same way, if at all.  Rather, it seems to use different
mechanisms.  There are four of these, although the only one which works
on my site is the one managed by the --vhome-script parameter.  Here's a
description of all four mechanisms, and the my reason for why
--vhome-script is the only useful parameter:

  1. Default to the ~user/.tmda directory.  This doesn't work under my
     environment, because tmda-ofmipd uses this python call to determine
     this location:

       os.path.join(os.path.expanduser('~'), '.tmda')

     This always gives /var/vmail/.tmda on my system, because as you'll
     recall, all email users have the same "vmail" user ID.

  2. The --vdomains-path option.  I can't use this on my system, because
     I don't use qmail nor vmailmgr.

  3. The --configdir option to determine the location of the .tmda
     directory.  However, this doesn't work on my system, either,
     because for any given email user "someone", the .tmda config
     directory is located as follows:

       CONFIGDIR/someone

     where CONFIGDIR is the argument to the --configdir parameter.  For
     example, if we have this

       --configdir /var/vmusers/tmda

     then tmda-ofmipd will use the following directory to find the TMDA
     config file:

       /var/vmusers/tmda/someone

     This is clearly different from the standard TMDA directory that
     is used for incoming mail on my system, namely this:

       /var/vmusers/domain.tld/someone/.tmda

  4. The --vhome-script parameter.  This works on my system, because I
     can cause it to generate the proper directory path to my TMDA
     config directory.  The tmda-ofmipd utility makes use of the
     Util.getvuserhomedir in order to resolve this directory name.
 
So far so good.  However, in the case of tmda-sendmail, I can't use
--vhome-script, because that option is not even offered in
tmda-sendmail.  Furthermore, instead of using Util.getvuserhomedir, that
utility uses Util.gethomedir in order to locate the .tmda directory.
That routine uses pwd.getpwdnam to get the path to HOME.  This means
that on my system, there is no way that tmda-sendmail can be made to
point to the same TMDA config directory that tmda-ofmipd uses.

In other words, there are at least three separate mechanisms for
calculating a user's TMDA configure directory that are currently in use
in TMDA:

1. For incoming messages, code and variables in /etc/tmdarc are used.

2. In tmda-ofmipd, /etc/tmdarc seems to be ignored, or if it's loaded,
   this is done at the wrong time in the program startup sequence.  This
   forces me to use one of the four mechanisms listed above, of which
   --vhome-script is the only one which works on my system.

3. In tmda-sendmail, /etc/tmdarc seems to be ignored or loaded at the
   wrong time, and to make things even worse, I can't use
   --vhome-script.

I'm sad to say that this is a mess.

In order to fix this, I recommend the following changes:

A. All TMDA utilities should load /etc/tmdarc at the appropriate time in
   order to determine the location of the TMDA config directory.

B. All TMDA utilities, especially tmda-sendmail, should make use of
   --vhome-script and --configdir in exactly the same ways that
   tmda-ofmipd uses them.  In fact, all these utilities should use the
   _exact_ _same_ logic for locating the TMDA config directory for a
   given user.  This sould be equally true for utilities used for
   incoming mail (such as tmda-pending, tmda-filter, etc.) and
   outgoing mail (such as tmda-ofmipd, tmda-sendmail, etc.).

C. In all TMDA utilities, only make calls to os.path.expanduser('~'),
   pwd.getpwdnam and other similar functions as a last resort, if all
   other higher-level mechanisms fail.  And again, always use the
   _exact_ _same_ logic for deciding when to call these functions
   and how to use their results, no matter what TMDA utility is in
   use.

In other words, I am advocating maximal consistency among all the
TMDA utilities with regard to something as basic and important as
locating the TMDA config directory.

I'm eager to hear your opinions and ideas concerning these issues.


-- 
 Lloyd Zusman
 [email protected]
 God bless you.