Re: Qmail-Specific?

> On 15 Feb 2010, at 15:58, Bruce Guenter wrote:
>> On Mon, Feb 15, 2010 at 12:36:34AM +0000, Sabahattin Gucukoglu wrote:
>>> Actually, the biggest problem with doing message personalization in
>>> ezmlm is not connection caching or other delivery issues.  How do =
you
>>> reliably queue up large numbers of messages without starving other =
users
>>> of the disk and/or the queue (depending on how the MTA dequeues the
>>> messages)?  And what happens if the process is interrupted?  The
>>> complete set of messages needs to be effectively queued as a single
>>> transaction, and that can't happen when the messages are =
personalized
>>> outside of the MTA without some alternate kind of queueing process.
>>>=20
>> I would expect the implementation details of the MTA to cover your =
first two questions.
>=20
> That is what I'm asking.  How is it done?  Are there *any* MTAs that =
can
> queue up more than one email in a single transaction, such that (1) =
the
> process is efficient, and (2) they either all queue or all do not =
queue?
> I've never heard of such a thing.  I've seen one or two rather =
intrusive
> patches for qmail that would allow for arbitrary substitutions within
> messages, but at least the one was not public when I saw it.
>=20
If I were still on Sendmail, I'd solve the first with the DeliveryMode=3DQ=
 kludge.  It isn't perfect, but it's the established way to get =
high-throughput Sendmail.  Otherwise you'd have to change the submission =
protocol like SMTP to recognise multiple-message transactions as a =
single transaction, and that's not something *any* MTA is designed to =
do, by design, seeing as how transactions for multiple recipients are =
normally addressed per message.  The closest to such a thing in an =
existing MTA might be ETRN, with automatic queue processing switched =
entirely off.

I'm still not clear why you need special handling in the case of =
interruptions: if your submissions to the MTA are interrupted, start =
where you left off; if the queue run is interrupted, the MTA will simply =
begin where it left off.  Why do you want to ensure an entire VERP-style =
delivery is transactional?  I must be missing something.

>> The advantage of doing VERP is simply that less state has to be
>> maintained for a given list message, rather than a message for each
>> recipient of that message, namely, who the last person you sent mail
>> for was.
>=20
> This is partly true, but the amount of state required is more than =
just
> who the last person you sent mail to.  If the process is interrupted
> (server crash or reboot, SIGXCPU, stray OOM killer, whatever), what
> restarts it?  If there were subscribers added or removed, how do you
> preserve the state of the list that you had been sending from?  Once =
you
> consider all the corner cases, and consider the need for a protocol
> sender as described above, you may as well just add a retry mechanism
> and call it a dedicated MDA or mini MTA.
>=20
Oh, well, if it's so unimportant a consequence of a failure to have a =
retry occasionally send duplicates in exchange for subscription updates, =
I think the best thing for you to do is to simply never return a =
submission status of zero to the incoming job until you've made a VERP =
explosion to the submitting MTA.  Then, you can rely on the client =
coming back if the job doesn't succeed the first time around, or of the =
queue on the sending host to do the same thing.  You needn't keep your =
own spool, check it occasionally, run it consistently, in short, write a =
new MTA.  And, with an appropriate memory allowance, the MDA can read =
the message and subscribers into memory once, and blast with no further =
disk I/O.  Bonus!  You can go one further step, a daemon does the =
submissions, and knows the recipient list and receives updates; the MTA =
calls a stub which talks to that daemon, feeding a list message and =
awaiting a response, making disk I/O nearly zero during deliveries.

Cheers,
Sabahattin