Re: Please explain -qq

Hi

I tried -qq again, and again, got abysmal results! Settings used:

 >     Queue option: -qq5m
 >     deliver_queue_load_max = 20 (same)
 >     queue_run_max = 5

Observations:

Start: all 22k emails received and queued.

2 mins in: load was 3.5, delivery slow (~100). Increased queue_run_max 
to 10.

7mins in: Still only delivered 500 emails. Load is 9-10.

25mins in: Still only delivered ~800 emails, load still over 10.

At this point I decided to abandon the -qq method because my client 
needs their emails sent. So I swapped back to -q and experimented with 
the number of queue runners.

Findings:

Using systemctl stop exim4 does *not* kill all the queue runners started 
by the main exim process(!). The result of this was that many -qq 
processes continued to run and hog CPU.

On a 2GB VM, I could run 117 -q queue runners at once (I started most of 
these manually), with load still around 6-8 (all CPU, 0.3% io, 80% RAM 
utilisation). With this many it was delivering 2k - 3k emails/minute.

I think tomorrow I will try

>     Queue option: -q30s
>     deliver_queue_load_max = 16
>     queue_run_max = 120

It still feels like something is odd with the -qq method.

I did not find anything useful to me in the output from exiwhat. See 
below. The pattern I observe is that for every job ID there's 3 
processes, one is "running queue: waiting for X", then there's 
"delivering X: waiting for a remote delivery subprocess...", and a 
"delivering X: to ..." one.

I guess I don't need -qq to work if I can get enough throughput with -q, 
it's just that I like to understand things and to have them do their 
work efficiently and -qq sounded promising on these fronts.


I'll keep experimenting. Any further input is still appreciated, but I'm 
grateful for the time you've already given.

Thanks

Rich


-----exiwhat output-----------8<--------------
1581083 running queue: waiting for 1vVWmN-004MVu-2N (1587691)
1582021 running queue: waiting for 1vVWbv-004MVu-0S (1587955)
1582043 running queue: waiting for 1vVWgk-004MVu-1M (1587949)
1582062 running queue: waiting for 1vVWlK-004aYX-1A (1587880)
1582074 running queue: waiting for 1vVWae-004MVu-0y (1586893)
1583007 running queue: waiting for 1vVWmG-004aYX-1c (1587952)
1583225 running queue: waiting for 1vVWcd-004MVu-2h (1587938)
1583251 running queue: waiting for 1vVWj7-004MVu-1h (1587941)
1583271 running queue: waiting for 1vVWmD-004aYX-1W (1587820)
1583288 running queue: waiting for 1vVWfE-004MVu-1i (1587914)
1583307 running queue: waiting for 1vVWfF-004MVu-1P (1587932)
1583320 running queue: waiting for 1vVWgB-004MVu-2m (1587943)
1586893 delivering 1vVWae-004MVu-0y: waiting for a remote delivery 
subprocess to finish
1586897 delivering 1vVWae-004MVu-0y to smtp.ufabc.edu.br 
[2801:a4:fabc:1034::111] REDACTED
1587691 delivering 1vVWmN-004MVu-2N: waiting for a remote delivery 
subprocess to finish
1587693 delivering 1vVWmN-004MVu-2N to mail.protonmail.ch 
[185.70.42.128] (REDACTEd
1587820 delivering 1vVWmD-004aYX-1W: waiting for a remote delivery 
subprocess to finish
1587855 delivering 1vVWmD-004aYX-1W toREDACTED [148.163.138.229] REDACTED
1587880 delivering 1vVWlK-004aYX-1A: waiting for a remote delivery 
subprocess to finish
1587882 delivering 1vVWlK-004aYX-1A to 
hotmail-com.olc.protection.outlook.com [52.101.41.5]REDACTED
1587914 delivering 1vVWfE-004MVu-1i: waiting for a remote delivery 
subprocess to finish
1587916 delivering 1vVWfE-004MVu-1i to 
hotmail-com.olc.protection.outlook.com [52.101.41.5]REDACTED
1587932 delivering 1vVWfF-004MVu-1P: waiting for a remote delivery 
subprocess to finish
1587934 delivering 1vVWfF-004MVu-1P to gmail-smtp-in.l.google.com 
[2a00:1450:400c:c07::1a]REDACTED
1587938 delivering 1vVWcd-004MVu-2h: waiting for a remote delivery 
subprocess to finish
1587940 delivering 1vVWcd-004MVu-2h to gmail-smtp-in.l.google.com 
[2a00:1450:400c:c07::1a] REDACTED
1587941 delivering 1vVWj7-004MVu-1h: waiting for a remote delivery 
subprocess to finish
1587943 delivering 1vVWgB-004MVu-2m: waiting for a remote delivery 
subprocess to finish
1587945 delivering 1vVWgB-004MVu-2m to gmail-smtp-in.l.google.com 
[2a00:1450:400c:c07::1a] REDACTED
1587946 delivering 1vVWj7-004MVu-1h to mx.vip.karoo.l.kcom.com 
[212.50.190.145] REDACTED
1587949 delivering 1vVWgk-004MVu-1M: waiting for a remote delivery 
subprocess to finish
1587951 delivering 1vVWgk-004MVu-1M to gmail-smtp-in.l.google.com 
[2a00:1450:400c:c07::1a] (REDACTED
1587952 delivering 1vVWmG-004aYX-1c: waiting for a remote delivery 
subprocess to finish
1587954 delivering 1vVWmG-004aYX-1c to gmail-smtp-in.l.google.com 
[2a00:1450:400c:c07::1a] REDACTED
1587955 delivering 1vVWbv-004MVu-0S: waiting for a remote delivery 
subprocess to finish
1587957 delivering 1vVWbv-004MVu-0S to aspmx.l.google.com 
[2a00:1450:400c:c1d::1b] REDACTED

---------------->8----------------------------------

-- 
## subscription configuration (requires account):
##   https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
##   [email protected]
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/