Re: AW: AW: AW: [EXT] Doveadm and iterate query

>>>>> "John" =3D=3D John Fawcett via dovecot <[email protected]> writes:

> On 24/07/2026 08:13, Aki Tuomi via dovecot wrote:
>>> On 24/07/2026 08:13 EEST Aki Tuomi via dovecot <[email protected]> wr=
ote:
>>>=20
>>>=20
>>>> On 21/07/2026 01:00 EEST John Fawcett via dovecot <[email protected]=
> wrote:
>>>>=20
>>>>=20
>>>> On 20/07/2026 16:53, MK via dovecot wrote:
>>>>>>>>> [snip]
>>>>>>>>>=20
>>>>>>>> Hi
>>>>>>>>=20
>>>>>>>> it appears to be a bug in dovecot 2.4.x when iterating thousands o=
f users.
>>>>>>>>=20
>>>>>>>> See
>>>>>>>> https://dovecot.org/mailman3/archives/list/[email protected]/mes=
sage/P5QBZAYXNAMNYHYA3BRLN4J3GUKX26WP/
>>>>>>>>=20
>>>>>>>> The fix I posted there did bypass the issue during a test I did. I=
 am
>>>>>>>> not sure it could be treated as a definitive fix. Another workarou=
nd is
>>>>>>>> to iterate outside dovecot.
>>>>>>>>=20
>>>>>>>> John
>>>>>>> Hi,
>>>>>>>=20
>>>>>>> did you get a feedback from someone of the dovecot developer? Since=
 your request is 4 month ago.
>>>>>>> So this sounds like my used workarround seem to be the best we can =
do at the moment, as Philip did the same.
>>>>>>> Do you have experiences if this works with about 40k users or more =
too? (doveadm user '*', cache this and then execute doveadm -u for each use=
r)
>>>>>>>=20
>>>>>>> Oliver
>>>>>>>=20
>>>>>>>=20
>>>>>>> __________
>>>>> Hi Oliver
>>>>>>=20
>>>>> so far I haven't seen any follow up from the development team.
>>>>> It's bad that no one is taking care of it.
>>>> I guess it is not one of the high priority issues at the moment.
>>>>> In my experience doveadm user '*' also runs into the same trouble for
>>>>> large numbers of users. I got issues with even 20K users on 2.4.x.
>>>>> At the moment I'm using it for ~ 4400 user and have no problems. I'm =
curious to see how it works with 40k users.
>>>>>=20
>>>>> Where the users are in a mysql database, you could query the db direc=
tly
>>>>> to get the usernames and then execute doveadm -u for each user.
>>>>> Sure, that's an option if the doveadm iterate did not work for that m=
utch user.
>>>>>=20
>>>>> Thanks
>>>>> Oliver
>>>>>=20
>>>>>=20
>>>> Hi all
>>>>=20
>>>> this is a revised fix. This looks like the minimal fix. The alternative
>>>> is to refactor the auth-master-connection code to make it compatible
>>>> with the 2.4 throttling logic. Setting output_throttle_size to zero wi=
ll
>>>> then just use the same throttling mechanism as there was in 2.3.21,
>>>> which is still present in 2.4 for this code path. The problem arises
>>>> when both mechanisms are active.
>>>>=20
>>>> John
>>>>=20
>>>> diff -u dovecot-2.4.4/src/auth/auth-master-connection.c.orig
>>>> dovecot-2.4.4/src/auth/auth-master-connection.c
>>>> --- dovecot-2.4.4/src/auth/auth-master-connection.c.orig 2026-07-20
>>>> 22:55:09.153747336 +0200
>>>> +++ dovecot-2.4.4/src/auth/auth-master-connection.c=A0 =A0 2026-07-20
>>>> 22:55:32.423163048 +0200
>>>> @@ -740,7 +740,7 @@
>>>> =A0 =A0 =A0.dont_send_version =3D TRUE,
>>>> =A0 =A0 =A0.input_max_size =3D MAX_INBUF_SIZE,
>>>> =A0 =A0 =A0.output_max_size =3D SIZE_MAX,
>>>> -=A0 =A0 .output_throttle_size =3D MAX_OUTBUF_SIZE,
>>>> +=A0 =A0 .output_throttle_size =3D 0,
>>>> =A0};
>>>>=20
>>>> =A0static int
>>>>=20
>>> Sorry for missing on this, we'll take a look at the problem and the pro=
posed fix.
>>>=20
>>> Aki
>>>=20
>> This bug is already fixed in 2.4.3 with https://github.com/dovecot/core/=
compare/41b244c9%5E...d4362583.patch
>>=20
>> Aki
>>=20
> Thanks Aki

> you're right it is solved in 2.4.3. That surprised me since I was still=20
> seeing iterate errors on 2.4.4. and I wrongly assumed it was due to the=20
> same cause. Looking into it I am now sometimes hitting a timeout of 155=20
> seconds. This is with 20K users configured. It takes approx 140 secs for =

> "doveadm quota get -A" with users in mysql, but turning on the debugging =

> tends to bring it over the 155 second limit. I guess that with 30K users =

> I'd be always hitting the timeout.

I'm curious, is the loss in speed because the quota is recalculated
for each user when this is run?  Or is your mysql query just slow?

If it's the first, I wonder if there is a way to optmize things by
keeping a quota cache in memory for the most recent users, and for
those inactive users, just keep it in a mysql table so you're not
scanning the disk all the time to recalculate. =20

But of course I don't know how this all works.  I'm just thinking that
20k records in mysql should be pretty damn quick to return honestly.=20

> Seems each user iteration is taking a bit longer in 2.4.4 than on 2.4.1. =

> If I get time I'll see if I can put some concrete numbers around the=20
> difference.

_______________________________________________
dovecot mailing list -- [email protected]
To unsubscribe send an email to [email protected]