Re: Dovecot 2.4 doveadm_worker_count corrupts tab-formatted output

Hi

I can confirm the issue of these iteration operation not completing for 
1000's of users is linked to:

MAX_OUTBUF_SIZE defined in auth/auth-master-connection.c

I made it 1024000 and managed to complete quota recalc for 10000 users. 
But that's not going to be a solution, since whatever it is set to 
provides a hard coded limit to how many users can be iterated for quota 
get, quota recalc and index operations. Surprisingly it is not a 
completely deterministic limit, since sometimes more users get processed 
than other times. I was looking for a time based limit before I thought 
of looking at the buffer size.

The strange thing is that the current value hasn't changed in more than 
20 years if ever, so that does not explain of itself why this issue 
arose now in 2.4.

#defineMAX_OUTBUF_SIZE(1024*50)

The likely explanation is that exceeding the buffer size is now being 
handled differently to before.

John

On 28/02/2026 15:35, John Fawcett via dovecot wrote:
>
> On 27/02/2026 17:56, Philip Iezzi via dovecot wrote:
>>   ...
>>
>>     Yes, I am definitely running into some kind of limit. It's not on 
>> MySQL
>>     level - tuned it a lot for high performance and connection 
>> hammering, and
>>     never found any error in MySQL log.
>>     Above workaround is now running for a full day, pulling all user 
>> quotas
>>     every 4mins, without running into this issue again.
>>     As `doveadm quota get -A` was always running fine on Dovecot 2.3 
>> and I can
>>     now (under Dovecot 2.4.2) retrieve all quotas by piping all users 
>> into
>>     `-F`, it should definitely be possible to fix/optimize the `-A`
>>     iterate_query for mailservers with larger user base.
>>     @Aki did you guys experience this issue and can I provide you any 
>> more
>>     information to help you get this fixed?
>>     I can definitely live with my workaround, just hope others are not
>>     struggling with it and wasting so much time as I did.
>>     Best regards,
>>     Philip
>>
>>
> Hi Philip
>
> this is reproducible by filling the usertable with around 2000 users 
> and running doveadm quota get -A. For me it also fails on
>
>      doveadm quota recalc -A
>
>      /usr/bin/doveadm index -A '*'
>
> As you already saw the process never completes and gives an error on a 
> different user each time though roughly somewhere between 2-3 seconds 
> into the execution for quota recalc.
>
> quota get seems to present slightly more variable behaviour in terms 
> of how long before it fails but that could be influenced by terminal 
> output time.
>
> The log entry I saw was "auth: Error: auth-worker: Aborted LIST 
> request for *: Shutting down"
>
> doveadm user '*' does complete ok, but that only does the iterate 
> query, whereas the other commands do the iterate and then lookup the 
> single users too.
>
> I don't see any configurable timeout limits that seem relevant to this 
> issue, so it looks like a bug when running commands needing to iterate 
> through 1000s of users with the iterate_query and then doing lookups 
> for the single users user the userdb query.
>
> John
>
> _______________________________________________
> dovecot mailing list -- [email protected]
> To unsubscribe send an email to [email protected]

_______________________________________________
dovecot mailing list -- [email protected]
To unsubscribe send an email to [email protected]