Re: frontend mupdate deadlock

Hello,

> Since we migrated from 2.5 to 3.8 my frontends start piling up processes from time to time.
> Maybe the cause is something I changed in our setup for 3.8. The options are
> *) improved_mboxlist_sort: 1

In Cyrus IMAP 3.6 and later versions the imapd.conf option improved_mboxlist_sort does nothing.

This is stated correctly at https://www.cyrusimap.org/dev/download/release-notes/3.6/x/3.6.0.html .  This is not articulated at https://www.cyrusimap.org/3.6/imap/download/release-notes/3.6/x/3.6.0.html .

All that said, this particular option is not the cause here.

Greetings // Дилян

-----Original Message-----
From: Wolfgang Breyha via Info <[email protected]>
Reply-To: Info <[email protected]>
To: [email protected]
Subject: frontend mupdate deadlock
Date: 10/04/26 14:43:55

Hi!

Since we migrated from 2.5 to 3.8 last year my frontends start piling up
primarily imap (and other?) processes from time to time (one frontend every
1-2 month) in high activity periods.

So it took a while to get some clue what's causing it. Yesterday I
recognized that many of the imap processes show "<uid> Status" as procinfo
(since I have a SETPROCTITLE build) instead of "<uid> Idle" what was the
first kind of "pattern" I detected.

After some digging through the logs and the source I came to the conclusion
that the affected frontends mupdate process seems to deadlock. It got some
fresh updates from the master in that time and then suddenly it stopped
processing them (compared to the other two frontends) and also stops
responding on the UNIX socket.

But mupdate did not crash so far and is visible in the process list with
state "Sl".

It seems that the UNIX socket is still open and the imapd processes which
call kick_mupdate() block waiting for answers from mupdate. At this point
it is a matter of time until the machines start swapping and thrashing.

The logs show nothing unusual until I stop the service.

When stopped I see a lot of imapds logging
kick_mupdate: can't connect to target: Interrupted system call

I was not able to pin the true cause of the deadlock yet. Since this never
happened with 2.5 and the code didn't change much maybe the cause is
something I changed in our setup for 3.8. The options are
*) twoskip instead of skiplist for mailboxes.db
*) improved_mboxlist_sort: 1
*) reverseacls: 1
*) prometheus_enabled: 1

Maybe somebody else with murder environment sees this as well and can help
tracking this down.

Greetings, Wolfgang

------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/Tcbcaf0fdcb0bc67b-Mdef360cb8b55363e5303b041
Delivery options: https://cyrus.topicbox.com/groups/info/subscription