frontend mupdate deadlock

Hi!

Since we migrated from 2.5 to 3.8 last year my frontends start piling up
primarily imap (and other?) processes from time to time (one frontend every
1-2 month) in high activity periods.

So it took a while to get some clue what's causing it. Yesterday I
recognized that many of the imap processes show "<uid> Status" as procinfo
(since I have a SETPROCTITLE build) instead of "<uid> Idle" what was the
first kind of "pattern" I detected.

After some digging through the logs and the source I came to the conclusion
that the affected frontends mupdate process seems to deadlock. It got some
fresh updates from the master in that time and then suddenly it stopped
processing them (compared to the other two frontends) and also stops
responding on the UNIX socket.

But mupdate did not crash so far and is visible in the process list with
state "Sl".

It seems that the UNIX socket is still open and the imapd processes which
call kick_mupdate() block waiting for answers from mupdate. At this point
it is a matter of time until the machines start swapping and thrashing.

The logs show nothing unusual until I stop the service.

When stopped I see a lot of imapds logging
kick_mupdate: can't connect to target: Interrupted system call

I was not able to pin the true cause of the deadlock yet. Since this never
happened with 2.5 and the code didn't change much maybe the cause is
something I changed in our setup for 3.8. The options are
*) twoskip instead of skiplist for mailboxes.db
*) improved_mboxlist_sort: 1
*) reverseacls: 1
*) prometheus_enabled: 1

Maybe somebody else with murder environment sees this as well and can help
tracking this down.

Greetings, Wolfgang
-- 
Wolfgang Breyha <[email protected]> | https://www.blafasel.at/
Vienna University Computer Center | Austria


------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/Tcbcaf0fdcb0bc67b-M5a39291cc062cd699140e520
Delivery options: https://cyrus.topicbox.com/groups/info/subscription