Re: cyr_expire suddenly segfaulting

Nick Howitt <[email protected]>
Newsgroups gmane.mail.imap.cyrus
Message-ID <[email protected]> 

On 12/03/2017 21:09, Patrick Boutilier wrote:
> On 03/12/2017 02:24 PM, Nick Howitt wrote:
>> Hi,
>> I've been running cyrus-imap for a few years not. I am using 2.4.17-8 on
>> ClearOS (a Centos derivative). I've been running with "delprune
>> cmd="cyr_expire -E 1 -X 7 -D 7" at=0001" in cyrus.conf, but in the last
>> week or so it has started segfaulting. I've tried moving the command to
>> cron.daily "/usr/lib/cyrus-imapd/cyr_expire -E 1 -X 7 -D 7" but it still
>> segfaulting.
>>
>> In /var/log/maillog I get:
>> Mar 12 03:15:53 server cyr_expire[15065]: Repacking mailbox 
>> user.mail-ebay
>> Mar 12 03:15:53 server master[15064]: setrlimit: Unable to set file
>> descriptors limit to -1: Operation not permitted
>> Mar 12 03:15:53 server master[15064]: retrying with 4096 (current max)
>> Mar 12 03:15:53 server cyr_expire[15065]: Repacking mailbox
>> user.mail-ebay.Drafts
>> Mar 12 03:15:54 server cyr_expire[15065]: Repacking mailbox user.nick
>> Mar 12 03:15:54 server cyr_expire[15065]: Repacking mailbox 
>> user.ourfamily
>> Mar 12 03:15:54 server cyr_expire[15065]: Repacking mailbox
>> user.ourfamily.Drafts
>> Mar 12 03:15:54 server cyr_expire[15065]: Expunged 0 out of 9150
>> messages from 57 mailboxes
>> Mar 12 03:15:54 server cyr_expire[15065]: duplicate_prune: pruning back
>> 0.00 days
>> Mar 12 03:15:54 server ctl_cyrusdb[15068]: recovering cyrus databases
>> Mar 12 03:15:55 server ctl_cyrusdb[15068]: skiplist: checkpointed
>> /var/lib/imap/mailboxes.db (57 records, 4816 bytes) in 0 seconds
>> Mar 12 03:15:55 server ctl_cyrusdb[15068]: skiplist: checkpointed
>> /var/lib/imap/annotations.db (0 records, 144 bytes) in 0 seconds
>> Mar 12 03:15:55 server ctl_cyrusdb[15068]: done recovering cyrus 
>> databases
>> Mar 12 03:15:55 server master[15064]: unable to setsocketopt(IP_TOS):
>> Operation not supported
>> Mar 12 03:15:55 server ctl_cyrusdb[15073]: checkpointing cyrus databases
>> Mar 12 03:15:55 server ctl_cyrusdb[15073]: done checkpointing cyrus
>> databases
>> Mar 12 03:15:55 server imaps[15075]: error sending to idled: 3
>> Mar 12 03:15:55 server imap[15074]: error sending to idled: 3
>> Mar 12 03:15:55 server imap[15078]: error sending to idled: 3
>> Mar 12 03:15:55 server imap[15076]: error sending to idled: 3
>> Mar 12 03:15:55 server imaps[15081]: error sending to idled: 3
>> Mar 12 03:15:55 server imap[15080]: error sending to idled: 3
>> Mar 12 03:15:55 server imaps[15077]: error sending to idled: 3
>> Mar 12 03:15:55 server imaps[15079]: error sending to idled: 3
>> Mar 12 03:15:56 server tls_prune[15071]: skiplist: checkpointed
>> /var/lib/imap/tls_sessions.db (575 records, 122352 bytes) in 0 seconds
>> Mar 12 03:15:56 server cyr_expire[15072]: skiplist: checkpointed
>> /var/lib/imap/deliver.db (37 records, 5464 bytes) in 0 seconds
>> Mar 12 03:15:56 server cyr_expire[15072]: Expunged 0 out of 9112
>> messages from 57 mailboxes
>> Mar 12 03:15:56 server cyr_expire[15072]: duplicate_prune: pruning back
>> 3.00 days
>> Mar 12 03:15:56 server cyr_expire[15072]: duplicate_prune: purged 0 out
>> of 37 entries
>>
>> The "error sending to idled: 3" is probably a red herring as cyrus-imap
>> has just restarted following a db backup, so earlier idled connections
>> have broken.
>>
>> Correspondingly in /var/log/messages I get:
>> Mar 12 03:15:56 server kernel: cyr_expire[15065]: segfault at
>> 7f9c8a79be78 ip 00007f9c8a91c5ed sp 00007ffd44b51d38 error 4 in
>> cyr_expire[7f9c8a8c9000+15c000]
>>
>> It may also be worth mentioning that on some nights, like the night
>> before the segfault, in /var/log/maillog I see:
>> Mar 11 03:42:09 server cyr_expire[3237]: Expunged 0 out of 9145 messages
>> from 61 mailboxes
>> Mar 11 03:42:09 server cyr_expire[3237]: duplicate_prune: pruning back
>> 1.00 days
>> Mar 11 03:42:11 server cyr_expire[3237]: duplicate_prune: purged 58 out
>> of 147 entries
>> Mar 11 03:45:39 server ctl_cyrusdb[25201]: checkpointing cyrus databases
>> Mar 11 03:45:39 server ctl_cyrusdb[25201]: done checkpointing cyrus
>> databases
>> Mar 11 03:49:57 server master[31633]: exiting on SIGTERM/SIGINT
>> Mar 11 03:50:58 server master[27624]: setrlimit: Unable to set file
>> descriptors limit to -1: Operation not permitted
>> Mar 11 03:50:58 server master[27624]: retrying with 4096 (current max)
>> Mar 11 03:50:58 server cyr_expire[27625]: Repacking mailbox 
>> user.mail-ebay
>> Mar 11 03:50:58 server cyr_expire[27625]: Repacking mailbox 
>> user.ourfamily
>> Mar 11 03:50:59 server cyr_expire[27625]: Repacking mailbox
>> user.ourfamily.Drafts
>> Mar 11 03:50:59 server ctl_cyrusdb[27628]: recovering cyrus databases
>> Mar 11 03:50:59 server ctl_cyrusdb[27628]: skiplist: checkpointed
>> /var/lib/imap/mailboxes.db (61 records, 5148 bytes) in 0 seconds
>> Mar 11 03:50:59 server ctl_cyrusdb[27628]: skiplist: checkpointed
>> /var/lib/imap/annotations.db (0 records, 144 bytes) in 0 seconds
>> Mar 11 03:50:59 server ctl_cyrusdb[27628]: done recovering cyrus 
>> databases
>> Mar 11 03:50:59 server cyr_expire[27625]: Expunged 0 out of 9145
>> messages from 61 mailboxes
>> Mar 11 03:50:59 server cyr_expire[27625]: duplicate_prune: pruning back
>> 0.00 days
>> Mar 11 03:50:59 server master[27624]: unable to setsocketopt(IP_TOS):
>> Operation not supported
>> Mar 11 03:50:59 server ctl_cyrusdb[27633]: checkpointing cyrus databases
>> Mar 11 03:50:59 server ctl_cyrusdb[27633]: done checkpointing cyrus
>> databases
>> Mar 11 03:51:00 server tls_prune[27631]: skiplist: checkpointed
>> /var/lib/imap/tls_sessions.db (303 records, 64348 bytes) in 1 second
>> Mar 11 03:51:00 server cyr_expire[27632]: skiplist: checkpointed
>> /var/lib/imap/deliver.db (84 records, 12424 bytes) in 0 seconds
>> Mar 11 03:51:00 server cyr_expire[27632]: Expunged 0 out of 9109
>> messages from 61 mailboxes
>> Mar 11 03:51:00 server cyr_expire[27632]: duplicate_prune: pruning back
>> 3.00 days
>> Mar 11 03:51:00 server tls_prune[27631]: tls_prune: purged 3 out of 303
>> entries
>> Mar 11 03:51:00 server cyr_expire[27632]: duplicate_prune: purged 0 out
>> of 83 entries
>> Mar 11 03:51:04 server cyr_expire[27625]: duplicate_prune: purged 89 out
>> of 89 entries
>> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1581 File
>> handles still open at environment close
>> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1582 Open file
>> handle: /var/lib/imap/db/__db.001
>> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1582 Open file
>> handle: /var/lib/imap/db/__db.002
>> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1582 Open file
>> handle: /var/lib/imap/db/__db.003
>> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB0060 PANIC:
>> fatal region error detected; run recovery
>> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR: critical database
>> situation
>>
>> I'm afraid I've absolutely no idea how to troubleshoot this. Can anyone
>> please help?
>>
>> Thanks,
>>
>> Nick
>
>
> Looks like you might have a corrupted BerlkeyDB . What is the output of ?
>
> grep _db /etc/imapd.conf
Nothing? I can't see any alternative file that may hold this info. 
imapd.conf is:

configdirectory: /var/lib/imap
partition-default: /var/spool/imap
admins: root
sievedir: /var/lib/imap/sieve
sendmail: /usr/sbin/sendmail
hashimapspool: true
sasl_pwcheck_method: saslauthd
sasl_mech_list: PLAIN
tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt

flushseenstate:       1
allowplaintext:       yes
reject8bit:           no
munge8bit:            no
lmtp_over_quota_perm_failure: 1
timeout:              30
imapidlepoll:         60
idlesocket:           /var/lib/imap/socket/idle
lmtpsocket:           /var/lib/imap/socket/lmtp
allowapop:            no
altnamespace:         0
unixhierarchysep:     yes
lmtp_downcase_rcpt:   yes
username_tolower:     1
autocreatequota:      -1
createonpost:         1
virtdomains:          off
expire-days:    1
expunge-days:    1
delete-days:    7

>
>
>
>>
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
>
>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.