Re: Problem with large twoskip mailboxes.db after upgrade

Hi Bron,

Thanks for your attention.
All is on Centos 8.4. For better tests on the same server we have
installed 2.5.15 (we are using compiled version), and with 2.5
reconstruct lasts:
real    0m0.206s
with full 550k records in mailboxes.db

With 3.4.2 and with 3.2.8(we also installed this version) about 4s.

I think that problem is with slow searching in twoskip, or some
misconfiguration. The only difference between 2.5 and 3.x we have found
is that in 2.5 version inside config/db directory are some DB files :
__db.001  __db.002  __db.003

and with 3.x this folder is empty.

I'm not sure if this is cause, but we were cleaning this folder , and
running reconstruct without starting master, and this files appeared.

Our compilation options are:

/configure --prefix=/opt/store21/imapd-2.5.15 --with-cyrus-user=cyrus
--with-cyrus-group=cyrus  --disable-gssapi
--with-cyrus-prefix=xxxxxxxxxxxx/cyrus
--with-service-path=xxxxxxxxxxxx/service --with-sasl=xxxxxxxxxxxx
--sysconfdir=xxxxxxxxxxxx/etc --disable-murder
--with-pidfile=xxxxxxxxxxxx/var/cyrus-master.pid
--with-syslogfacility=LOCAL0 --without-perl --without-snmp --enable-idled

(nothing special)

and config file (imapd.conf)

configdirectory:        xxxxxxxxxxxxxxxxx/config
defaultpartition: default
partition-default:     xxxxxxxxxxxxxxx/mailbox
sievedir:               xxxxxxxxxxxxxxxxx/sieve
annotation_definitions: xxxxxxxxxxxxxxxxxxxxx/etc/imapd.annotations.conf
servername: xxxxxxxxxxxxxxx
hashimapspool: true
fulldirhash: 1
altnamespace: 1
allowusermoves: 1
virtdomains: 1
unixhierarchysep: 1
allowplaintext: 1
sasl_mech_list: plain login
sasl_pwcheck_method: auxprop saslauthd
sasl_auxprop_plugin: sasldb
admins: cyrusadmin
defaultdomain: xxxxxxxxxxxxxxxxx
autocreate_quota: 10485760
autocreate_sieve_folders: Junk
syslog_prefix: xxxxxxxxxxxxxxxxxxxxx
lmtp_over_quota_perm_failure: on
quotawarn: 85
proxyservers: xxxxxxxxxxxxxxxxxxxxxxx
proxy_authname: xxxxxxxxxxxxxxxxxxxxxxxxxx
lmtp_admins: xxxxxxxxxxxxxxxxxxxxxxxxxx

Also nothing special,
reverseacls: true didn't helped.

This is standalone instalation without murder, replication and other
features.

Regards
AK


W dniu 05.11.2021 o 01:30, Bron Gondwana pisze:
> Hi Andrzej,
> 
> What operating system is this on?  I suspect maybe it's that the locks 
> or mmaps are significantly more expensive on this platform than on the 
> places I've used it.
> 
> Also if you can give specifics of your config that might help a lot in 
> tracking down what's going on.
> 
> I wonder if setting |reverseacls: true| in your imapd.conf might fix 
> this.  If it's locking and unlocking that frequently then it seems like 
> maybe it's doing a mailbox scan that we could be avoiding.
> 
> Cheers,
> 
> Bron.
> 
> On Fri, Nov 5, 2021, at 10:20, Andrzej Kwiatkowski wrote:
>>
>> Before Upgrade on  old 2.5.x cyrus we have about 250k mailboxes with
>> about 500k folders.
>> We made upgrade with imapsync , because of problem with path
>> compatibility with unixhierarchysep set on yes.
>>
>> After upgrade to 3.4.2 we have problem that fcntl locks on mailboxes.db
>> are very long. For tests we have made copy of one production servers,
>> and run without load, reconstruct for empty mailbox lasts 4-5 seconds,
>> and strace shows us:
>>
>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>> st_size=104625064, ...}) = 0
>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>> st_size=104625064, ...}) = 0
>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>> st_size=104625064, ...}) = 0
>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>> st_size=104625064, ...}) = 0
>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>> st_size=104625064, ...}) = 0
>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>> st_size=104625064, ...}) = 0
>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>> l_len=0}) = 0
>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>
>> For reconstruct single mailbox there is sometimes 27k operation for RDLCK.
>>
>> One of our server have database with only 50k folders, and there is
>> everythink ok.We also made dump, restore mailboxes.db, conversion to
>> skipstamp and back for twoskip.
>> Without progress. With zeroskip is much worse.
>>
>> We also test on XFS,EXT4. Mailboxes.db on ramdisk for removing disk
>> performance issue..
>>
>> Any idea ?
>>
>> Regards
>>
>> AK
>>
>> ------------------------------------------
>> Cyrus: Info
>> Permalink: 
>> https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-Mf02e8d0ad71ef6d7ac900a1f 
>> <https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-Mf02e8d0ad71ef6d7ac900a1f>
>> Delivery options: https://cyrus.topicbox.com/groups/info/subscription 
>> <https://cyrus.topicbox.com/groups/info/subscription>
>>
> 
> --
>    Bron Gondwana, CEO, Fastmail Pty Ltd
>    [email protected]
> 
> 
> *Cyrus <https://cyrus.topicbox.com/latest>* / Info / see discussions 
> <https://cyrus.topicbox.com/groups/info> + participants 
> <https://cyrus.topicbox.com/groups/info/members> + delivery options 
> <https://cyrus.topicbox.com/groups/info/subscription> Permalink 
> <https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-M668cd6a988cc43afb98e4755> 
> 

------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-M2f58fb436c7d065a07a74af0
Delivery options: https://cyrus.topicbox.com/groups/info/subscription