Re: Problem with large twoskip mailboxes.db after upgrade

Hi,

I've made another test. I was wondering if problem exists in 
mailbox/folder name. I generated  file for ctl_mboxlist -u with script:

for i in {1..375000} ; do printf 
"domainname.pl"'!'"user.imap${i}-test\t0 default 
imap${i}[email protected]\tlrswipkxtecdan\tcyrusadmin\tlrswipxtecda\t\n" 
; done

And tested varius scenarios on stopped cyrus

50k rec : import 2min : reconstruct on one mailbox 0,4s
100k rec : import 3,31min : reconstruct on one mailbox 0,66s
250k rec : import 9min : reconstruct on one mailbox 1,63s
375k rec : import 13min : reconstruct on one mailbox 2,55s
550k rec : import 19min : reconstruct on one mailbox 3,4s

It shows that amount of records in mailboxes.db
have coincidence with time needed for operations.


Regards,
AK

W dniu 05.11.2021 o 09:12, Andrzej Kwiatkowski pisze:
> Hi Bron,
> 
> Thanks for your attention.
> All is on Centos 8.4. For better tests on the same server we have
> installed 2.5.15 (we are using compiled version), and with 2.5
> reconstruct lasts:
> real    0m0.206s
> with full 550k records in mailboxes.db
> 
> With 3.4.2 and with 3.2.8(we also installed this version) about 4s.
> 
> I think that problem is with slow searching in twoskip, or some
> misconfiguration. The only difference between 2.5 and 3.x we have found
> is that in 2.5 version inside config/db directory are some DB files :
> __db.001  __db.002  __db.003
> 
> and with 3.x this folder is empty.
> 
> I'm not sure if this is cause, but we were cleaning this folder , and
> running reconstruct without starting master, and this files appeared.
> 
> Our compilation options are:
> 
> /configure --prefix=/opt/store21/imapd-2.5.15 --with-cyrus-user=cyrus
> --with-cyrus-group=cyrus  --disable-gssapi
> --with-cyrus-prefix=xxxxxxxxxxxx/cyrus
> --with-service-path=xxxxxxxxxxxx/service --with-sasl=xxxxxxxxxxxx
> --sysconfdir=xxxxxxxxxxxx/etc --disable-murder
> --with-pidfile=xxxxxxxxxxxx/var/cyrus-master.pid
> --with-syslogfacility=LOCAL0 --without-perl --without-snmp --enable-idled
> 
> (nothing special)
> 
> and config file (imapd.conf)
> 
> configdirectory:        xxxxxxxxxxxxxxxxx/config
> defaultpartition: default
> partition-default:     xxxxxxxxxxxxxxx/mailbox
> sievedir:               xxxxxxxxxxxxxxxxx/sieve
> annotation_definitions: xxxxxxxxxxxxxxxxxxxxx/etc/imapd.annotations.conf
> servername: xxxxxxxxxxxxxxx
> hashimapspool: true
> fulldirhash: 1
> altnamespace: 1
> allowusermoves: 1
> virtdomains: 1
> unixhierarchysep: 1
> allowplaintext: 1
> sasl_mech_list: plain login
> sasl_pwcheck_method: auxprop saslauthd
> sasl_auxprop_plugin: sasldb
> admins: cyrusadmin
> defaultdomain: xxxxxxxxxxxxxxxxx
> autocreate_quota: 10485760
> autocreate_sieve_folders: Junk
> syslog_prefix: xxxxxxxxxxxxxxxxxxxxx
> lmtp_over_quota_perm_failure: on
> quotawarn: 85
> proxyservers: xxxxxxxxxxxxxxxxxxxxxxx
> proxy_authname: xxxxxxxxxxxxxxxxxxxxxxxxxx
> lmtp_admins: xxxxxxxxxxxxxxxxxxxxxxxxxx
> 
> Also nothing special,
> reverseacls: true didn't helped.
> 
> This is standalone instalation without murder, replication and other
> features.
> 
> Regards
> AK
> 
> 
> W dniu 05.11.2021 o 01:30, Bron Gondwana pisze:
>> Hi Andrzej,
>> 
>> What operating system is this on?  I suspect maybe it's that the locks 
>> or mmaps are significantly more expensive on this platform than on the 
>> places I've used it.
>> 
>> Also if you can give specifics of your config that might help a lot in 
>> tracking down what's going on.
>> 
>> I wonder if setting |reverseacls: true| in your imapd.conf might fix 
>> this.  If it's locking and unlocking that frequently then it seems like 
>> maybe it's doing a mailbox scan that we could be avoiding.
>> 
>> Cheers,
>> 
>> Bron.
>> 
>> On Fri, Nov 5, 2021, at 10:20, Andrzej Kwiatkowski wrote:
>>>
>>> Before Upgrade on  old 2.5.x cyrus we have about 250k mailboxes with
>>> about 500k folders.
>>> We made upgrade with imapsync , because of problem with path
>>> compatibility with unixhierarchysep set on yes.
>>>
>>> After upgrade to 3.4.2 we have problem that fcntl locks on mailboxes.db
>>> are very long. For tests we have made copy of one production servers,
>>> and run without load, reconstruct for empty mailbox lasts 4-5 seconds,
>>> and strace shows us:
>>>
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>>> st_size=104625064, ...}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>>> st_size=104625064, ...}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>>> st_size=104625064, ...}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>>> st_size=104625064, ...}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>>> st_size=104625064, ...}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>>> st_size=104625064, ...}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>>
>>> For reconstruct single mailbox there is sometimes 27k operation for RDLCK.
>>>
>>> One of our server have database with only 50k folders, and there is
>>> everythink ok.We also made dump, restore mailboxes.db, conversion to
>>> skipstamp and back for twoskip.
>>> Without progress. With zeroskip is much worse.
>>>
>>> We also test on XFS,EXT4. Mailboxes.db on ramdisk for removing disk
>>> performance issue..
>>>
>>> Any idea ?
>>>
>>> Regards
>>>
>>> AK
>>>
>>> ------------------------------------------
>>> Cyrus: Info
>>> Permalink: 
>>> https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-Mf02e8d0ad71ef6d7ac900a1f 
>>> <https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-Mf02e8d0ad71ef6d7ac900a1f>
>>> Delivery options: https://cyrus.topicbox.com/groups/info/subscription 
>>> <https://cyrus.topicbox.com/groups/info/subscription>
>>>
>> 
>> --
>>    Bron Gondwana, CEO, Fastmail Pty Ltd
>>    [email protected]
>> 
>> 
>> *Cyrus <https://cyrus.topicbox.com/latest>* / Info / see discussions 
>> <https://cyrus.topicbox.com/groups/info> + participants 
>> <https://cyrus.topicbox.com/groups/info/members> + delivery options 
>> <https://cyrus.topicbox.com/groups/info/subscription> Permalink 
>> <https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-M668cd6a988cc43afb98e4755> 
>> 

------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-M099f301bbd68ec13d55942d0
Delivery options: https://cyrus.topicbox.com/groups/info/subscription