Re: Various questions about databases (upgrade and migration)

Raphaël Halimi <[email protected]> Thu, 14 Feb 2019 15:46:03 +0100
Newsgroups	gmane.mail.imap.cyrus
Message-ID	<[email protected]>
Hi Egoitz,

Thank you for your quick answer.

Le 14/02/2019 à 14:46, Egoitz Aurrekoetxea a écrit :
> Now for the databases themselves. In /var/lib/cyrus the global databases
> were converted on-the-fly:
>
> # file /var/lib/cyrus/*.db
> /var/lib/cyrus/annotations.db:  Cyrus twoskip DB
> /var/lib/cyrus/deliver.db:      Cyrus twoskip DB
> /var/lib/cyrus/mailboxes.db:    Cyrus twoskip DB
> /var/lib/cyrus/statuscache.db:  Cyrus twoskip DB
> /var/lib/cyrus/tls_sessions.db: Cyrus twoskip DB
> /var/lib/cyrus/user_deny.db:    empty
>
> However, the user databases were not converted:
>
> # file /var/lib/cyrus/user/*/*
> /var/lib/cyrus/user/u/user1.seen: Cyrus skiplist DB
> /var/lib/cyrus/user/u/user1.sub:  ASCII text
> /var/lib/cyrus/user/u/user2.seen:   Cyrus skiplist DB
> /var/lib/cyrus/user/u/user2.sub:    ASCII text
> /var/lib/cyrus/user/u/user3.seen:   Cyrus skiplist DB
> /var/lib/cyrus/user/u/user3.sub:    ASCII text
>  
>> *Cyrus 2.4 converted databases on the fly. Cyrus 2.5 and newer don't.
>> You should launch a "reconstruct -r -V max" for that purpose.*
>
> So my next questions are: why are the databases still in skiplist
> format, whereas according to /usr/lib/cyrus/cyrus-db-types.txt, they
> should be twoskip ? Why didn't Cyrus convert them on-the-fly like the
> global databases ? Do I have to manually do it myself ? And if I do
> convert them, will it change anything (performance, reliability, etc
> etc) ?
>  
> *As said perhaps is a Debian derived config for the package. Yes you
> should with the command above.*
>>
>> Also, what about the various databases in the mail directories
>> (cyrus.cache, cyrus.header, cyrus.index) ? For most of them, the "file"
>> command only reports "data". What format are they actually in ? Do I
>> have to convert them too ?
>>  
>> *Sure... Just launch a reconstruct -r -V max...*

I just ran this command as cyrus user on my server (after reading the
manual page). Unfortunately, the "seen" databases in /var/lib/cyrus/user
are still reported by "file" as skiplist, and the "cyrus.cache",
"cyrus.header" and "cyrus.index" in the various (sub)mailboxes, are
still reported as simply "data".

It did create "cyrus.annotations" databases in each subfolder, though
(in twoskip format).

Also,I'm a bit worried. I did see in the logs lines that said:

repacking mailbox user.<user>.<folder>.<subfolder>

and

reconstructing user.<user>.<folder>.<subfolder>

...but also some more worrying lines that said:

uniqueid clash with user.<user>.<folder>.<subfolder> for <some hex
value> - changing user.<user>.<folder>.<subfolder>

Is it something I should worry about ?

Regarding the fact that they're still not in the twoskip format, should
I use cvt_cyrusdb instead ? That would be unfortunate, since I'll have
to create a script fed to the "find" command to mass-convert all
databases; plus, I still don't know what the input format (the "data"
that file talks about) is.

>> When I mill migrate, will I have to convert the databases through the
>> flat format and back, or can I blindly copy the whole contents of
>> /var/spool/cyrus and /var/lib/cyrus to the new server and expect it to
>> work out of the box ?
>>  
>> *When doing such a migration, it would be better to setup a
>> replication between the 2.4 and the new 2.5 in the hosting. You should
>> encrypt that communication. You could use the own cyrus encription for
>> replication or something like OpenVPN. Although it should work, I
>> wouldn't copy directly (with an rsync or scp) the files.*

Yes, both servers communicate through a VPN, but since both will have
the same Cyrus version, I thought I could just copy the files. Why is it
a bad idea ?

Regards,

-- 
Raphaël Halimi

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
signature.asc (application/pgp-signature, 963 B)