Re: How to «bootstrap» xapian/squatter

On 2024-09-26 5:05 AM, Luca Olivetti via Info wrote:
> El 26/9/24 a les 12:16, Albert Shih ha escrit:
> 
>>> 3) I see that the (incomplete?) conversation db for my 9.8GB mailbox is
>>> 4.1GB, is that normal?
>>>
>>> root@vmail:~# ls -lh /var/lib/cyrus/user/l/luca.conversations
>>> -rw------- 1 cyrus mail 4.1G Sep 26 09:06
>>> /var/lib/cyrus/user/l/luca.conversations
>>> root@vmail:~# du -sh /var/spool/cyrus/mail/user/luca/
>>> 9.8G    /var/spool/cyrus/mail/user/luca/
>>>
>>
>> weird, I'm using also cyrus 3.6 from debian on Debian 12, but I got the
>> uuid file tree meaning the conversationdb/maibox/etc are in something
>>
>> root@zen-imap:/var/lib/cyrus/user/uuid/4/1/4153panofb1j8g3r508fdite# 
>> ls -lh
>> total 245M
>> -rw------- 1 cyrus mail 413M Sep 26 10:49 conversations.db
>> -rw------- 1 cyrus mail  296 Sep 25 02:59 counters.db
>> -rw------- 1 cyrus mail 4.2K Sep 20 10:18 sub.db
>> -rw------- 1 cyrus mail    4 Sep 26 10:41 xapianactive.db
>>
>> and this «guy» got  ~740 000 mails.
> 
> I "only" have around 600000 messages.
> 
> Anyway, ctl_conversationsdb also segfaulted for another user with just a 
> handful of messages (around 7k, 650MB) when the conversations db, again, 
> reached the 4.1GB size.
> 
> These are my search parameters in imapd.conf
> 
> search_engine: xapian
> search_index_headers: no
> search_batchsize: 1000000
> defaultsearchtier: t1
> t1searchpartition-default: /var/spool/cyrus/search
> conversations: 1
> sync_log: on
> sync_log_channels: squatter
> search_fuzzy_always: 1
> 
> 
> I though that maybe the segfault was due to a memory error, but the 
> virtual machine (actually an lxc container in proxmox) has 8GB assigned 
> and the usage stays well below 2GB.

Possibly noteworthy is that your debian package is only 3.6.1; most of 
the 3.6.x point releases up to 3.6.4 include fixes for squatter related 
bugs.

On your test server, did you migrate data from an older Cyrus IMAP version?

I'd be curious to see what happens if you relocate_by_id the mailbox 
you're testing and then try to rebuild the conversations database for it 
again, since the "new style" conversations database ends up in a 
different spot.

Do you have any other configuration directives related to conversations? 
You don't mention whether you switched to twoskip for the database, for 
example.

It doesn't look like configuration defaults have changed for 
conversations_* directives between 3.6 and 3.8 (which is what we're 
using) but for reasons I can't recall I have a couple of additional 
options set:

conversations: yes
conversations_db: twoskip
conversations_expire_after: 90d
conversations_keep_existing: yes

The latter two are supposed to be defaults, but might contribute to 
limiting the conversations database size.


-- 
Nels Lindquist
[email protected]


------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/Tb146f45ff23d5ffa-Mf01444a8b23ae529a3242c05
Delivery options: https://cyrus.topicbox.com/groups/info/subscription