Re: How to «bootstrap» xapian/squatter
Nels Lindquist <[email protected]>
| Newsgroups | gmane.mail.imap.cyrus |
|---|---|
| Message-ID | <[email protected]> |
On 2024-09-25 1:23 PM, Albert Shih wrote: > Hi everyone. > > So I would like to know what would be the most efficient method to > bootstrap squatter. > > On my new server after sync almost all the mail I got, I try to «bootstrap» > squatter, meaning I launch the command squatter. But it's take lot of time. > So I try to run 64 threads (the server got 32 cores) of squatter, but still > after 10 h I just run 10% of the mailbox. > > Is they are any way to speed up ? Or the only solution is to wait ? > > Regards I presume you have conversations.db enabled? It's required for xapian to work. Use twoskip for the conversations db format. You could try rebuilding the conversations databases with "ctl_conversationsdb -v -b -r". Did you try increasing search_batchsize? That was the "one thing" that fixed it for me: On 2024-04-24 2:41 AM, [email protected] wrote: > BTW. I see a strange issue in Xapian initial indexing. We have many > folders with 10,000 and more messages. Say, N messages. If I set > "search_batchsize" to value M less than N to prevent long mailbox > locking in live server, squatter re-enters the folder many times > more than N/M, with permanent intensive writing to search partition. > So, I had to set search_batchsize to 1,000,000. Run "squatter -v -i" in a screen/tmux window so you can see what it's doing; if you observe repeated processing of the same folder/mailbox this could be your answer. -- Nels Lindquist [email protected] ------------------------------------------ Cyrus: Info Permalink: https://cyrus.topicbox.com/groups/info/Tb146f45ff23d5ffa-M09c56ed12f65c99ba0f613d8 Delivery options: https://cyrus.topicbox.com/groups/info/subscription