Re: How to «bootstrap » xapian/squatter

Le 25/09/2024 à 17:39:36-0600, Nels Lindquist a écrit
Hi, 

> On 2024-09-25 1:23 PM, Albert Shih wrote:
> > Hi everyone.
> > 
> > So I would like to know what would be the most efficient method to
> > bootstrap squatter.
> > 
> > On my new server after sync almost all the mail I got, I try to «bootstrap»
> > squatter, meaning I launch the command squatter. But it's take lot of time.
> > So I try to run 64 threads (the server got 32 cores) of squatter, but still
> > after 10 h I just run 10% of the mailbox.
> > 
> > Is they are any way to speed up ? Or the only solution is to wait ?
> > 
> > Regards
> 
> I presume you have conversations.db enabled? It's required for xapian to
> work. Use twoskip for the conversations db format.

Yes. 

> 
> You could try rebuilding the conversations databases with
> "ctl_conversationsdb -v -b -r".
> 
> Did you try increasing search_batchsize? That was the "one thing" that fixed
> it for me:

Indeed !!!. 

I'v  put 524288 for the search_batchsize and I think I will be
over it in few hours.

Big thanks. 

> 
> On 2024-04-24 2:41 AM, [email protected] wrote:
> 
> > BTW. I see a strange issue in Xapian initial indexing. We have many
> > folders with 10,000 and more messages. Say, N messages. If I set
> > "search_batchsize" to value M less than N to prevent long mailbox
> > locking in live server, squatter re-enters the folder many times
> > more than N/M, with permanent intensive writing to  search partition.
> > So, I had to set search_batchsize to 1,000,000.

How do you launch squatter ? Because I intend to launch squatter using
sync_log so as I understand (but maybe I'm wrong that's why I ask),
squatter are just going to use what inside the log file and index only the
new mail. 

> Run "squatter -v -i" in a screen/tmux window so you can see what it's doing;
> if you observe repeated processing of the same folder/mailbox this could be
> your answer.

And if it's the case what should I do ? 

Regards
-- 
Albert SHIH 🦫 🐸
Observatoire de Paris
France
Heure locale/Local time:
jeu. 26 sept. 2024 11:43:17 CEST

------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/Tb146f45ff23d5ffa-Mdbc49bf2606c26fb475e3a4d
Delivery options: https://cyrus.topicbox.com/groups/info/subscription