Re: Trying to rebuild xapian indexes, indexer is indexing more messages than exist

I appreciate your help, but i have elected to switch back from Xapian to 
squat search until the "longstanding issue" in Xapian is fixed. 🤷

This is a family email server with only five people's email on it, we 
can definitely make do without Xapian.

On 8/15/25 2:52 PM, Nels Lindquist wrote:
> On 2025-08-14 6:22 PM, Jonathan Kamens wrote:
>
>> On 8/14/25 2:23 PM, Nels Lindquist wrote:
>>> What configuration directives do you have for the conversations 
>>> database (which is required for xapian) and for search?
>>
>> conversations: 1
>> conversations_db: twoskip
>> search_engine: xapian
>> defaultsearchtier: default
>> defaultsearchpartition-default: /var/spool/imap-search
>> sync_log: on
>> sync_log_channels: squatter
>
> You probably also want to add:
>
> search_fuzzy_always: yes
>
> That won't affect the behaviour you're seeing, but since Xapian can 
> only use fuzzy searches, it'll translate client requests for standard 
> searches for use with the Xapian engine which may increase client 
> performance.
>
>>> There's a longstanding issue with xapian indexing which can be 
>>> worked around by setting search_batchsize to a very large number 
>>> (like 1000000), blowing away the contents of your search partition 
>>> and then regenerating with "squatter -v -i".
>>
>> Yes, that sort of resolves the issue. I was able to rebuild my 
>> indexes, but squatter -R still appears to be doing a ton of extra 
>> work every time it starts up and causing excessive locking which is 
>> blocking clients from functioning for noticeable delays while it is 
>> considering reindexing every single message in every mailbox every 
>> time it starts up.
>>
>> It's disappointing that this longstanding issue hasn't been fixed and 
>> doesn't seem to have been documented anywhere obvious that I could 
>> find. Because of that I wasted many hours upgrading to 3.12.1 and 
>> then digging into squatter in a debugger to try to figure out what's 
>> going on.
>>
>> Of note: when I said in my last message that the rolling squatter 
>> process was doing the right thing I was mistaken. In fact the rolling 
>> squatter was having the same problem, it was just rotating through 
>> the mailboxes and reindexing the same messages in each one in turn, 
>> rather than sticking with a single mailbox and indexing the same 
>> messages in that mailbox over and over, so I didn't notice right away 
>> that it was also malfunctioning.
>
> Is your "squatter -R" setup in cyrus.conf still in the EVENTS section, 
> by chance?
>
> If so, it should be moved to the DAEMON section. Once it's up and 
> running it should be continuously rolling search indexing from the 
> squatter replication channel you've defined, and shouldn't be doing 
> much of anything otherwise.
>
> If you set up multiple search tiers for different types of storage, 
> you might still need some squatter EVENTS for periodically repacking 
> the search indexes between storage tiers. We haven't bothered with 
> that as the performance has been great with a single tier with our 
> typical loads. (See 
> http://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html 
> for some examples.)
>
> We haven't noted mailbox locking due to squatter activity (even on 
> startup) in our environment, but we're still running Cyrus IMAPD 3.8.5 
> (self-built RPMs) on AlmaLinux 8, so no idea whether the behaviour's 
> changed in 3.10.x or 3.12.x
>
> What kind of squatter logs are you seeing? This is pretty typical for us:
>
> Aug 15 11:54:35 edm-cmbe01 cyrus/squatter[2716]: indexing mailbox 
> user/[redacted1]@maei.ca...
> Aug 15 11:54:35 edm-cmbe01 cyrus/squatter[2716]: Xapian committed 3 
> updates in 0.353203 sec
> Aug 15 12:10:49 edm-cmbe01 cyrus/squatter[2716]: indexing mailbox 
> user/[redacted2]/[email protected]...
> Aug 15 12:10:53 edm-cmbe01 cyrus/sync_client[2700]: Reprocessing sync 
> log file /var/lib/imap/sync/squatter/log-run
> Aug 15 12:10:54 edm-cmbe01 cyrus/squatter[2716]: Xapian committed 2 
> updates in 4.963152 sec
> Aug 15 12:10:54 edm-cmbe01 cyrus/squatter[2716]: mappedfile: longlock 
> /var/lib/imap/user/uuid/5/c/5cf27ed7-050e-44dd-924d-eefb6835df76/xapianactive.db 
> for 5.0 seconds
>
>
>

------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/T5a3c00cdf15ad8da-M2f4c2e7da4c04f1b926f9b20
Delivery options: https://cyrus.topicbox.com/groups/info/subscription