Re: fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)

Hello Ihor, hello Michael,

We tested this on our setup (2.4.4-5+debian12 (8b687aa65c)) as well and could not reproduce the issue. We are using a proprietary storage implementation (not NFS), with the following Dovecot settings:

mail_nfs_storage = yes
mail_nfs_index = yes

After running doveadm fts optimize -u <user>, the index files were consolidated into a single index.* directory, and the existing current.* directory was removed. No current.* directory remained after the optimization. We then manually triggered an FTS search (we do not use fts_autoindex), after which a new current.* directory was created correctly with a complete Xapian database.

Given that the reported setup uses NFSv3 with local_lock=none, I also wonder whether NFS locking and cache semantics may be involved here. In particular, I would verify that the required NFS locking infrastructure is working correctly before assuming that this is a generic flatcurve optimization issue.

————————————————————————————————————————
In addition, based on our testing of FTS Flatcurve, I would recommend reviewing the following settings:

1. fts_flatcurve_min_term_size
We use: fts_flatcurve_min_term_size = 1

instead of the default value of 2. During testing we found that indexing respects fts_flatcurve_min_term_size and therefore drops one-character terms, while query tokenization may still generate such terms. This can result in false negatives, for example when searching for URLs or identifiers containing single-character tokens.

A simple example is: doveadm fts tokenize -u <USER> 'http://www.youtube.com/watch?v=kgu_fz1zytE'

which generates a "v" token even with fts:flatcurve_min_term_size=2.
In our measurements, changing min_term_size from 2 to 1 resulted in only a negligible increase in index size.


2. fts_flatcurve_substring_search
We deliberately use: fts_flatcurve_substring_search = no

In our tests, enabling substring search caused massive index growth due to the additional index-only tokens, while the query tokens remained identical. Email addresses are already split into searchable tokens by the email-address tokenizer, so we found little practical benefit from enabling substring search in our environment.

————————————————————————————————————————
There are many other FTS settings worth tuning for the specific environment. In particular, we found that fts_autoindex can cause significant performance impact during indexing for users with mailboxes containing tens of thousands of messages. In such environments, it may be preferable to disable automatic indexing and perform the indexing asynchronously using a background service instead.


Regards
Jens Urban


Von: Michael Slusarz via dovecot <[email protected]>
Datum: Mittwoch, 12. August 2026 um 02:24
An: Ihor Rusyn <[email protected]>; Ihor Rusyn via dovecot <[email protected]>
Betreff: Re: fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)

> On 08/07/2026 10:22 AM MDT Ihor Rusyn via dovecot <[email protected]> wrote:
>
> Hello,
>
> "doveadm fts optimize" leaves the mailbox in a state where no new mail can
> be
> indexed. The existing index survives and searches over already-indexed mail
> keep
> working, so the damage is invisible until the next message arrives.
>
> Version:      2.4.4 (8b687aa65c), packaged in a container image
> FTS:          fts_flatcurve, substring_search = yes, min_term_size default
> (2)
>               fts_autoindex = yes, fts_search_add_missing = yes
> Index store:  NFSv3 (rsize/wsize 32768, local_lock=none)
> Mail store:   NFSv3, maildir

[snip]

I cannot reproduce this, probably due to some combination of NFS latency and semantics.  (Xapian itself notes that NFS operation can be problematic. https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fxapian.org%2Fdocs%2Fadmin_notes.html&data=05%7C02%7Cjens.urban%40freenet.ag%7C23f8ee82ed544f08372d08def8080ff9%7C7d95deb30bca4c9da61e1fd6c47ed60c%7C0%7C0%7C639220910671970524%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=eqoHjH9Pbt8h0AGo%2FDPRw%2F15YYeuXSTRMr3r8isznpE%3D&reserved=0)<https://xapian.org/docs/admin_notes.html>

It's likely fixed by adding fts_flatcurve_xapian_refresh() in  fts_flatcurve_xapian_optimize_box_do(), but I am unable to verify.

michael

_______________________________________________
dovecot mailing list -- [email protected]
To unsubscribe send an email to [email protected]

_______________________________________________
dovecot mailing list -- [email protected]
To unsubscribe send an email to [email protected]