Re: fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)

Hi Jens,


Quick update — we can reproduce this reliably, specifically on NFS. We've
also tested the same optimize path on ext4 (local) and CephFS, and neither
shows the problem. Here's why we think that lines up:

The core issue is unlink()/rmdir() semantics for files/directories that are
still open by the calling process. On a local POSIX filesystem (ext4) and
on CephFS, unlinking a file that's still open by the same process is fully
supported: the directory entry is removed immediately, and the file's data
blocks stay allocated until the last file descriptor closes. So rmdir() on
the parent directory succeeds right away, because as far as the filesystem
is concerned the directory is already empty — the still-open file just has
no name anymore.


NFS can't do that. There's no way for an NFS client to tell the server
"unlink this, but I still have it open." When a client unlinks a file it
still holds open, the NFS client silently works around it with what's
usually called "silly rename": instead of an actual unlink, it renames the
file to a hidden .nfsXXXX name, so the directory entry survives until the
file is finally closed and the client can clean it up. In
fts_flatcurve_xapian_optimize_box_do(), the shard's write handles are still
open when the old current.* directory is deleted — fine on ext4/CephFS, but
on NFS it leaves .nfsXXXX entries behind, so the rmdir() on that directory
fails with ENOTEMPTY. That failure doesn't appear to be surfaced anywhere —
unlink_directory() reports success regardless — so the directory is left
behind, and once the open handles are eventually closed, what remains is an
empty current.* directory that is still treated as the active shard.


We also tested whether mail_nfs_storage/mail_nfs_index change anything here
— enabled both on our stand and reproduced the exact same failure chain
(optimize succeeds silently, current.* ends up empty, next delivery fails
to index with "Failed to open glass revision file," subsequent search fails
with an internal error). That tracks with the code: as far as we can tell,
fts-flatcurve doesn't consult either setting anywhere — they only affect
the mail_index (MAIL_INDEX_OPEN_FLAG_NFS_FLUSH) and dotlock behavior in
lib-storage, not the Xapian glass files flatcurve manages itself. So this
doesn't look like something those settings, or NFS locking configuration in
general, can fix.

Let us know if it'd help to see the exact optimize-path trace, or anything
else from our reproduction.

On Wed, Aug 12, 2026 at 12:12 PM Jens Urban <[email protected]> wrote:

> Hello Ihor, hello Michael,
>
> We tested this on our setup (2.4.4-5+debian12 (8b687aa65c)) as well and
> could not reproduce the issue. We are using a proprietary storage
> implementation (not NFS), with the following Dovecot settings:
>
> mail_nfs_storage = yes
> mail_nfs_index = yes
>
> After running doveadm fts optimize -u <user>, the index files were
> consolidated into a single index.* directory, and the existing current.*
> directory was removed. No current.* directory remained after the
> optimization. We then manually triggered an FTS search (we do not use
> fts_autoindex), after which a new current.* directory was created correctly
> with a complete Xapian database.
>
> Given that the reported setup uses NFSv3 with local_lock=none, I also
> wonder whether NFS locking and cache semantics may be involved here. In
> particular, I would verify that the required NFS locking infrastructure is
> working correctly before assuming that this is a generic flatcurve
> optimization issue.
>
> ————————————————————————————————————————
> In addition, based on our testing of FTS Flatcurve, I would recommend
> reviewing the following settings:
>
> 1. fts_flatcurve_min_term_size
> We use: *fts_flatcurve_min_term_size = 1*
>
> instead of the default value of 2. During testing we found that indexing
> respects fts_flatcurve_min_term_size and therefore drops one-character
> terms, while query tokenization may still generate such terms. This can
> result in false negatives, for example when searching for URLs or
> identifiers containing single-character tokens.
>
> A simple example is: doveadm fts tokenize -u <USER> '
> http://www.youtube.com/watch?v=kgu_fz1zytE'
>
> which generates a "v" token even with fts:flatcurve_min_term_size=2.
> In our measurements, changing min_term_size from 2 to 1 resulted in only a
> negligible increase in index size.
>
>
> 2. fts_flatcurve_substring_search
> We deliberately use: *fts_flatcurve_substring_search = no*
>
> In our tests, enabling substring search caused massive index growth due to
> the additional index-only tokens, while the query tokens remained
> identical. Email addresses are already split into searchable tokens by the
> email-address tokenizer, so we found little practical benefit from enabling
> substring search in our environment.
>
> ————————————————————————————————————————
> There are many other FTS settings worth tuning for the specific
> environment. In particular, we found that fts_autoindex can cause
> significant performance impact during indexing for users with mailboxes
> containing tens of thousands of messages. In such environments, it may be
> preferable to disable automatic indexing and perform the indexing
> asynchronously using a background service instead.
>
>
> Regards
> Jens Urban
>
>
> *Von: *Michael Slusarz via dovecot <[email protected]>
> *Datum: *Mittwoch, 12. August 2026 um 02:24
> *An: *Ihor Rusyn <[email protected]>; Ihor Rusyn via dovecot <
> [email protected]>
> *Betreff: *Re: fts-flatcurve 2.4.4: after "doveadm fts optimize" a
> mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)
>
> > On 08/07/2026 10:22 AM MDT Ihor Rusyn via dovecot <[email protected]>
> wrote:
> >
> > Hello,
> >
> > "doveadm fts optimize" leaves the mailbox in a state where no new mail
> can
> > be
> > indexed. The existing index survives and searches over already-indexed
> mail
> > keep
> > working, so the damage is invisible until the next message arrives.
> >
> > Version:      2.4.4 (8b687aa65c), packaged in a container image
> > FTS:          fts_flatcurve, substring_search = yes, min_term_size
> default
> > (2)
> >               fts_autoindex = yes, fts_search_add_missing = yes
> > Index store:  NFSv3 (rsize/wsize 32768, local_lock=none)
> > Mail store:   NFSv3, maildir
>
> [snip]
>
> I cannot reproduce this, probably due to some combination of NFS latency
> and semantics.  (Xapian itself notes that NFS operation can be problematic.
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fxapian.org%2Fdocs%2Fadmin_notes.html&data=05%7C02%7Cjens.urban%40freenet.ag%7C23f8ee82ed544f08372d08def8080ff9%7C7d95deb30bca4c9da61e1fd6c47ed60c%7C0%7C0%7C639220910671970524%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=eqoHjH9Pbt8h0AGo%2FDPRw%2F15YYeuXSTRMr3r8isznpE%3D&reserved=0)
> <https://xapian.org/docs/admin_notes.html>
>
> It's likely fixed by adding fts_flatcurve_xapian_refresh() in
> fts_flatcurve_xapian_optimize_box_do(), but I am unable to verify.
>
> michael
>
> _______________________________________________
> dovecot mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
>


-- 
Best regards,
Ihor Rusyn

_______________________________________________
dovecot mailing list -- [email protected]
To unsubscribe send an email to [email protected]