Re: fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)

Hi Michael,


Thanks for looking into this. I don't think fts_flatcurve_xapian_refresh()
alone would close the gap here — as far as I can tell it re-opens/refreshes
an already-open db handle (to pick up externally-committed changes or drop
a stale NFS attribute cache), but it doesn't release the write handle the
process is holding open against the *old* shard directory during optimize.
On NFS, deleting a directory whose files are still open by the client
doesn't unlink them — the files get silently renamed to .nfsXXXX, rmdir()
on the directory then fails with ENOTEMPTY, and (as far as I can tell from
the code) that failure isn't surfaced — the directory is left behind, still
pointing at what should have been the removed shard, and empty of anything
usable. A refresh call on the handle wouldn't change that, since the handle
to the old path is what needs to be dropped before the delete, not
refreshed afterward.


Happy to be wrong here since I haven't run this against your tree — is
there something in the optimize path that already closes out the old
shard's handles before the delete step that I'm missing?

If it'd help you reproduce it locally, I'm glad to send over the exact NFS
client mount options and the relevant doveconf -n
fts_flatcurve/mail_location section from our setup. NFS server behavior
around silly-rename can vary a fair bit between implementations, so it
might just be that your test NFS server doesn't hit the same path ours does.

On Tue, Aug 11, 2026 at 11:59 PM Michael Slusarz <
[email protected]> wrote:

> > On 08/07/2026 10:22 AM MDT Ihor Rusyn via dovecot <[email protected]>
> wrote:
> >
> > Hello,
> >
> > "doveadm fts optimize" leaves the mailbox in a state where no new mail
> can
> > be
> > indexed. The existing index survives and searches over already-indexed
> mail
> > keep
> > working, so the damage is invisible until the next message arrives.
> >
> > Version:      2.4.4 (8b687aa65c), packaged in a container image
> > FTS:          fts_flatcurve, substring_search = yes, min_term_size
> default
> > (2)
> >               fts_autoindex = yes, fts_search_add_missing = yes
> > Index store:  NFSv3 (rsize/wsize 32768, local_lock=none)
> > Mail store:   NFSv3, maildir
>
> [snip]
>
> I cannot reproduce this, probably due to some combination of NFS latency
> and semantics.  (Xapian itself notes that NFS operation can be problematic.
> https://xapian.org/docs/admin_notes.html)
>
> It's likely fixed by adding fts_flatcurve_xapian_refresh() in
> fts_flatcurve_xapian_optimize_box_do(), but I am unable to verify.
>
> michael
>
>

-- 
Best regards,
Ihor Rusyn

_______________________________________________
dovecot mailing list -- [email protected]
To unsubscribe send an email to [email protected]