2.4.4 fts-flatcurve: fts optimize silently destroys a folder's index when the index is on NFS

Hello,


Summary

=======


On Dovecot 2.4.4 with fts_flatcurve and mail_index_path on NFS, "doveadm
fts optimize" leaves behind an empty shard directory. That directory
remains the shard every subsequent write opens, so from that point on the
folder can no longer index new mail. The operation reports success and logs
nothing at the time. The damage is permanent and, on a folder that

receives no new mail, invisible.


Two separate defects are involved, and both are reported below:


  (1) fts optimize deletes a shard directory whose files it still holds
open. On a local filesystem the unlink succeeds invisibly; NFS cannot
express "unlink but keep open" and silly-renames instead, so the directory
survives.


  (2) unlink_directory() reports success when rmdir() failed, which is what
makes (1) silent rather than loud.


Environment

===========


  Dovecot 2.4.4 CE (8b687aa65c)

    container image
sha256:d6b2f80db2e656a9250583712a82397e2752b35ad7e4e2e71c85fdc0d510137d

    base Debian 13 trixie

  fts_flatcurve (Xapian glass)

    libxapian30 1.4.29-3 (libxapian.so.30.14.1)

  substring_search = yes, min_term_size = 2

  optimize_limit at its default of 10

  dovecot_storage_version = 2.3.0

  mail_index_path on NFSv3; mail storage on a separate NFS export


  NFS client is the host, not the container:

    Debian 12 bookworm, kernel 6.1.0-51-amd64 x86_64

    nfs-common 1:2.6.2-4+deb12u1

    mount options: rw,noatime,vers=3,rsize=32768,wsize=32768,hard,

                   nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,

                   local_lock=none


Impact

======


An operator does not have to invoke optimize to be exposed: with the
default optimize_limit = 10, flatcurve runs it itself once a folder
accumulates ten shards. Every active folder therefore reaches this
eventually.


Already-indexed mail stays searchable, so the folder looks healthy. Only
indexing of new mail fails. With fts_search_add_missing = yes, the first
SEARCH after new mail arrives attempts to index it, fails, and the client
receives NO [SERVERBUG].


I hit this while migrating a host from 2.3 to 2.4: 114 mailboxes were left
in this state in a single run, and it surfaced only because a monitoring
mailbox began answering SERVERBUG.


Defect 1: fts optimize leaves an empty shard directory on NFS

=============================================================


Steps to reproduce

------------------


1. Configure fts_flatcurve with the index path on an NFSv3 mount.


2. Take a mailbox with mail in INBOX (a few hundred messages is enough) and
build the index:


     doveadm index -u <user> INBOX


   Confirm one shard exists:


     ls <index>/.INBOX/fts-flatcurve/   ->   current.<id>


3. Run:


     doveadm fts optimize -u <user> INBOX


   (Alternatively, let it fire on its own by driving the folder past
optimize_limit shards.)


4. While it runs, list the shard directory repeatedly - .nfsXXXX entries
appear inside current.<id>.


5. After it finishes, note the exit status is 0 and no error was logged.

   List the directory again: index.<n> was created as expected, and
current.<id> is still there, now empty.


6. Deliver one new message, then:


     doveadm index -u <user> INBOX


Expected vs actual

------------------


Expected: after optimize, only the compacted index.<n> remains, and
subsequent indexing writes to a new shard.


Actual: the stale empty current.<id> remains and is reopened for writing,
so step 6 fails:


  Cannot open DB (RW, current.<id>): .../current.<id>/iamglass:

    Failed to open glass revision file for reading

    (No such file or directory)

  cmd index: Mailbox INBOX: Precache for UID=59 failed


and IMAP SEARCH on that folder returns NO [SERVERBUG].


What I observed while diagnosing

--------------------------------


strace of "doveadm fts optimize" on NFSv3 shows the old shard's files being
unlinked while handles on them are still open, so the directory cannot be
removed:


  unlink(...) = 0                 (x7, every file in the shard)

  rmdir(...)  = -1 ENOTEMPTY      (Directory not empty)

  rename(...) = 0                 (optimize -> index.N, proceeds regardless)


Once the process exits, the NFS client reaps the .nfsXXXX files and the
directory is simply empty.


What does not help

------------------


NFSv4.1, mounting with nolock, "doveadm fts rescan", "doveadm index", and
full re-indexing all take the same write path into the same empty shard and
fail identically. Removing the empty directory (or the folder's whole
fts-flatcurve tree) and re-indexing restores the folder; setting
optimize_limit = 0 avoids the situation but leaves the index spread across
many shards.


Defect 2: unlink_directory() returns success when rmdir() failed

================================================================


The ENOTEMPTY above never reaches the caller. unlink_directory() in

src/lib/unlink-directory.c returns 1 when the final rmdir() fails, the

same value it returns on success:


  if (rmdir(dir) < 0 && errno != ENOENT) {

          *error_r = t_strdup_printf("rmdir(%s) failed: %m", dir);

          return errno == ENOENT ? 0 : 1;

  }


The "errno != ENOENT" guard already establishes that errno is not ENOENT,
so the ternary always yields 1. Callers cannot distinguish "directory
removed" from "directory still there", and the error message it composed is
discarded.


Any condition that makes rmdir() fail with something other than ENOENT
reproduces this, so the return value is wrong independently of NFS - every
caller of unlink_directory() is affected. The code path is unchanged in
current main.


--

Best regards,

Ihor Ru

_______________________________________________
dovecot mailing list -- [email protected]
To unsubscribe send an email to [email protected]