fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)

Hello,

"doveadm fts optimize" leaves the mailbox in a state where no new mail can
be
indexed. The existing index survives and searches over already-indexed mail
keep
working, so the damage is invisible until the next message arrives.

Version:      2.4.4 (8b687aa65c), packaged in a container image
FTS:          fts_flatcurve, substring_search = yes, min_term_size default
(2)
              fts_autoindex = yes, fts_search_add_missing = yes
Index store:  NFSv3 (rsize/wsize 32768, local_lock=none)
Mail store:   NFSv3, maildir


Reproduction
------------

1) Starting point: a healthy index.

   # doveadm fts flatcurve check -u [email protected] INBOX
   INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1

   # ls /indexes/.../.INBOX/fts-flatcurve/*/
   current.1786117302443530: flintlock iamglass postlist.glass
termlist.glass

   # doveadm search -u [email protected] mailbox INBOX text a | tail -1
   88885837bb33136365390000d09efc50 58

2) Optimize. Exits 0, prints nothing.

   # doveadm fts optimize -u [email protected]
   # echo $?
   0

3) State afterwards: the compacted data is in a new index.<n> directory,
and the
   pre-existing current.<id> directory is still present but EMPTY.

   # ls /indexes/.../.INBOX/fts-flatcurve/*/
   current.1786117302443530:
   index.1464: flintlock iamglass postlist.glass termlist.glass

   # ls -ld .../fts-flatcurve/current.1786117302443530
   drwx------ 2 dovecot dovecot 4096 Aug  7 16:08
.../current.1786117302443530

   # doveadm fts flatcurve check -u [email protected] INBOX
   Error: fts-flatcurve(INBOX): Cannot open DB (RO;
current.1786117302443530);
          DatabaseNotFoundError: Couldn't detect type of database
   Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError:
          Directory does not contain a Xapian database

   Searching already-indexed mail still works at this point:

   # doveadm search -u [email protected] mailbox INBOX text a | tail -1
   88885837bb33136365390000d09efc50 58

4) Deliver one message and index it. This is where it breaks:

   # printf 'From: [email protected]\nSubject: reproprobe\n\nbody\n' \
       | doveadm save -u [email protected] -m INBOX
   Info: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe

   # doveadm index -u [email protected] INBOX
   Error: fts-flatcurve(INBOX): Cannot open DB (RW,
current.1786117302443530):
          .../current.1786117302443530/iamglass: Failed to open glass
revision
          file for reading (No such file or directory)
   Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal
error

5) From now on SEARCH fails too, because fts_search_add_missing makes it
index
   first:

   # doveadm search -u [email protected] mailbox INBOX text reproprobe
   Error: Mailbox INBOX: indexer failed to index mailbox
   Error: cmd search: Searching mailbox INBOX failed: Internal error
occurred.

   Over IMAP the client sees: NO [SERVERBUG] Internal error occurred.

It does not recover by itself
-----------------------------

"doveadm index" and autoindexing both take the same write path and fail the
same
way. "doveadm fts rescan" does not help either -- it also cannot open the
empty
shard. The only repair we found is to delete the folder's fts-flatcurve
directory and index the folder again.

Impact
------

A quiet mailbox looks fine for as long as nothing is delivered to it, so
this can
sit unnoticed. We found it because a monitoring mailbox that receives one
message
per minute started returning SERVERBUG on SEARCH. In our case a maintenance
run
that called "doveadm fts optimize" per mailbox left 114 mailboxes in this
state,
which is all of the ones that had any mail.

Possible cause (not verified)
-----------------------------

In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for
writing
("We need to lock all of the mailboxes so nothing changes while we are
optimizing"), and the old current.*/index.* directories are then deleted
from
disk while those write handles are still open. A handle that is closed
afterwards
appears to recreate its directory without a glass revision file, which
matches
what is left behind.

Also, in the same function the result of

    fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r)

is discarded: rename_db() does not update the path it is given and returns
the
new path only through newpath_r, which is NULL at this call site.

I have not tried to confirm either of these beyond reading the code.

Not the same as the known root-owned iamglass report
----------------------------------------------------

There is an older report where "doveadm fts optimize -A" run from cron as
root
left iamglass owned by root, producing the same error string. That is not
this:
here the directories are owned by the mail user (dovecot:dovecot, 0700) and
the
revision file does not exist at all.



-- 
Best regards,
Ihor Rusyn

_______________________________________________
dovecot mailing list -- [email protected]
To unsubscribe send an email to [email protected]