MultiDatabase shard count limitations

Eric Wong <[email protected]> Fri, 21 Aug 2020 09:06:59 +0000
Newsgroups gmane.comp.search.xapian.general
Message-ID <20200821090659.GA30800@dcvr>
Going back to the "prioritizing aggregated DBs" thread from
February 2020, I've got 390 Xapian shards for 130 public inboxes
I want to search against(*).  There's more on the horizon (we're
expecting tens of thousands of public inboxes).

After bumping RLIMIT_NOFILE and running ->add_database a bunch,
the actual queries seem to be taking ~30s (not good :x).

Now I'm thinking, MultiDatabase isn't the right way to go about
this...

Perhaps creating a new, all-encompassing Xapian index with a
reasonable shard count would be wise, at least for the normal
WWW frontend?

Managing removals of entire inboxes from an all-encompassing
Xapian DB would get much trickier.

IMAP search would still require per-mailbox indices, I think;
because UIDs are currently tied to NNTP article numbers.
Some attributes such as INTERNALDATE (Received: time) and
exact byte sizes would differ if the same message is
cross-posted to multiple public mailing lists, too.