manual flushing thresholds for deletes?
Eric Wong <[email protected]> Fri, 24 Mar 2023 10:37:41 +0000
| Newsgroups | gmane.comp.search.xapian.general |
|---|---|
| Message-ID | <20230324103741.M171609@dcvr> |
Years ago, I ran into OOM problems with the default flush threshold of 10000 documents while indexing (add/replace). Realizing I had documents of hugely varying sizes (0.5KB..20MB) and little RAM, I instead tracked the number of raw bytes in the text being indexed and flushed whenever I'd seen a configurable byte count. Not the most scientific way, but it seems to work well enough on low-end systems. Now, I'm dealing with many deletes at once and hitting OOM again. Since the raw text is no longer available and I didn't store its original size anywhere, would calculating something based on get_doclength be a reasonable approximation? I'm wondering if something like: get_doclength * (1 + 3) * mean_term_length where 1x is for the mean term length itself, and 3x for the position overhead And perhaps assume mean_term_length is 10 bytes, so maybe: get_doclength * 40 ? I'm using Search::Xapian XS since it's in Debian stable; and don't think there's a standard way to show the amount of memory uncommitted changes are taking up. Thanks for any thoughts you have.