[PR 16358] Making the doc values skip index extensible
Sagar Upadhyay <[email protected]>
| Newsgroups | gmane.comp.jakarta.lucene.devel |
|---|---|
| Message-ID | <CA+UjSDbD2KnVeUv86FT-dWMMZkKZWK_CyTFWZUQ0jB7dus0bQQ@mail.gmail.com> |
Hi all, I opened PR #16358 a few weeks back and would appreciate some feedback: https://github.com/apache/lucene/pull/16358 *Current pain point:* - Adding any new stat to the skip index today requires a format version bump, recalculating the fixed byte offsets, and gating reads behind version checks because the layout is a fixed schema where every byte position is predetermined. So it makes it hard to add new stats, or write a custom implementation if someone wants to. - Additionally, DocValuesSkipper as of today currently carries both navigation and stats, so a caller that only wants field-level stats has to open the .dvs file to get them *The PR* first replaces the fixed layout with length-prefixed, type-tagged entries, so readers skip stats they don't recognize and adding one needs no version bump or offset math. Second, it adds DocValuesField and SkipStat<T>: field-level stats come from .dvm with no I/O, and callers can check which optional stats a field has before opening the skipper. Thanks, Sagar