[PR 16358] Making the doc values skip index extensible

Sagar Upadhyay <[email protected]>
Newsgroups gmane.comp.jakarta.lucene.devel
Message-ID <CA+UjSDbD2KnVeUv86FT-dWMMZkKZWK_CyTFWZUQ0jB7dus0bQQ@mail.gmail.com>
Hi all,

I opened PR #16358 a few weeks back and would appreciate some feedback:

https://github.com/apache/lucene/pull/16358

*Current pain point:*


   - Adding any new stat to the skip index today requires a format version
   bump, recalculating the fixed byte offsets, and gating reads behind version
   checks because the layout is a fixed schema where every byte position is
   predetermined. So it makes it hard to add new stats, or write a custom
   implementation if someone wants to.
   - Additionally, DocValuesSkipper as of today currently carries both
   navigation and stats, so a caller that only wants field-level stats has to
   open the .dvs file to get them



*The PR* first replaces the fixed layout with length-prefixed, type-tagged
entries, so readers skip stats they don't recognize and adding one needs no
version bump or offset math.


Second, it adds DocValuesField and SkipStat<T>: field-level stats come from
.dvm with no I/O, and callers can check which optional stats a field has
before opening the skipper.


Thanks,
Sagar
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.