Re: Extensibility of DocValueSkippers

--00000000000034a26c065259e2d9
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I also like the idea of having a pluggable skipper system where it is easy
for anyone to introduce additional metadata, instead of using one
monolithic DocValuesSkipper. I also have some ideas to store additional
metadata to improve the performance for doc values.

Also, I wonder if we can make per-block operations pluggable too. Today we
fetch metadata, classify the block, then read and decode values separately
through virtual calls. With a per-block evaluator (e.g.,
skipper.getBlockEvaluator().evaluateRange()), we could push predicate
evaluation down to the codec, combine decoding and comparison in a single
pass, probably eliminating per-element virtual dispatch along the way. This
might also unlock more SIMD benefits.

On Mon, May 18, 2026 at 4:52=E2=80=AFAM Ignacio Vera <[email protected]> wr=
ote:

> +1 I like the idea of having a proper extensibility mechanism. I feel
> adding sentinel values to signal if a value is present or not is fragile.
>
> I do think we should follow the Points and Terms design and have an
> intermediate object that allows accessing the static metadata of an index
> without having to create any search data structures.
>
> Cheers,
>
> Ignacio
>
> On Fri, May 15, 2026 at 11:00=E2=80=AFAM Alan Woodward <romseygeek@gmail.=
com>
> wrote:
>
>> Hi folks,
>>
>> We have a few open PRs adding new data to the DocValuesSkipper interface
>> (eg https://github.com/apache/lucene/pull/15993,
>> https://github.com/apache/lucene/pull/15737), and other open issues
>> discussing adding more (https://github.com/apache/lucene/issues/15884).
>> We also have some ideas here at elastic for other bits of information th=
at
>> would be useful in highly specific circumstances but not really in the
>> general case.  These all run into issues with backwards compatibility, a=
nd
>> questions of how to reliably signal to clients what data is available fo=
r a
>> given field and segment.
>>
>> One idea I had that would make this a bit more pluggable, and allow
>> Codecs to add additional block-based data without having to alter the ba=
se
>> API too much, is to add a SkipType object which would be passed to the
>> LeafReader like so:
>>
>> T getDocValuesSkipper(SkipType<T extends DocValuesSkipper> type)
>>
>> The codec would check the class of the SkipType and see if it knows how
>> to return that information.  If yes, it returns an instance of T, if not=
 it
>> returns null.  The default type would be a Range<DocValuesSkipper>, whic=
h
>> would return the basic DocValuesSkipper that we have now, but we can ext=
end
>> things with a Count or Cardinality type.  On the indexing side, the
>> FieldInfo could record the SkipType so that the codec knows what metadat=
a
>> to generate.
>>
>> Some of these bits of information are useful both as global metadata and
>> as part of a skip block; some are only really relevant at the global
>> level.  Tying into the work that Ignacio is doing in
>> https://github.com/apache/lucene/issues/16052, the global metadata tends
>> to be loaded at segment open time and so can be accessed cheaply without
>> doing any IO, but because it is part of the general DocValuesSkipper obj=
ect
>> it can only be accessed by calling LeafReader.getDocValuesSkipper() whic=
h
>> loads a bunch of extra data (and declares that it does IO via its throws
>> clause).
>>
>> We could add an intermediate object here, analogous to Points or Terms,
>> called DocValues (or something similar, I know this is already a class w=
ith
>> static helper methods on it); this would make the global min, max and
>> docCount (and maybe cardinality) available without having to do any furt=
her
>> IO, and the getSkipper() method could optionally be moved onto the
>> intermediate object.
>>
>> What do people think?
>>
>> - Alan
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>>
>>

--00000000000034a26c065259e2d9
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I also like the idea of having a pluggable skipper system =
where it is easy for anyone to introduce additional metadata, instead of us=
ing one monolithic=C2=A0DocValuesSkipper. I also have some ideas to=C2=A0st=
ore additional metadata=C2=A0to improve the performance for doc values.<div=
><br></div><div>





<p class=3D"gmail-p1" style=3D"margin:0px;font-style:normal;font-variant:no=
rmal;font-size-adjust:none;font-kerning:auto;font-feature-settings:normal;f=
ont-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face=3D"arial=
, sans-serif" style=3D""><span class=3D"gmail-s1" style=3D"font-variant-lig=
atures:no-common-ligatures">Also, I wonder if we can make per-block operati=
ons pluggable too. Today we fetch metadata, classify the block, then read a=
nd decode values separately through virtual calls. With a per-block evaluat=
or (e.g., </span><span class=3D"gmail-s2" style=3D"font-variant-ligatures:n=
o-common-ligatures;color:rgb(60,95,255)">skipper.getBlockEvaluator().evalua=
teRange()</span><span class=3D"gmail-s1" style=3D"font-variant-ligatures:no=
-common-ligatures">), we could push=C2=A0</span><span style=3D"font-variant=
-ligatures:no-common-ligatures">predicate evaluation down to the codec, com=
bine decoding and comparison in a single pass, probably eliminating per-ele=
ment virtual dispatch along the way. This might also unlock more SIMD benef=
its.</span></font></p></div>











</div><br><div class=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr"=
 class=3D"gmail_attr">On Mon, May 18, 2026 at 4:52=E2=80=AFAM Ignacio Vera =
&lt;<a href=3D"mailto:[email protected]">[email protected]</a>&gt; wrote:<b=
r></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex=
;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr">=
+1 I like the idea of having a proper extensibility mechanism. I feel addin=
g sentinel values to signal if a value is present or not is fragile.<br><di=
v><br></div><div>I do think we should follow the Points and Terms design an=
d have an intermediate object that allows accessing=C2=A0the static metadat=
a of an index without having to create any search data structures.=C2=A0</d=
iv><div><br></div><div>Cheers,</div><div><br></div><div>Ignacio</div></div>=
<br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Fri=
, May 15, 2026 at 11:00=E2=80=AFAM Alan Woodward &lt;<a href=3D"mailto:roms=
[email protected]" target=3D"_blank">[email protected]</a>&gt; wrote:<br>=
</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;b=
order-left:1px solid rgb(204,204,204);padding-left:1ex">Hi folks,<br>
<br>
We have a few open PRs adding new data to the DocValuesSkipper interface (e=
g <a href=3D"https://github.com/apache/lucene/pull/15993" rel=3D"noreferrer=
" target=3D"_blank">https://github.com/apache/lucene/pull/15993</a>, <a hre=
f=3D"https://github.com/apache/lucene/pull/15737" rel=3D"noreferrer" target=
=3D"_blank">https://github.com/apache/lucene/pull/15737</a>), and other ope=
n issues discussing adding more (<a href=3D"https://github.com/apache/lucen=
e/issues/15884" rel=3D"noreferrer" target=3D"_blank">https://github.com/apa=
che/lucene/issues/15884</a>). We also have some ideas here at elastic for o=
ther bits of information that would be useful in highly specific circumstan=
ces but not really in the general case.=C2=A0 These all run into issues wit=
h backwards compatibility, and questions of how to reliably signal to clien=
ts what data is available for a given field and segment.<br>
<br>
One idea I had that would make this a bit more pluggable, and allow Codecs =
to add additional block-based data without having to alter the base API too=
 much, is to add a SkipType object which would be passed to the LeafReader =
like so:<br>
<br>
T getDocValuesSkipper(SkipType&lt;T extends DocValuesSkipper&gt; type)<br>
<br>
The codec would check the class of the SkipType and see if it knows how to =
return that information.=C2=A0 If yes, it returns an instance of T, if not =
it returns null.=C2=A0 The default type would be a Range&lt;DocValuesSkippe=
r&gt;, which would return the basic DocValuesSkipper that we have now, but =
we can extend things with a Count or Cardinality type.=C2=A0 On the indexin=
g side, the FieldInfo could record the SkipType so that the codec knows wha=
t metadata to generate.<br>
<br>
Some of these bits of information are useful both as global metadata and as=
 part of a skip block; some are only really relevant at the global level.=
=C2=A0 Tying into the work that Ignacio is doing in <a href=3D"https://gith=
ub.com/apache/lucene/issues/16052" rel=3D"noreferrer" target=3D"_blank">htt=
ps://github.com/apache/lucene/issues/16052</a>, the global metadata tends t=
o be loaded at segment open time and so can be accessed cheaply without doi=
ng any IO, but because it is part of the general DocValuesSkipper object it=
 can only be accessed by calling LeafReader.getDocValuesSkipper() which loa=
ds a bunch of extra data (and declares that it does IO via its throws claus=
e).<br>
<br>
We could add an intermediate object here, analogous to Points or Terms, cal=
led DocValues (or something similar, I know this is already a class with st=
atic helper methods on it); this would make the global min, max and docCoun=
t (and maybe cardinality) available without having to do any further IO, an=
d the getSkipper() method could optionally be moved onto the intermediate o=
bject.<br>
<br>
What do people think?<br>
<br>
- Alan<br>
---------------------------------------------------------------------<br>
To unsubscribe, e-mail: <a href=3D"mailto:[email protected]=
" target=3D"_blank">[email protected]</a><br>
For additional commands, e-mail: <a href=3D"mailto:[email protected]=
rg" target=3D"_blank">[email protected]</a><br>
<br>
</blockquote></div>
</blockquote></div>

--00000000000034a26c065259e2d9--