Re: Codec-specific index structures on LeafReader

--00000000000099e0020655f5c4c9
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Alan,

Having 3 different needs for this would probably help clarify what common
denominator they need. :) Class-based signature is appealing from code
point of view but how will it work across versions, codecs, etc? A working
example (or, like I said, three examples) would probably help capture the
common needs and problems that may arise.

This said - it does feel like Lucene got a bit stagnant recently (call it
mature, if you like), so perhaps it would be a good breath of fresh air to
have something entirely new to work on, not just minor improvements.

Dawid

On Fri, Jul 3, 2026 at 3:35=E2=80=AFPM Alan Woodward <[email protected]>=
 wrote:

> Hi all,
>
> I=E2=80=99ve been catching up with the talks from Buzzwords this year and=
 watching
> this (excellent!) talk from Shailesh (
> https://www.youtube.com/watch?v=3DziRcxCJPU6s) on star-tree indexes in
> OpenSearch made me think again about the way we make index structures
> available on LeafReader.  The star-tree index is a novel structure that I
> don=E2=80=99t think it would make sense to add to the base API, but on th=
e other
> hand there isn=E2=80=99t currently an obvious extension point that makes =
it easy to
> use without doing some hairy casting of LeafReader types.  We have a
> similar issue in elasticsearch with bloom filters - useful in a restricte=
d
> set of circumstances, probably not helpful to have cluttering up the API,
> so we end up implementing it in a custom Codec and then casting at runtim=
e
> to see if things are available (with the usual problems about
> FilterLeafReader wrapping that comes with relying on casts).  This also
> applies to Codec-specific extensions to things like DocValueSkippers.
>
> What do people think about adding a new method to the LeafReader API that
> looks like this:
>
> /** Return a data structure accessed through type T if available */
> T getDataStructure(Class<T> structureType) {
> return null;
> }
>
> Expert users implementing their own Codecs can use this to return
> specialised data structures without polluting the top-level API or having
> to use fragile casts.
>
> Any opinions?
>

--00000000000099e0020655f5c4c9
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><br><div>Hi Alan,</div><div><br></div><div>Having 3 differ=
ent needs for this would probably help clarify what common denominator they=
 need. :) Class-based signature is appealing from code point of view but ho=
w will it work across versions, codecs, etc? A working example (or, like I =
said, three examples) would probably help capture the common needs and prob=
lems that may arise.</div><div><br></div><div>This said - it does feel like=
 Lucene got a bit stagnant recently (call it mature, if you like), so perha=
ps it would be a good breath of fresh air to have something entirely new to=
 work on, not just minor improvements.</div><div><br></div><div>Dawid</div>=
</div><br><div class=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr"=
 class=3D"gmail_attr">On Fri, Jul 3, 2026 at 3:35=E2=80=AFPM Alan Woodward =
&lt;<a href=3D"mailto:[email protected]">[email protected]</a>&gt; wr=
ote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px=
 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Hi all=
,<div><br></div><div>I=E2=80=99ve been catching up with the talks from Buzz=
words this year and watching this (excellent!) talk from Shailesh (<a href=
=3D"https://www.youtube.com/watch?v=3DziRcxCJPU6s" target=3D"_blank">https:=
//www.youtube.com/watch?v=3DziRcxCJPU6s</a>) on star-tree indexes in OpenSe=
arch made me think again about the way we make index structures available o=
n LeafReader.=C2=A0 The star-tree index is a novel structure that I don=E2=
=80=99t think it would make sense to add to the base API, but on the other =
hand there isn=E2=80=99t currently an obvious extension point that makes it=
 easy to use without doing some hairy casting of LeafReader types.=C2=A0 We=
 have a similar issue in elasticsearch with bloom filters - useful in a res=
tricted set of circumstances, probably not helpful to have cluttering up th=
e API, so we end up implementing it in a custom Codec and then casting at r=
untime to see if things are available (with the usual problems about Filter=
LeafReader wrapping that comes with relying on casts).=C2=A0 This also appl=
ies to Codec-specific extensions to things like DocValueSkippers.</div><div=
><br></div><div>What do people think about adding a new method to the LeafR=
eader API that looks like this:</div><div><br></div><div><span style=3D"whi=
te-space:pre-wrap">	</span>/** Return a data structure accessed through typ=
e T if available */</div><div><span style=3D"white-space:pre-wrap">	</span>=
T getDataStructure(Class&lt;T&gt; structureType) {</div><div><span style=3D=
"white-space:pre-wrap">		</span>return null;</div><div><span style=3D"white=
-space:pre-wrap">	</span>}</div><div><br></div><div>Expert users implementi=
ng their own Codecs can use this to return specialised data structures with=
out polluting the top-level API or having to use fragile casts.</div><div><=
br></div><div>Any opinions?</div></div></blockquote></div>

--00000000000099e0020655f5c4c9--