Re: Codec-specific index structures on LeafReader
"Uwe Schindler via dev" <[email protected]> Tue, 7 Jul 2026 09:18:23 +0200
| Newsgroups | gmane.comp.jakarta.lucene.devel |
|---|---|
| Message-ID | <[email protected]> |
--------------meOaNs3kOrE30BaFz60DihOr
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Hi,
basically the same mechanism is used in JDBC and DataSource:
https://docs.oracle.com/en/java/javase/25/docs/api/java.sql/java/sql/Wrapper.html
The above example in JDBC is well-known to me. E.g., the Postgresql
driver has speciifc methods to escape table names or other identifiers
that you need to construct dynamic sql queries in a safe way. On the
other hand, JDBC connections are almost always wrapped in connection
caches and the only thing you have are the abstract Connection
interface. Casting never works as it is almost always somehow wrapped.
Generally it is a good idea to do this when you want calsses be extended
and make it useable with wrappers like FilterIndexReader. But you have
to be careful when unwrapping stuff that may possible need handling in
the filter! If we define such a method it should by default NOT return
NULL and instead throw an exception. The default impl would return
itsself if the type fits. FilterReader would call it on the delegate by
default.
P.S.: We have the functionality already in Lucene, but it is missing the
"cast" functionality:
https://lucene.apache.org/core/10_5_0/core/org/apache/lucene/util/Unwrappable.html
Uwe
Am 03.07.2026 um 15:34 schrieb Alan Woodward:
> Hi all,
>
> I’ve been catching up with the talks from Buzzwords this year and
> watching this (excellent!) talk from Shailesh
> (https://www.youtube.com/watch?v=ziRcxCJPU6s) on star-tree indexes in
> OpenSearch made me think again about the way we make index structures
> available on LeafReader. The star-tree index is a novel structure
> that I don’t think it would make sense to add to the base API, but on
> the other hand there isn’t currently an obvious extension point that
> makes it easy to use without doing some hairy casting of LeafReader
> types. We have a similar issue in elasticsearch with bloom filters -
> useful in a restricted set of circumstances, probably not helpful to
> have cluttering up the API, so we end up implementing it in a custom
> Codec and then casting at runtime to see if things are available (with
> the usual problems about FilterLeafReader wrapping that comes with
> relying on casts). This also applies to Codec-specific extensions to
> things like DocValueSkippers.
>
> What do people think about adding a new method to the LeafReader API
> that looks like this:
>
> /** Return a data structure accessed through type T if available */
> T getDataStructure(Class<T> structureType) {
> return null;
> }
>
> Expert users implementing their own Codecs can use this to return
> specialised data structures without polluting the top-level API or
> having to use fragile casts.
>
> Any opinions?
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:[email protected]
--------------meOaNs3kOrE30BaFz60DihOr
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,</p>
<p>basically the same mechanism is used in JDBC and DataSource:</p>
<p><a class="moz-txt-link-freetext" href="https://docs.oracle.com/en/java/javase/25/docs/api/java.sql/java/sql/Wrapper.html">https://docs.oracle.com/en/java/javase/25/docs/api/java.sql/java/sql/Wrapper.html</a></p>
<p>The above example in JDBC is well-known to me. E.g., the
Postgresql driver has speciifc methods to escape table names or
other identifiers that you need to construct dynamic sql queries
in a safe way. On the other hand, JDBC connections are almost
always wrapped in connection caches and the only thing you have
are the abstract Connection interface. Casting never works as it
is almost always somehow wrapped.</p>
<p>Generally it is a good idea to do this when you want calsses be
extended and make it useable with wrappers like
FilterIndexReader. But you have to be careful when unwrapping
stuff that may possible need handling in the filter! If we define
such a method it should by default NOT return NULL and instead
throw an exception. The default impl would return itsself if the
type fits. FilterReader would call it on the delegate by default.</p>
<p>P.S.: We have the functionality already in Lucene, but it is
missing the "cast" functionality:
<a class="moz-txt-link-freetext" href="https://lucene.apache.org/core/10_5_0/core/org/apache/lucene/util/Unwrappable.html">https://lucene.apache.org/core/10_5_0/core/org/apache/lucene/util/Unwrappable.html</a></p>
<p>Uwe</p>
<div class="moz-cite-prefix">Am 03.07.2026 um 15:34 schrieb Alan
Woodward:<br>
</div>
<blockquote type="cite"
cite="mid:[email protected]">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
Hi all,
<div><br>
</div>
<div>I’ve been catching up with the talks from Buzzwords this year
and watching this (excellent!) talk from Shailesh (<a
href="https://www.youtube.com/watch?v=ziRcxCJPU6s"
moz-do-not-send="true" class="moz-txt-link-freetext">https://www.youtube.com/watch?v=ziRcxCJPU6s</a>)
on star-tree indexes in OpenSearch made me think again about the
way we make index structures available on LeafReader. The
star-tree index is a novel structure that I don’t think it would
make sense to add to the base API, but on the other hand there
isn’t currently an obvious extension point that makes it easy to
use without doing some hairy casting of LeafReader types. We
have a similar issue in elasticsearch with bloom filters -
useful in a restricted set of circumstances, probably not
helpful to have cluttering up the API, so we end up implementing
it in a custom Codec and then casting at runtime to see if
things are available (with the usual problems about
FilterLeafReader wrapping that comes with relying on casts).
This also applies to Codec-specific extensions to things like
DocValueSkippers.</div>
<div><br>
</div>
<div>What do people think about adding a new method to the
LeafReader API that looks like this:</div>
<div><br>
</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>/**
Return a data structure accessed through type T if available */</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>T
getDataStructure(Class<T> structureType) {</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>return
null;</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div>
<div><br>
</div>
<div>Expert users implementing their own Codecs can use this to
return specialised data structures without polluting the
top-level API or having to use fragile casts.</div>
<div><br>
</div>
<div>Any opinions?</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
<a class="moz-txt-link-freetext" href="https://www.thetaphi.de">https://www.thetaphi.de</a>
eMail: <a class="moz-txt-link-abbreviated" href="mailto:[email protected]">[email protected]</a></pre>
</body>
</html>
--------------meOaNs3kOrE30BaFz60DihOr--