Re: Exploring Telemetry for Lucene

IMO Lucene should just be a library with ~no dependencies.  It just needs
to expose the right callbacks to allow for easy observability higher up the
stack.

Today I was looking at Solr's hooks into Lucene on this topic.
SolrIndexWriter overrides a protected method doAfterFlush allowing Solr to
increment a counter for every flushed segment.  Pretty easy.  Although it's
not clear, looking at IndexWriter, how to get any information about the
size of that segment in docs or bytes; if someone sees how to do that
elegantly, please let me know.  Without that detail, you can't compute a
good write-amplification-factor.

~ David

On Wed, Mar 19, 2025 at 2:16 AM Vigya Sharma <[email protected]> wrote:

> Hi All,
>
> I am debugging an unexpected throughput regression in my feature branch
> which has changes across multiple files and functions, and was wondering if
> there's a better way to identify the slowdown than adding elapsed time logs
> to infostream.
>
> Do we have support in Lucene for telemetry on internal operations? It
> would be nice to be able to add timers, counters, and metrics, to
> identify performance changes. This would hide disabled behind a gradle
> flag. We could enable it for benchmark runs and possibly chart these
> metrics in nightly runs.
>
> Has this been discussed before? I'm happy to explore this more if it
> sounds like a good idea. Any suggestions for lightweight, open-source
> telemetry libraries that I should look into? I'm considering OpenTelemetry
> but open to alternate suggestions and ideas.
>
> Vigya
>