Re: Exploring Telemetry for Lucene

Hi,

> On 3 Oct 2025, at 14:37, David Smiley <[email protected]> wrote:
> 
> IMO Lucene should just be a library with ~no dependencies.  It just needs to expose the right callbacks to allow for easy observability higher up the stack.

++ This makes sense to me. I’d rather make telemetry more straightforward to capture, rather than try to capture it.

-Chris.

> 
> Today I was looking at Solr's hooks into Lucene on this topic.  SolrIndexWriter overrides a protected method doAfterFlush allowing Solr to increment a counter for every flushed segment.  Pretty easy.  Although it's not clear, looking at IndexWriter, how to get any information about the size of that segment in docs or bytes; if someone sees how to do that elegantly, please let me know.  Without that detail, you can't compute a good write-amplification-factor.
> 
> ~ David
> 
> On Wed, Mar 19, 2025 at 2:16 AM Vigya Sharma <[email protected]> wrote:
> Hi All,
> 
> I am debugging an unexpected throughput regression in my feature branch which has changes across multiple files and functions, and was wondering if there's a better way to identify the slowdown than adding elapsed time logs to infostream. 
> 
> Do we have support in Lucene for telemetry on internal operations? It would be nice to be able to add timers, counters, and metrics, to identify performance changes. This would hide disabled behind a gradle flag. We could enable it for benchmark runs and possibly chart these metrics in nightly runs.
> 
> Has this been discussed before? I'm happy to explore this more if it sounds like a good idea. Any suggestions for lightweight, open-source telemetry libraries that I should look into? I'm considering OpenTelemetry but open to alternate suggestions and ideas.
> 
> Vigya