Re: Exploring Telemetry for Lucene

On Fri, Oct 3, 2025 at 7:37 AM Chris Hegarty
<[email protected]> wrote:

> Hi,
>
> > On 3 Oct 2025, at 14:37, David Smiley <[email protected]> wrote:
> >
> > IMO Lucene should just be a library with ~no dependencies.  It just
> needs to expose the right callbacks to allow for easy observability higher
> up the stack.
>
> ++ This makes sense to me. I’d rather make telemetry more straightforward
> to capture, rather than try to capture it.
>

To this end I think it would be nice if there was an observer pattern for
these kinds of callbacks rather than extend-and-override. This would also
make it easier to build integrations for different observability frameworks.


> -Chris.
>
> >
> > Today I was looking at Solr's hooks into Lucene on this topic.
> SolrIndexWriter overrides a protected method doAfterFlush allowing Solr to
> increment a counter for every flushed segment.  Pretty easy.  Although it's
> not clear, looking at IndexWriter, how to get any information about the
> size of that segment in docs or bytes; if someone sees how to do that
> elegantly, please let me know.  Without that detail, you can't compute a
> good write-amplification-factor.
> >
> > ~ David
> >
> > On Wed, Mar 19, 2025 at 2:16 AM Vigya Sharma <[email protected]>
> wrote:
> > Hi All,
> >
> > I am debugging an unexpected throughput regression in my feature branch
> which has changes across multiple files and functions, and was wondering if
> there's a better way to identify the slowdown than adding elapsed time logs
> to infostream.
> >
> > Do we have support in Lucene for telemetry on internal operations? It
> would be nice to be able to add timers, counters, and metrics, to identify
> performance changes. This would hide disabled behind a gradle flag. We
> could enable it for benchmark runs and possibly chart these metrics in
> nightly runs.
> >
> > Has this been discussed before? I'm happy to explore this more if it
> sounds like a good idea. Any suggestions for lightweight, open-source
> telemetry libraries that I should look into? I'm considering OpenTelemetry
> but open to alternate suggestions and ideas.
> >
> > Vigya
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>