Re: Lucene PR request for review: #16357

I guess what I'm missing here is why cross-node coordination belongs
in Lucene, which generally only concerns itself with operations on a
single node.  Maybe I haven't properly understood the proposal? My
understanding is that the new classes would operate in a
coordinator/collector/collator node that is responsible for calling
services running on shards and then combining the results: is that
right?

On Sat, Aug 15, 2026 at 8:02 PM Kristian Rickert <[email protected]> wrote:
>
> Hi lucene devs,
>
> With the advice of some lucene devs, I've been developing a shared-floor kNN collection for Lucene.  Having a shared floor allows searches to have a collaborative results live over multiple shards. I've created a PoC search so a gRPC bidi coordinator stream can judge if it has collected the correct number of K values, rather than requiring all K values in an HNSW search as standard distributed search engines demand.
>
> The surface area is small, and tests show no impact on baseline performance or recall. After developing and testing this for nearly a year, I have seen latency decrease by over 50% in high-latency environments for large values of K.  For values of K < 100, the results vary but not any better or worse in most setups.  In low latency environments (fast machines), the feature provides minimal impact for low values of K (under 100).  More testing is needed to see if multiple machines could add value. Since I only have 2 fast machines so I can't test distributed shards with more.
>
> On a fleet of raspberry pis, it helps tremendously with multiple shards.
>
> I am still tweaking the implementation and I'm preparing for a large-scale test using over 500GB of text across 8 shards (I've tested 100GB with success). However, I would like to get a review at this stage to understand what else would be required to land this feature on the mainline.  Or if someone has a fleet of fast machines, we can test a PoC search engine that collaborates lucene results.
>
> Also, I'd love some feedback and see if we can get next steps.
>
> Could someone please review the code and provide feedback?
>
> Best regards,
> Kristian Rickert
>