Re: New features and release of Lucene 11 at end of year

+1 bumping this up!

The list of cool features in Lucene 11 has grown over the past year (not exhaustive, please add more here!):

* Native vector computations (https://github.com/apache/lucene/pull/15508)
* More performant query cache (https://github.com/apache/lucene/pull/15558)
* FP16 vector encoding (https://github.com/apache/lucene/pull/16383)
...

Panama FFM was also finalized in Java 22 (https://openjdk.org/jeps/454), I wonder if that makes memory segment related changes easier to maintain too?

I'm also happy to help with the release process, unless someone else wants to do it!

- Kaival

On 2025/07/09 12:52:21 Uwe Schindler wrote:
> Hi,
> 
> as you might have followed by talk at BerlinBuzzwords 
> (https://2025.berlinbuzzwords.de/session/state-of-native-access-in-apache-lucene/, 
> https://www.youtube.com/watch?v=4RxkahnUBvg), I'd like to make the 
> proposal to release Lucene 11 with Java 25 in November or December this 
> year.
> 
> The development model of Lucene has changed a bit the last years like 
> the main branch is now always on latest Java version and we only 
> backport features which are easy to backport to stable branch (no 
> general refactorings or features that require new Java versions). The 
> effect of this (see discussion in the talk above) is that we may need to 
> align the release of new major versions of Lucene a bit with the LTS 
> schedule of Java. Java releases a LTS relaese every 2 years, so we could 
> do the same and release a Lucene major version also every 2 years based 
> on main branch once it has all updates for the previous LTS release of 
> Java. This does not mean we release directly after the LTS, but it 
> should be done between the LTS release of Java and before we upgrade 
> main branch again. E.g., we could fork the next stable branch after the 
> LTS release and update of build system.
> 
> I think a new major release every 2 years is a good plan and also fits 
> what Adrien and Luca told us on Berlinbuzzwords 
> (https://2025.berlinbuzzwords.de/session/shipping-lucene-10-0-25-years-in-the-making/, 
> https://www.youtube.com/watch?v=GRhzgCEL_ac). So lets start with this. 
> OK, it is a bit shorter than 1 year, but we were lazy with
> 
> New features for Lucene 11:
> 
>   * rewrite of IOContext and hopefully better madvise soon (it is ongoing)
>   * FAISS vector format in sandbox module (this was backported, but
>     reverted as I don't think we should backport this due to the Java 21
>     problems with preview APIs and incompatible changes in the FFM Api:
>     https://github.com/apache/lucene/pull/14843)
>   * Binary partitioning merge policy
>   * expressions module no longer depending on ASM
>   * priorityqueue hopefully gets faster once its final
>   * ongoing panama vectorization improvements (I would not backport
>     those), so they are new features
>   * new fast and speedy build system
>   * cleanup of all AccessController/SecurityManager/SecurityException
>     (to be done...., will open PR soon)
> 
> If you have any cool feature that I forgot please add it here!
> 
> In general due to the changes in the build system, backports are getting 
> harder from week to week. Sometimes even one-line backports get hard 
> (change from Groovy to Java code,...), so I'd like to see Lucene 10 as 
> some intermediate version and 11 should come soon.
> 
> We should possibly solve all issues with backwards compatibility that 
> are long ongoing discussions before release:
> 
>   * allow to read older indexes or not? I would say we should only
>     prevent anybody from reading indexes older than Lucene 8 or like that.
> 
> Uwe
> 
> -- 
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail:[email protected]
>