New features and release of Lucene 11 at end of year

Hi,

as you might have followed by talk at BerlinBuzzwords 
(https://2025.berlinbuzzwords.de/session/state-of-native-access-in-apache-lucene/, 
https://www.youtube.com/watch?v=4RxkahnUBvg), I'd like to make the 
proposal to release Lucene 11 with Java 25 in November or December this 
year.

The development model of Lucene has changed a bit the last years like 
the main branch is now always on latest Java version and we only 
backport features which are easy to backport to stable branch (no 
general refactorings or features that require new Java versions). The 
effect of this (see discussion in the talk above) is that we may need to 
align the release of new major versions of Lucene a bit with the LTS 
schedule of Java. Java releases a LTS relaese every 2 years, so we could 
do the same and release a Lucene major version also every 2 years based 
on main branch once it has all updates for the previous LTS release of 
Java. This does not mean we release directly after the LTS, but it 
should be done between the LTS release of Java and before we upgrade 
main branch again. E.g., we could fork the next stable branch after the 
LTS release and update of build system.

I think a new major release every 2 years is a good plan and also fits 
what Adrien and Luca told us on Berlinbuzzwords 
(https://2025.berlinbuzzwords.de/session/shipping-lucene-10-0-25-years-in-the-making/, 
https://www.youtube.com/watch?v=GRhzgCEL_ac). So lets start with this. 
OK, it is a bit shorter than 1 year, but we were lazy with

New features for Lucene 11:

  * rewrite of IOContext and hopefully better madvise soon (it is ongoing)
  * FAISS vector format in sandbox module (this was backported, but
    reverted as I don't think we should backport this due to the Java 21
    problems with preview APIs and incompatible changes in the FFM Api:
    https://github.com/apache/lucene/pull/14843)
  * Binary partitioning merge policy
  * expressions module no longer depending on ASM
  * priorityqueue hopefully gets faster once its final
  * ongoing panama vectorization improvements (I would not backport
    those), so they are new features
  * new fast and speedy build system
  * cleanup of all AccessController/SecurityManager/SecurityException
    (to be done...., will open PR soon)

If you have any cool feature that I forgot please add it here!

In general due to the changes in the build system, backports are getting 
harder from week to week. Sometimes even one-line backports get hard 
(change from Groovy to Java code,...), so I'd like to see Lucene 10 as 
some intermediate version and 11 should come soon.

We should possibly solve all issues with backwards compatibility that 
are long ongoing discussions before release:

  * allow to read older indexes or not? I would say we should only
    prevent anybody from reading indexes older than Lucene 8 or like that.

Uwe

-- 
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:[email protected]