Re: Solr performance degradation for collapse queries with string sort fields after migration from Solr 8.11.2 to Solr 9.10.1

--000000000000f0288e065505c49c
Content-Type: multipart/alternative; boundary="000000000000f0288b065505c49b"

--000000000000f0288b065505c49b
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I don't know a direct answer to your questions, but some context of why you
are you running a collapse query on 7m documents could help provide
insight? What are you trying to achieve? Are the results to be paged in a
ui? Is it an analytics workload?



On Wed, Jun 24, 2026, 21:46 Bartosz Fidrysiak <[email protected]>
wrote:

> We identified a 2=E2=80=933x performance regression in Solr 9.10.1 compar=
ed to
> Solr 8.11.2 for collapse
> queries that use a string field as a collapse sort field.
>
>
> Test setup
> ----------
>
> To measure the regression under real production conditions, we configured
> both clusters to receive identical traffic simultaneously =E2=80=94 every=
 Solr
> request is sent to both instances at the same time, making the comparison
> direct and unbiased. Both clusters have the same number of nodes,
> documents, shards, and shard ranges. The data is sharded by tenant ID, so
> each request is served by a single shard with no cross-shard overhead. So=
lr
> schema is the same for both clusters.
>
> We tested six query variants covering different combinations of collapse
> sort fields: no collapse, collapse with date sort, date+long sort,
> date+string sort, and string-only sort (see attachments). The results sho=
w
> that queries with a string field in the collapse sort are consistently an=
d
> significantly slower in Solr 9, while queries using only numeric or date
> sort fields show no regression. Notably, the string field used in the
> collapse sort has very high cardinality, and the worst-case queries proce=
ss
> millions of documents.
>
>
> [image: image.png]
> [image: image.png]
> [image: image.png]
>
> Root cause
> ----------
>
> JFR profiling of the worst-case query (sort=3D"modified_date desc,
> document_id asc", ~7M documents) confirmed the root cause.
> [image: image.png]
>
> Lucene 9 changed the internal format for SortedDocValues
> (Lucene90DocValuesProducer). The term dictionary (TermsDict) now stores
> string values in LZ4-compressed blocks. In Lucene 8, the same data was he=
ld
> uncompressed in direct memory =E2=80=94 reads were instant. In Lucene 9, =
every time
> the collapse logic needs to materialize a string value for comparison or =
to
> record a new group winner, it must decompress an LZ4 block. For ~7M
> documents, this decompression is triggered on nearly every document via t=
he
> following call chain:
>
>   SortFieldsCompare
>     -> TermOrdValLeafComparator.copy()
>     -> lookupOrd()
>     -> TermsDict.decompressBlock()
>     -> LZ4.decompress()
>
> LZ4 decompression accounts for almost 40% of CPU time in the query-servin=
g
> thread in Solr 9,
> versus near zero in Solr 8.
>
> Similar concerns were raised in
> https://github.com/apache/lucene/issues/11485
>
> Questions
> ---------
>
> Q1: What are your recommendations for improving the performance of
> collapse queries that use a string field as a sort tiebreaker in Solr 9?
>
> Q2: Is it possible to disable LZ4 compression for SortedDocValues term
> dictionaries =E2=80=94 either via a configuration property or a docValues=
Format
> option =E2=80=94 or is this something that could be planned for a future =
release?
>
> Q3: Would it be feasible to lazily materialize string field values in
> CollapsingQParserPlugin for group winners, so that lookupOrd() is only
> called when a cross-segment comparison is actually needed? This could
> improve performance for queries where most groups contain only one docume=
nt.
>
> Kind regards,
> Bartosz
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]

--000000000000f0288b065505c49b
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"auto"><div>I don&#39;t know a direct answer to your questions, =
but some context of why you are you running a collapse query on 7m document=
s could help provide insight? What are you trying to achieve? Are the resul=
ts to be paged in a ui? Is it an analytics workload?=C2=A0</div><div dir=3D=
"auto"><br></div><div dir=3D"auto"><br><br><div class=3D"gmail_quote gmail_=
quote_container" dir=3D"auto"><div dir=3D"ltr" class=3D"gmail_attr">On Wed,=
 Jun 24, 2026, 21:46 Bartosz Fidrysiak &lt;<a href=3D"mailto:solr.universe@=
gmail.com">[email protected]</a>&gt; wrote:<br></div><blockquote clas=
s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad=
ding-left:1ex"><div dir=3D"ltr"><div class=3D"gmail_quote"><div dir=3D"ltr"=
 class=3D"gmail_attr"><span style=3D"background-color:transparent">We ident=
ified a 2=E2=80=933x performance regression in Solr 9.10.1 compared to Solr=
 8.11.2 for collapse</span></div><div dir=3D"ltr">queries that use a string=
 field as a collapse sort field.<br><br><br>Test setup<br>----------<br><br=
>To measure the regression under real production conditions, we configured =
both clusters to receive identical traffic simultaneously =E2=80=94 every S=
olr request is sent to both instances at the same time, making the comparis=
on direct and unbiased. Both clusters have the same number of nodes, docume=
nts, shards, and shard ranges. The data is sharded by tenant ID, so each re=
quest is served by a single shard with no cross-shard overhead. Solr schema=
 is the same for both clusters.<br><br>We tested six query variants coverin=
g different combinations of collapse sort fields: no collapse, collapse wit=
h date sort, date+long sort, date+string sort, and string-only sort (see at=
tachments). The results show that queries with a string field in the collap=
se sort are consistently and significantly slower in Solr 9, while queries =
using only numeric or date sort fields show no regression. Notably, the str=
ing field used in the collapse sort has very high cardinality, and the wors=
t-case queries process millions of documents.<div><br></div><div></div><br>=
<div><img src=3D"cid:ii_mqs3ex2i0" alt=3D"image.png" width=3D"542" height=
=3D"269"><br><img src=3D"cid:ii_mqs3fqdu2" alt=3D"image.png" width=3D"542" =
height=3D"430"></div><div><img src=3D"cid:ii_mqs3g6c53" alt=3D"image.png" w=
idth=3D"542" height=3D"361"><br><br>Root cause<br>----------<br><br>JFR pro=
filing of the worst-case query (sort=3D&quot;modified_date desc, document_i=
d asc&quot;, ~7M documents) confirmed the root cause.</div><div><img src=3D=
"cid:ii_mqs3v9e511" alt=3D"image.png" width=3D"562" height=3D"467"><br><br>=
Lucene 9 changed the internal format for SortedDocValues (Lucene90DocValues=
Producer). The term dictionary (TermsDict) now stores string values in LZ4-=
compressed blocks. In Lucene 8, the same data was held uncompressed in dire=
ct memory =E2=80=94 reads were instant. In Lucene 9, every time the collaps=
e logic needs to materialize a string value for comparison or to record a n=
ew group winner, it must decompress an LZ4 block. For ~7M documents, this d=
ecompression is triggered on nearly every document via the following call c=
hain:<br><br>=C2=A0 SortFieldsCompare<br>=C2=A0 =C2=A0 -&gt; TermOrdValLeaf=
Comparator.copy()<br>=C2=A0 =C2=A0 -&gt; lookupOrd()<br>=C2=A0 =C2=A0 -&gt;=
 TermsDict.decompressBlock()<br>=C2=A0 =C2=A0 -&gt; LZ4.decompress()<br><br=
>LZ4 decompression accounts for almost 40% of CPU time in the query-serving=
 thread in Solr 9,<br>versus near zero in Solr 8.<br><br>Similar concerns w=
ere raised in <a href=3D"https://github.com/apache/lucene/issues/11485" tar=
get=3D"_blank" rel=3D"noreferrer">https://github.com/apache/lucene/issues/1=
1485</a><br><br>Questions<br>---------<br><br>Q1: What are your recommendat=
ions for improving the performance of collapse queries that use a string fi=
eld as a sort tiebreaker in Solr 9?<br><br>Q2: Is it possible to disable LZ=
4 compression for SortedDocValues term dictionaries =E2=80=94 either via a =
configuration property or a docValuesFormat option =E2=80=94 or is this som=
ething that could be planned for a future release?<br><br>Q3: Would it be f=
easible to lazily materialize string field values in CollapsingQParserPlugi=
n for group winners, so that lookupOrd() is only called when a cross-segmen=
t comparison is actually needed? This could improve performance for queries=
 where most groups contain only one document.<br></div><div><br></div><div>=
Kind regards,</div><div>Bartosz</div></div>
</div></div>
<br>
---------------------------------------------------------------------<br>
To unsubscribe, e-mail: <a href=3D"mailto:[email protected]=
he.org" target=3D"_blank" rel=3D"noreferrer">[email protected]=
pache.org</a><br>
For additional commands, e-mail: <a href=3D"mailto:[email protected]=
ache.org" target=3D"_blank" rel=3D"noreferrer">[email protected]=
.org</a></blockquote></div></div></div>

--000000000000f0288b065505c49b--
--000000000000f0288e065505c49c--