Re: Crashes caused by high deleted .dvd file mmap counts

Hi,

Did the sharedArenaMaxPermits=64 help.

Actually sorry for the answer, I did not recognize that you were talking 
about doc values updates. I just saw deleted. But basically the issue is 
the same: Every update or delete will create a new file belonging to 
same segment. As each segment by default can have 1024 mappings this can 
sum up quite fast to the max mmap count. A typical index has 20 
segments, so this could sum up to 20.000 mappings per index.

I don't remember why Chris set the limit to 1024. In most cases segments 
will only have a dozen files maximum and if you do normal update ratios 
the number of open mappings should be fine to keep limited to 64 or even 
lower.

If the change helps for you we can open an issue to adapt the defaults 
for the shared arenas. BTW, if you want to go back to the default of 
previous lucene versions use 1, but this could have a degradion of 
performance when you have many updates.

Uwe

Am 07.05.2025 um 21:48 schrieb Justin Borromeo:
> Hi Uwe,
>
> Thanks for the response.  We've tried setting sharedArenaMaxPermits to 64;
> I'll update this thread once we get some data.
>
> One thing I don't understand is why does the list of deleted mmapped
> fields only include doc values files?  If your theory is correct and this
> is caused by deletes being updated over and over, wouldn't we expect only
> .liv files to be deleted?
>
> Output:
> ```
> 7ed8a2754000-7ed8a2757000 r--s 00000000 08:10 80872480
> /usr/share/opensearch/data/nodes/0/indices/Ci3MyIbNTceUmC67d1IlwQ/196/index/_9h0q_de_Lucene90_0.dvd
> (deleted)
> 7ed8a2757000-7ed8a275c000 r--s 00000000 08:10 78838113
> /usr/share/opensearch/data/nodes/0/indices/Ci3MyIbNTceUmC67d1IlwQ/119/index/_912j_m9_Lucene90_0.dvd
> (deleted)
> 7ed8a275c000-7ed8a275f000 r--s 00000000 08:10 78830146
> /usr/share/opensearch/data/nodes/0/indices/Ci3MyIbNTceUmC67d1IlwQ/126/index/_9buk_4e_Lucene90_0.dvd
> (deleted)
> ```
>
> Justin
>
> On Wed, May 7, 2025 at 9:50 AM Uwe Schindler <[email protected]> wrote:
>
>> Hi,
>>
>> this could be related to a bug or limitation of the following change:
>>
>>   1. GITHUB#13570
>>      <https://github.com/apache/lucene/pull/13570>,GITHUB#13574
>>      <https://github.com/apache/lucene/pull/13574>,GITHUB#13535
>>      <https://github.com/apache/lucene/pull/13535>: Avoid performance
>>      degradation with closing shared Arenas. Closing many individual
>>      index files can potentially lead to a degradation in execution
>>      performance. Index files are mmapped one-to-one with the JDK's
>>      foreign shared Arena. The JVM deoptimizes the top few frames of all
>>      threads when closing a shared Arena (see JDK-8335480). We mitigate
>>      this situation when running with JDK 21 and greater, by *1) using a
>>      confined Arena where appropriate, and 2) grouping files from the
>>      same segment to a single shared Arena*. A system property has been
>>      added that allows to control the total maximum number of mmapped
>>      files that may be associated with a single shared Arena. For
>>      example, to set the max number of permits to 256, pass the following
>>      on the command line
>>      -Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=256.
>>      Setting a value of 1 associatesa single file to a single shared arena.
>>      (Chris Hegarty, Michael Gibney, Uwe Schindler)
>>
>> Actually it looks like there are many deletes on the same index segment
>> so the segment itsself is not closed but the deltes are updated over an
>> over. As the whole segment uses the same shared memory arena and it
>> won't delete all 1024 (the default value) mappings and this would count
>> against the maxMapCount limit.
>>
>> To work around the issue you can choose to reduce the setting as
>> described above by passing it as a separate system property on
>> Opensearch's command line. I'd recomment to use a smaller value like 64
>> for systems with many indexes.
>>
>> Please tell us what you found out! Did reducing the
>> sharedArenaMaxPermits limit help? Maybe a good idea would be to change
>> Lucene / Opensearch to open deletion files in a separate arena or use
>> READONCE to load them to memory.
>>
>> Uwe
>>
>> Am 07.05.2025 um 03:44 schrieb Justin Borromeo:
>>> Hi all,
>>>
>>> After upgrading our OpenSearch cluster from 2.16.0 to 2.19.1 (moving from
>>> Lucene 9.10 to Lucene 9.12), our largest clusters started crashing with
>> the
>>> following error:
>>>
>>> # There is insufficient memory for the Java Runtime Environment to
>> continue.
>>> # Native memory allocation (malloc) failed to allocate 2097152 bytes.
>> Error
>>> detail: AllocateHeap
>>>
>>> We narrowed down the issue to the vm max map count (262144) being
>> reached.
>>> Prior to server crash, we see map count (measured by `cat
>> /proc/{pid}/maps
>>> | wc -l`) approach the 262144 limit we set.  Looking at one of the
>> outputs
>>> of `cat /proc/{pid}/maps`, we observed that 246K of the 252K maps are for
>>> deleted doc values (.dvd) files.
>>>
>>> Is this expected?  If so, were there any changes in the Lucene codebase
>>> between those two versions that could have caused this?  Any suggestions
>> on
>>> debugging?
>>>
>>> Thanks in advance and sorry if this is a better question for the OS
>>> community or the Lucene developer list.
>>>
>>> Justin Borromeo
>>>
>> --
>> Uwe Schindler
>> Achterdiek 19, D-28357 Bremen
>> https://www.thetaphi.de
>> eMail:[email protected]
>>
-- 
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: [email protected]