ThreadLocal performance degradation

Hi all,

This has been discussed before in this thread: https://cs.oswego.edu/pipermail/concurrency-interest/2018-October/016628.html
...and I think Gil Tene's comment about collisions exacerbating the issue is especially helpful: https://cs.oswego.edu/pipermail/concurrency-interest/2018-October/016685.html

What happens is that the ThreadLocalMap can degenerate to a linear scan when there are many collisions, and performance drops significantly on affected Threads in all code using any ThreadLocal instance (even code unrelated to the 'problem').

Is there interest in fixing this? Within the same discussion, Peter Levart had asked if there was a reproducer. I've managed to create one that works quickly on at least my two test systems. Does that help?
Reproducer and sample output: https://gist.github.com/megfigura/67d6972aa19fa1a4a1e13b4c7e7380fb

Watching as it runs, what's most concerning is how it suddenly blows up. Monitoring for the longest run of non-nulls in the ThreadLocalMap, the number will be small, small, small, but then it suddenly explodes - 6...23...48...29863... ...257207. Runs of non-nulls mean that long linear-scans will happen with some of the ThreadLocal API calls.

I ran into this issue indirectly by creating very many instances of the otherwise wonderful ChronicleMap which has a per-instance ThreadLocal. I captured a heapdump of our process in the bad state, and it showed a continuous run of non-null entries in the ThreadLocalMap of about 8000 items and another of about 4000. The backing array of the map is 64k, so there is a good chance a new instance will end up within one of these large blocks, doing a linear-scan. I could see the same behavior on different worker threads in the same process. The problem will reproduce again in new instances of the process within a few days in a specific customer environment. Once in the bad state, we saw an unrelated area which was using ReentrantReadWriteLock (which also uses ThreadLocal) show up as the hottest
  methods in the profiler.

In my case, the solution is to reduce the rate we are creating ChronicleMap/ThreadLocal instances, but a workaround is to trigger a GC via JMX periodically or to get the internal ThreadLocal via reflection and schedule it to be .remove()'d on all threads that accessed it.

Thanks a lot!
Meg