Re: Memory Leak Issue with Symlinked WAR Files and autoDeploy

> On 2025 Nov 24, at 22:14, Harshit Sharma <[email protected]> wrote:
> We're experiencing a significant memory leak issue with our production Tomcat setup and would appreciate guidance on best practices and potential fixes.
> 
> Environment Details
> Tomcat Version: Apache Tomcat 7.0.41


Support for Tomcat 7 ended in March of 2021 - over 4.5 years ago, and your particular version was released over 12 years ago. Your efforts would be much better spent in upgrading to a supported version, which would also fix numerous bugs and security weaknesses. Trying to continue with Tomcat 7 is irresponsible.

  - Chuck


> Heap Size: 6GB (max)
> Number of WAR files: 7
> Application Type: Java & Spring Boot
> Deployment Mode: hot-updates using symlinked WAR files
> OS: Linux
> JVM: OpenJDK 8 (1.8.0_412)
> 
> Current Deployment Approach
> We're using a symlink-based deployment strategy:
> 
> Actual Tomcat webapps path: /path/to/tomcat/webapps/
> Git repository containing WAR files: /path/to/repo/warFiles/
> Symlinks created from Git repo to Tomcat webapps directory
> Deployment process: git pull to update WAR files, no Tomcat restart
> server.xml configuration:
> 
> <Host name="localhost" appBase="webapps"
>       unpackWARs="true" autoDeploy="true">
> Problem Description
> Over the past 5 months (since the last Tomcat restart), we've observed a progressive memory leak with the following pattern:
> 
> Class Count Growth (Staircase Pattern):
> 
> Initial: 50k-56k classes
> After each git pull/deployment: Incremental increase
> Peak: 250k classes
> After restart: Back to 50k-55k classes
> 
> Performance Degradation:
> 
> Metric
> Before Restart
> After Restart
> Class Count
> 250k
> 55k
> Max GC Count
> 56
> 46
> Avg GC Count
> 10
> 6
> Max GC Time
> 2000ms
> 963ms
> Avg GC Time
> 400ms
> 215ms
> Avg JVM Threads
> 1.44k
> 277
> Monitoring
> We've been monitoring this issue through Zabbix, and the graphs clearly show the progressive degradation over time (includes incident & after-incident pattern as well):
> 
> Class count growth over the months (showing staircase pattern from 50k to 250k)
> 
> 
> GC activity trends (count and time)
> 
> 
> Memory/Heap usage over time
> 
> 
> Thread count growth
> 
> 
> Incident: After a recent git pull, Tomcat experienced an OutOfMemoryError, generated heap dump files, and crashed. The system was unresponsive until we performed a restart.
> 
> Questions for the Community
> Is our symlink-based deployment approach with autoDeploy="true" causing classloader leaks??
> 
> What is the recommended deployment strategy for production environments to avoid memory leaks when updating WAR files??
> 
> Are there known issues with Tomcat 7.0.41 related to classloader leaks during redeployment??
> 
> What monitoring or preventive measures can we implement to detect and mitigate this issue before it causes production outages??
> 
> We suspect the combination of symlinked WAR files, autoDeploy="true", and no-restart deployments is preventing proper cleanup of old classloaders, but we'd like to understand the root cause and implement a robust solution.
> 
> Any insights, best practices, or references to similar issues would be greatly appreciated.
> 
> Thank you for your time and assistance.
> 
> 
> Thanks & Regards
> Harshit Sharma