Re: spontaneous redeployment of a webapp

On 12/12/2025 01:11, Linus Kamb wrote:
> Mark and Chris,
> 
> Thank you again for your replies.   Responses inline below.
> 
> TLDR;  I have answers and comments below, but I feel this has taken up a
> fair amount of your time.
> If you wish to dig further into this, I'd certainly like to understand what
> (at least may have) happened.
> If not, I don't think this is very likely to have been a security incident,
> and I suppose I'll just chalk it up to
> being related to the disk being full.

That seems the most likely cause at this point.

<snip/>

> Yes.  All tomcats are running on one physical machine.
> The tomcat instances are all separate, complete installs with their own
> webapps directories, and each distinct webapp has its own data directory on
> a separate local (logical - /dev/mapper ) disk.

OK. That rules out quite a few possibilities.

>>> Are any files shared between the instances?
>>
> 
> There should not be.

As does that.

>>> How is the web application deployed? WAR file that Tomcat then expands?
>>
> 
> Yes.
> autoDeploy had been turned on.  We are turning that off.

Good decision.

>>> I want to make sure I understand the architecture before I spend too
>>> much time thinking about what might be going on.
>>
> 
> Understandably.  And I don't want to take too much of your time.

It isn't that I don't want to get to the bottom of this. More that there 
are lots possibilities and it is better to narrow down the options a bit 
before speculating.

>>>> Could the fact that there was no space left on the disk be treated as
>>>> being unavailable?
>>>
>>> Seems unlikely. Tomcat has a list of files it checks for each webapp and
>>> looks to see if the modification time has changed.
>>
> 
> Is there a list of those files?

I think you want this:

https://tomcat.apache.org/tomcat-11.0-doc/config/automatic-deployment.html

I think you have the WAR and the DIR present and both are in 
$CATALINA_BASE/webapps (i.e. not external) with no XMl file.

 From that, the things that could trigger redeployment are:
- deleting the DIR
- modified WAR file

Which really narrows down what we need to look at.

>>>> And why only the one webapp?  Perhaps because it was the one
> 
> 
> Full disclosure, in case it makes a difference:
> I missed one of the tomcats -- it did not have the one particular webapp.
> In that tomcat, *all* of its webapps were redeployed.
> However, there was a tomcat with the main webapp that had one additional
> webapp.  That additional webapp was *not* redeployed.
 >
>   FWIW.

Hmm.

Web applications in webapps are checked for modification one at a time. 
It does look like something (and the disk filling up is a strong 
candidate) either made Tomcat think the DIRs had been deleted or that 
the WARs had a different (including invalid) last modified time. And 
that triggered the redeploy.

Interesting that, despite the disk being full, the redeploy succeeded.

>>>> attempting to
>>>> write to the disk?
>>>
>>> Again, seems unlikely.
>>>
>>> Might some external process changed the last modification time on any
>>> files? A back-up process maybe? Those times should not be sensitive to
>>> clock / time zone changes but I'd check if there were any changes of
>>> that nature just in case.
>>
>>
> I am not aware of any external process that would touch files in the webapp
> directories.
> Looking at the webapp directories now,  nothing has changed since
> redeployment.
> 
> All the webapps were undeployed (as noted in the logs) and then redeployed,
> presumably unpacking the war. The webapp deployment directory was restored
> to its
> state as in the war.  The timestamps on all the files in the deployment
> directory are
> equal to their timestamps in the war, but all the directories had
> timestamps
> ~equal to the time of redeployment.

That is as expected for redeployment.

> Unfortunately, since the webapps were all undeployed, I cannot evaluate the
> contents and
> file timestamps prior to undeployment.

No problem. The timestamps of the files in the DIR aren't the 
interesting ones since a change doesn't trigger a redeploy.

The interesting ones are those on the WAR files. Did those change? Thta 
would trigger a redeploy.

> (I am assuming that when the logs say "Undeploying context,"  that means it
> removes the
> webapp directory.  Either way, everything was restored as if unpacked from
> the war.)

Yes. Delete the DIR. Unpack the WAR. Set the timestamps of the unpacked 
files to match those of the files in the WAR.

>> Is the web application configured to auto-reload? Tomcat will watch more
>> than just the timestamp on the application's deployment directory for
>> changes. There are other files that can cause a reload. (But generally
>> not a re-deployment.)
>>
>>
> I don't believe the application or server were configured to auto-reload.
> I find no mention of reloadable.

See the autodeployment page I referenced above for what changes can 
trigger a reload. Also look at $CATALINA_BASE/conf/context.xml for other 
resources that might be checked.

> The  logs (that were able to be written) stated "Undeploying <webapp>
> context."  (Or maybe context <webapp>)
> 
> And followed later by indicating the redeployent of the webapp.

Yep. That is what I'd expect to see for a deletion of the DIR followed 
by a re-expansion of the WAR.

Mark