Re: Deadlock in Cache

Hi Brett,

At this point the code hasn't yet gone through formal testing. The 
changes are minimal though (reduction of synchronized section), and a 
backport of some of the changes already in 5.0

Regards,
  Serge Huber.

Brett Randall wrote:
> Hi Serge,
>
> Thanks for that.  I note these revisions on the following branches for
> this change:
>
> 4.0 - 14113
> 4.1 - 14117
> TRUNK - 14118
>
> Has this patch been tested against a system setup including JMS cache
> replication?
>
> Regards
> Brett Randall
>
>
> -----Original Message-----
> From: Serge Huber [mailto:[email protected]] 
> Sent: Thursday, 4 May 2006 19:07
> To: [email protected]
> Subject: Re: Deadlock in Cache
>
>
> Hi Brett,
>
> I have committed modifications to the cache system so that we don't lock
>
> everything on a cache flush. This will be in the next release of the 
> 4.0.x and 4.1.x versions.
>
> Regards,
> Serge Huber.
>
> Brett Randall wrote:
>   
>> We have a production site suffering the same intermittent deadlock 
>> issue identified in http://www.jahia.net/jira/browse/JAHIA-639 . This 
>> is a Jahia 4.0.5 system. The result is a completely deadlocked Jahia 
>> engine, no rendering, no authoring.
>>
>> I'm enquiring whether this bug is likely to receive any attention in 
>> the near future, given the severity.
>>
>> Stack traces below - I will also update the Jira.
>>
>> Found one Java-level deadlock:
>>
>> =============================
>>
>> "TP-Processor70593":
>>
>> waiting to lock monitor 0x080cc0bc (object 0x92ffe878, a 
>> org.jahia.services.cache.Cache),
>>
>> which is held by "TP-Processor70118"
>>
>> "TP-Processor70118":
>>
>> waiting to lock monitor 0x080cc0f4 (object 0x92ffe8a8, a 
>> org.jahia.services.cache.Cache),
>>
>> which is held by "TP-Processor70593"
>>
>> Java stack information for the threads listed above:
>>
>> ===================================================
>>
>> "TP-Processor70593":
>>
>> at org.jahia.services.cache.Cache.putCacheEntry(Cache.java:280)
>>
>> - waiting to lock <0x92ffe878> (a org.jahia.services.cache.Cache)
>>
>> at org.jahia.services.cache.Cache.put(Cache.java:259)
>>
>> at org.jahia.services.cache.Cache.put(Cache.java:235)
>>
>> at 
>>
>>     
> org.jahia.registries.JahiaContainerDefinitionsRegistry.loadAllDefinition
> s(JahiaContainerDefinitionsRegistry.java:106)
>   
>> - locked <0x92ffc790> (a 
>> org.jahia.registries.JahiaContainerDefinitionsRegistry)
>>
>> at 
>>
>>     
> org.jahia.registries.JahiaContainerDefinitionsRegistry.onCacheFlush(Jahi
> aContainerDefinitionsRegistry.java:301)
>   
>> at org.jahia.services.cache.Cache.flush(Cache.java:442)
>>
>> - locked <0x92ffe8a8> (a org.jahia.services.cache.Cache)
>>
>> at org.jahia.services.cache.Cache.flush(Cache.java:412)
>>
>> at 
>>
>>     
> org.jahia.services.cache.CacheFactory.flushAllCaches(CacheFactory.java:3
> 56)
>   
>> - locked <0x92ffe8a8> (a org.jahia.services.cache.Cache)
>>
>> - locked <0x92c46d70> (a org.jahia.services.cache.CacheFactory) at 
>>
>>     
> org.jahia.admin.status.ManageStatus.processSettings(ManageStatus.java:19
> 0)
>   
>> at 
>>
>>     
> org.jahia.admin.status.ManageStatus.userRequestDispatcher(ManageStatus.j
> ava:111)
>   
>> at org.jahia.admin.status.ManageStatus.<init>(ManageStatus.java:91)
>>
>> at 
>>
>>     
> org.jahia.bin.JahiaAdministration.userRequestDispatcher(JahiaAdministrat
> ion.java:374)
>   
>> at
>>     
> org.jahia.bin.JahiaAdministration.service(JahiaAdministration.java:245)
>   
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:809)
>>
>> at 
>>
>>     
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
> tionFilterChain.java:200)
>   
>> <snip>
>>
>> "TP-Processor70118":
>>
>> at org.jahia.services.cache.Cache.putCacheEntry(Cache.java:280)
>>
>> - waiting to lock <0x92ffe8a8> (a org.jahia.services.cache.Cache)
>>
>> at org.jahia.services.cache.Cache.put(Cache.java:259)
>>
>> at org.jahia.services.cache.Cache.put(Cache.java:235)
>>
>> at 
>>
>>     
> org.jahia.registries.JahiaContainerDefinitionsRegistry.loadDefinitionByI
> D(JahiaContainerDefinitionsRegistry.java:122)
>   
>> at 
>>
>>     
> org.jahia.registries.JahiaContainerDefinitionsRegistry.getDefinition(Jah
> iaContainerDefinitionsRegistry.java:162)
>   
>> - locked <0x92ffe878> (a org.jahia.services.cache.Cache)
>>
>> at 
>>
>>     
> org.jahia.data.containers.JahiaContainerStructure.<init>(JahiaContainerS
> tructure.java:75)
>   
>> at 
>>
>>     
> org.jahia.services.containers.JahiaContainerStructuresDB.db_load_contain
> er_structure(JahiaContainerStructuresDB.java:88)
>   
>> at 
>>
>>     
> org.jahia.services.containers.JahiaContainerDefinitionsDB.db_load_contai
> ner_definition(JahiaContainerDefinitionsDB.java:124)
>   
>> at 
>>
>>     
> org.jahia.services.containers.JahiaContainersBaseService.loadContainerDe
> finition(JahiaContainersBaseService.java:2053)
>   
>> at 
>>
>>     
> org.jahia.registries.JahiaContainerDefinitionsRegistry.loadDefinitionByI
> D(JahiaContainerDefinitionsRegistry.java:115)
>   
>> at 
>>
>>     
> org.jahia.registries.JahiaContainerDefinitionsRegistry.getDefinition(Jah
> iaContainerDefinitionsRegistry.java:162)
>   
>> - locked <0x92ffe878> (a org.jahia.services.cache.Cache)
>>
>> at 
>>
>>     
> org.jahia.data.containers.JahiaContainerStructure.<init>(JahiaContainerS
> tructure.java:75)
>   
>> at 
>>
>>     
> org.jahia.services.containers.JahiaContainerStructuresDB.db_load_contain
> er_structure(JahiaContainerStructuresDB.java:88)
>   
>> at 
>>
>>     
> org.jahia.services.containers.JahiaContainerDefinitionsDB.db_load_contai
> ner_definition(JahiaContainerDefinitionsDB.java:124)
>   
>> at 
>>
>>     
> org.jahia.services.containers.JahiaContainersBaseService.loadContainerDe
> finition(JahiaContainersBaseService.java:2053)
>   
>> at 
>>
>>     
> org.jahia.registries.JahiaContainerDefinitionsRegistry.loadDefinitionByI
> D(JahiaContainerDefinitionsRegistry.java:115)
>   
>> at 
>>
>>     
> org.jahia.registries.JahiaContainerDefinitionsRegistry.getDefinition(Jah
> iaContainerDefinitionsRegistry.java:162)
>   
>> - locked <0x92ffe878> (a org.jahia.services.cache.Cache)
>>
>> at 
>>
>>     
> org.jahia.engines.containerlistproperties.ContainerListProperties_Engine
> .renderLink(ContainerListProperties_Engine.java:113)
>   
>> at org.jahia.gui.GuiBean.drawUrlCheckWriteAccess(GuiBean.java:720)
>>
>> at
>>     
> org.jahia.gui.GuiBean.drawContainerListPropertiesUrl(GuiBean.java:488)
>   
>> at 
>>
>>     
> org.jahia.gui.HTMLToolBox.drawContainerListPropertiesLauncher(HTMLToolBo
> x.java:536)
>   
>> at 
>>
>>     
> org.jahia.gui.HTMLToolBox.drawContainerListPropertiesLauncher(HTMLToolBo
> x.java:515)
>   
>> - locked <0xa1f16b80> (a org.jahia.gui.HTMLToolBox)
>>
>> at 
>>
>>     
> org.jahia.taglibs.url.ContainerListPropertiesURLTag.getJSPopupURL(Contai
> nerListPropertiesURLTag.java:69)
>   
>> at 
>>
>>     
> org.jahia.taglibs.url.ContainerListPropertiesURLTag.getIdObject(Containe
> rListPropertiesURLTag.java:43)
>   
>> at
>>     
> org.jahia.taglibs.url.AbstractURLTag.doStartTag(AbstractURLTag.java:91)
>   
>> at org.apache.jsp.columns_jsp._jspService(columns_jsp.java:3858)
>>
>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:92)
>>
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:809)
>>
>> at 
>>
>>     
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.ja
> va:162)
>   
>> at 
>>
>>     
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:240)
>   
>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:187)
>>
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:809)
>>
>> at 
>>
>>     
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatc
> her.java:627)
>   
>> at 
>>
>>     
> org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDisp
> atcher.java:518)
>   
>> at 
>>
>>     
> org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispat
> cher.java:448)
>   
>> at 
>>
>>     
> org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.ja
> va:768)
>   
>> at 
>>
>>     
> org.apache.jsp.general_columns_jsp._jspService(general_columns_jsp.java:
> 1895)
>   
>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:92)
>>
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:809)
>>
>> at 
>>
>>     
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.ja
> va:162)
>   
>> at 
>>
>>     
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:240)
>   
>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:187)
>>
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:809)
>>
>> at 
>>
>>     
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatc
> her.java:627)
>   
>> at 
>>
>>     
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDisp
> atcher.java:382)
>   
>> at 
>>
>>     
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispat
> cher.java:306)
>   
>> at
>>     
> org.jahia.engines.EngineRenderer.renderCore(EngineRenderer.java:182)
>   
>> at org.jahia.engines.EngineRenderer.render(EngineRenderer.java:78)
>>
>> at
>>     
> org.jahia.engines.core.Core_Engine.processCore(Core_Engine.java:186)
>   
>> at
>>     
> org.jahia.engines.core.Core_Engine.handleActions(Core_Engine.java:107)
>   
>> at 
>>
>>     
> org.jahia.operations.OperationManager.handleOperations(OperationManager.
> java:279)
>   
>> at org.jahia.bin.JahiaAction.execute(JahiaAction.java:50)
>>
>> at 
>>
>>     
> org.apache.struts.action.RequestProcessor.processActionPerform(RequestPr
> ocessor.java:484)
>   
>> at 
>>
>>     
> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:
> 274)
>   
>> at
>>     
> org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
>   
>> at org.jahia.bin.Jahia.process(Jahia.java:1522)
>>
>> at org.jahia.bin.Jahia.service(Jahia.java:846)
>>
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:809)
>>
>> at 
>>
>>     
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
> tionFilterChain.java:200)
>   
>> <snip>
>>
>> Found 1 deadlock.
>>
>> Thanks
>>
>> Brett Randall
>>
>>     
>
>
>