Re: Server goes OOM when

--===============6724736956731072164==
Content-Type: multipart/alternative; boundary=001a11c1e6aa6ea51d05033ecc42

--001a11c1e6aa6ea51d05033ecc42
Content-Type: text/plain; charset=UTF-8

Hi All

Mistakenly email was sent without correct subject

Subject should be "Axis2 server goes OOM when ServiceTCCL parameter is used"

Thanks
Geeth

On Wed, Sep 17, 2014 at 1:07 PM, Geeth Munasinghe <[email protected]>
wrote:

> Hi all,
>
> We are using axis2-1.6.1 version, it uses axiom 1.2.11 version, We
> recently encountered a OOM problem with two services which uses ServiceTCCL
> parameter. We are using this parameter because both those services use
> spring and hibernate with them. Scenario is one service is calling the
> other service. So one service acts as the client.
>
> We analyzed the heap dumps and found out that OOM issue was caused by
> org.apache.axiom.om.util.StAXUtils class. There are few weakhashmaps used
> in that class. Two of them are
>
>    1. inputFactoryPerCLMap
>    2. outputFactoryPerCLMap
>
> And those two weak hashmaps has other weak hashmaps inside of them. Those
> inner hashmaps cause the issue.
>
> We found out that when we use ServiceTCCL parameter  in the service.xml,
> in AbstractMessageReceiver.java  [1] of axis2 (line number 152 - 170)
> creates a new class loader object of the MultiParentClassLoader [2] for
> every request. So in StAXUtils class [3] in axiom, methods [4] and [5] uses
> inner  weak hashmaps of inputFactoryPerCLMap,  outputFactoryPerCLMap and
> fill them with the class loader as the key and XMLInputFactory /
> XMLOutputFactory as the value.
>
> Because every request gets a new class loader object of the
> MultiParentClassLoader, their hash values are different. So they are keep
> getting filled into those inner weak hash maps. Because the same key
> (classloder instance) gets inserted into both weak hashmaps, garbage
> collector does not remove them. So server goes OOM, When we analyze the
> heap dumps we found out that java.util.WeakHashMap fills over 80% of the
> memory when it goes OOM.
>
> I have made fix in StAXUtils in axiom as follows.
>
> Instead map.get(cl) I change it to map.get(cl.getClass().getName())
>
> Instead map.put(cl, factory) I changed it to
> map.put(cl.getClass().getName(), factory)
>
> I have attached the fix (svn diff) here with email. I am not sure I have
> done the correct fix for the issue. But I found that it solves my problem.
> Can some one please verify weather I have done the correct fix.
>
> Please consider that upgrading to new axis2 is not a solution for us at
> the moment.
>
> [1]
> http://svn.apache.org/repos/asf/axis/axis2/java/core/tags/v1.6.1/modules/kernel/src/org/apache/axis2/receivers/AbstractMessageReceiver.java
>
> [2] org.apache.axis2.classloader.MultiParentClassLoader
>
> [3] org.apache.axiom.om.util.StAXUtils
>
> [4] getXMLInputFactory_perClassLoader(StAXParserConfiguration
> configuration)
>
> [5] getXMLOutputFactory_perClassLoader(StAXWriterConfiguration
> configuration)
>
>
> Thanks in advance.
>
> Geeth
>

--001a11c1e6aa6ea51d05033ecc42
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div>Hi All<br><br></div>Mistakenly email w=
as sent without correct subject <br><br></div>Subject should be &quot;Axis2=
 server goes OOM when ServiceTCCL parameter is used&quot;<br><br></div>Than=
ks<br></div>Geeth<br></div><div class=3D"gmail_extra"><br><div class=3D"gma=
il_quote">On Wed, Sep 17, 2014 at 1:07 PM, Geeth Munasinghe <span dir=3D"lt=
r">&lt;<a href=3D"mailto:[email protected]" target=3D"_blank">geethmks@gma=
il.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"=
margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"=
ltr"><div><div>Hi all,<br><br></div>We are using axis2-1.6.1 version, it us=
es axiom 1.2.11 version, We recently encountered a OOM problem with two ser=
vices which uses ServiceTCCL parameter. We are using this parameter because=
 both those services use spring and hibernate with them. Scenario is one se=
rvice is calling the other service. So one service acts as the client. <br>=
<br></div>We analyzed the heap dumps and found out that OOM issue was cause=
d by org.apache.axiom.om.util.StAXUtils class. There are few weakhashmaps u=
sed in that class. Two of them are <br><ol><li>inputFactoryPerCLMap</li><li=
>outputFactoryPerCLMap</li></ol><p>And those two weak hashmaps has other we=
ak hashmaps inside of them. Those inner hashmaps cause the issue. <br></p><=
p>We found out that when we use ServiceTCCL parameter=C2=A0 in the service.=
xml, in AbstractMessageReceiver.java=C2=A0 [1] of axis2 (line number 152 - =
170) creates a new class loader object of the MultiParentClassLoader [2] fo=
r every request. So in StAXUtils class [3] in axiom, methods [4] and [5] us=
es inner=C2=A0 weak hashmaps of inputFactoryPerCLMap,=C2=A0 outputFactoryPe=
rCLMap and fill them with the class loader as the key and XMLInputFactory /=
 XMLOutputFactory as the value.=C2=A0</p><p>Because every request gets a ne=
w class loader object of the MultiParentClassLoader, their hash values are =
different. So they are keep getting filled into those inner weak hash maps.=
 Because the same key (classloder instance) gets inserted into both weak ha=
shmaps, garbage collector does not remove them. So server goes OOM, When we=
 analyze the heap dumps we found out that java.util.WeakHashMap fills over =
80% of the memory when it goes OOM.</p><p>I have made fix in StAXUtils in a=
xiom as follows.</p><p>Instead map.get(cl) I change it to map.get(cl.getCla=
ss().getName())</p><p>Instead map.put(cl, factory) I changed it to map.put(=
cl.getClass().getName(), factory)</p><p>I have attached the fix (svn diff) =
here with email. I am not sure I have done the correct fix for the issue. B=
ut I found that it solves my problem. Can some one please verify weather I =
have done the correct fix. <br></p><p>Please consider that upgrading to new=
 axis2 is not a solution for us at the moment.<br></p><p>[1] <a href=3D"htt=
p://svn.apache.org/repos/asf/axis/axis2/java/core/tags/v1.6.1/modules/kerne=
l/src/org/apache/axis2/receivers/AbstractMessageReceiver.java" target=3D"_b=
lank">http://svn.apache.org/repos/asf/axis/axis2/java/core/tags/v1.6.1/modu=
les/kernel/src/org/apache/axis2/receivers/AbstractMessageReceiver.java</a><=
/p><p>[2] org.apache.axis2.classloader.MultiParentClassLoader</p><p>[3] org=
.apache.axiom.om.util.StAXUtils</p><p>[4] getXMLInputFactory_perClassLoader=
(StAXParserConfiguration configuration)</p><p>[5] getXMLOutputFactory_perCl=
assLoader(StAXWriterConfiguration configuration)</p><p><br></p><p>Thanks in=
 advance.</p><span class=3D"HOEnZb"><font color=3D"#888888"><p>Geeth<br></p=
></font></span></div>
</blockquote></div><br></div>

--001a11c1e6aa6ea51d05033ecc42--


--===============6724736956731072164==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Axiom-mail mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/axiom-mail

--===============6724736956731072164==--