Server goes OOM when

--047d7b16350b411b2105033df19a
Content-Type: multipart/alternative; boundary=047d7b16350b411b1c05033df198

--047d7b16350b411b1c05033df198
Content-Type: text/plain; charset=UTF-8

Hi all,

We are using axis2-1.6.1 version, it uses axiom 1.2.11 version, We recently
encountered a OOM problem with two services which uses ServiceTCCL
parameter. We are using this parameter because both those services use
spring and hibernate with them. Scenario is one service is calling the
other service. So one service acts as the client.

We analyzed the heap dumps and found out that OOM issue was caused by
org.apache.axiom.om.util.StAXUtils class. There are few weakhashmaps used
in that class. Two of them are

   1. inputFactoryPerCLMap
   2. outputFactoryPerCLMap

And those two weak hashmaps has other weak hashmaps inside of them. Those
inner hashmaps cause the issue.

We found out that when we use ServiceTCCL parameter  in the service.xml, in
AbstractMessageReceiver.java  [1] of axis2 (line number 152 - 170) creates
a new class loader object of the MultiParentClassLoader [2] for every
request. So in StAXUtils class [3] in axiom, methods [4] and [5] uses
inner  weak hashmaps of inputFactoryPerCLMap,  outputFactoryPerCLMap and
fill them with the class loader as the key and XMLInputFactory /
XMLOutputFactory as the value.

Because every request gets a new class loader object of the
MultiParentClassLoader, their hash values are different. So they are keep
getting filled into those inner weak hash maps. Because the same key
(classloder instance) gets inserted into both weak hashmaps, garbage
collector does not remove them. So server goes OOM, When we analyze the
heap dumps we found out that java.util.WeakHashMap fills over 80% of the
memory when it goes OOM.

I have made fix in StAXUtils in axiom as follows.

Instead map.get(cl) I change it to map.get(cl.getClass().getName())

Instead map.put(cl, factory) I changed it to
map.put(cl.getClass().getName(), factory)

I have attached the fix (svn diff) here with email. I am not sure I have
done the correct fix for the issue. But I found that it solves my problem.
Can some one please verify weather I have done the correct fix.

Please consider that upgrading to new axis2 is not a solution for us at the
moment.

[1]
http://svn.apache.org/repos/asf/axis/axis2/java/core/tags/v1.6.1/modules/kernel/src/org/apache/axis2/receivers/AbstractMessageReceiver.java

[2] org.apache.axis2.classloader.MultiParentClassLoader

[3] org.apache.axiom.om.util.StAXUtils

[4] getXMLInputFactory_perClassLoader(StAXParserConfiguration configuration)

[5] getXMLOutputFactory_perClassLoader(StAXWriterConfiguration
configuration)


Thanks in advance.

Geeth

--047d7b16350b411b1c05033df198
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Hi all,<br><br></div>We are using axis2-1.6.1 ve=
rsion, it uses axiom 1.2.11 version, We recently encountered a OOM problem =
with two services which uses ServiceTCCL parameter. We are using this param=
eter because both those services use spring and hibernate with them. Scenar=
io is one service is calling the other service. So one service acts as the =
client. <br><br></div>We analyzed the heap dumps and found out that OOM iss=
ue was caused by org.apache.axiom.om.util.StAXUtils class. There are few we=
akhashmaps used in that class. Two of them are <br><ol><li>inputFactoryPerC=
LMap</li><li>outputFactoryPerCLMap</li></ol><p>And those two weak hashmaps =
has other weak hashmaps inside of them. Those inner hashmaps cause the issu=
e. <br></p><p>We found out that when we use ServiceTCCL parameter=C2=A0 in =
the service.xml, in AbstractMessageReceiver.java=C2=A0 [1] of axis2 (line n=
umber 152 - 170) creates a new class loader object of the MultiParentClassL=
oader [2] for every request. So in StAXUtils class [3] in axiom, methods [4=
] and [5] uses inner=C2=A0 weak hashmaps of inputFactoryPerCLMap,=C2=A0 out=
putFactoryPerCLMap and fill them with the class loader as the key and XMLIn=
putFactory / XMLOutputFactory as the value.=C2=A0</p><p>Because every reque=
st gets a new class loader object of the MultiParentClassLoader, their hash=
 values are different. So they are keep getting filled into those inner wea=
k hash maps. Because the same key (classloder instance) gets inserted into =
both weak hashmaps, garbage collector does not remove them. So server goes =
OOM, When we analyze the heap dumps we found out that java.util.WeakHashMap=
 fills over 80% of the memory when it goes OOM.</p><p>I have made fix in St=
AXUtils in axiom as follows.</p><p>Instead map.get(cl) I change it to map.g=
et(cl.getClass().getName())</p><p>Instead map.put(cl, factory) I changed it=
 to map.put(cl.getClass().getName(), factory)</p><p>I have attached the fix=
 (svn diff) here with email. I am not sure I have done the correct fix for =
the issue. But I found that it solves my problem. Can some one please verif=
y weather I have done the correct fix. <br></p><p>Please consider that upgr=
ading to new axis2 is not a solution for us at the moment.<br></p><p>[1] <a=
 href=3D"http://svn.apache.org/repos/asf/axis/axis2/java/core/tags/v1.6.1/m=
odules/kernel/src/org/apache/axis2/receivers/AbstractMessageReceiver.java">=
http://svn.apache.org/repos/asf/axis/axis2/java/core/tags/v1.6.1/modules/ke=
rnel/src/org/apache/axis2/receivers/AbstractMessageReceiver.java</a></p><p>=
[2] org.apache.axis2.classloader.MultiParentClassLoader</p><p>[3] org.apach=
e.axiom.om.util.StAXUtils</p><p>[4] getXMLInputFactory_perClassLoader(StAXP=
arserConfiguration configuration)</p><p>[5] getXMLOutputFactory_perClassLoa=
der(StAXWriterConfiguration configuration)</p><p><br></p><p>Thanks in advan=
ce.</p><p>Geeth<br></p></div>

--047d7b16350b411b1c05033df198--
--047d7b16350b411b2105033df19a
Content-Type: text/plain; charset=US-ASCII; name="axiom.diff"
Content-Disposition: attachment; filename="axiom.diff"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_i06cesna0

SW5kZXg6IG1vZHVsZXMvYXhpb20tYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9heGlvbS9v
bS91dGlsL1N0QVhVdGlscy5qYXZhCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIG1vZHVsZXMvYXhpb20tYXBpL3Ny
Yy9tYWluL2phdmEvb3JnL2FwYWNoZS9heGlvbS9vbS91dGlsL1N0QVhVdGlscy5qYXZhCShyZXZp
c2lvbiAxNjI1NDczKQorKysgbW9kdWxlcy9heGlvbS1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBh
Y2hlL2F4aW9tL29tL3V0aWwvU3RBWFV0aWxzLmphdmEJKHdvcmtpbmcgY29weSkKQEAgLTU1MCw3
ICs1NTAsNyBAQAogICAgICAgICAgICAgICAgIGlucHV0RmFjdG9yeVBlckNMTWFwLnB1dChjb25m
aWd1cmF0aW9uLCBtYXApOwogICAgICAgICAgICAgICAgIGZhY3RvcnkgPSBudWxsOwogICAgICAg
ICAgICAgfSBlbHNlIHsKLSAgICAgICAgICAgICAgICBmYWN0b3J5ID0gKFhNTElucHV0RmFjdG9y
eSltYXAuZ2V0KGNsKTsKKyAgICAgICAgICAgICAgICBmYWN0b3J5ID0gKFhNTElucHV0RmFjdG9y
eSltYXAuZ2V0KGNsLmdldENsYXNzKCkuZ2V0TmFtZSgpKTsKICAgICAgICAgICAgIH0KICAgICAg
ICAgICAgIAogICAgICAgICAgICAgLy8gSWYgbm90IGZvdW5kIGluIHRoZSBjYWNoZSBtYXAsIGNy
YXRlIGEgbmV3IGZhY3RvcnkKQEAgLTU3OCw3ICs1NzgsNyBAQAogICAgICAgICAgICAgICAgICAg
ICAKICAgICAgICAgICAgICAgICBpZiAoZmFjdG9yeSAhPSBudWxsKSB7CiAgICAgICAgICAgICAg
ICAgICAgIC8vIENhY2hlIHRoZSBuZXcgZmFjdG9yeQotICAgICAgICAgICAgICAgICAgICBtYXAu
cHV0KGNsLCBmYWN0b3J5KTsKKyAgICAgICAgICAgICAgICAgICAgbWFwLnB1dChjbC5nZXRDbGFz
cygpLmdldE5hbWUoKSwgZmFjdG9yeSk7CiAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAg
ICAgICAgICAgICBpZiAobG9nLmlzRGVidWdFbmFibGVkKCkpIHsKICAgICAgICAgICAgICAgICAg
ICAgICAgIGxvZy5kZWJ1ZygiQ3JlYXRlZCBYTUxJbnB1dEZhY3RvcnkgPSAiICsgZmFjdG9yeS5n
ZXRDbGFzcygpICsgCkBAIC02NzMsNyArNjczLDcgQEAKICAgICAgICAgICAgICAgICBvdXRwdXRG
YWN0b3J5UGVyQ0xNYXAucHV0KGNvbmZpZ3VyYXRpb24sIG1hcCk7CiAgICAgICAgICAgICAgICAg
ZmFjdG9yeSA9IG51bGw7CiAgICAgICAgICAgICB9IGVsc2UgewotICAgICAgICAgICAgICAgIGZh
Y3RvcnkgPSAoWE1MT3V0cHV0RmFjdG9yeSltYXAuZ2V0KGNsKTsKKyAgICAgICAgICAgICAgICBm
YWN0b3J5ID0gKFhNTE91dHB1dEZhY3RvcnkpbWFwLmdldChjbC5nZXRDbGFzcygpLmdldE5hbWUo
KSk7CiAgICAgICAgICAgICB9CiAgICAgICAgICAgICAKICAgICAgICAgICAgIGlmIChmYWN0b3J5
ID09IG51bGwpIHsKQEAgLTY5Nyw3ICs2OTcsNyBAQAogICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIGNvbmZpZ3VyYXRpb24pOwogICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgICAgICBp
ZiAoZmFjdG9yeSAhPSBudWxsKSB7Ci0gICAgICAgICAgICAgICAgICAgIG1hcC5wdXQoY2wsIGZh
Y3RvcnkpOworICAgICAgICAgICAgICAgICAgICBtYXAucHV0KGNsLmdldENsYXNzKCkuZ2V0TmFt
ZSgpLCBmYWN0b3J5KTsKICAgICAgICAgICAgICAgICAgICAgaWYgKGxvZy5pc0RlYnVnRW5hYmxl
ZCgpKSB7CiAgICAgICAgICAgICAgICAgICAgICAgICBsb2cuZGVidWcoIkNyZWF0ZWQgWE1MT3V0
cHV0RmFjdG9yeSA9ICIgKyBmYWN0b3J5LmdldENsYXNzKCkgCiAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgKyAiIGZvciBjbGFzc2xvYWRlcj0iICsgY2wpOwo=
--047d7b16350b411b2105033df19a
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Axiom-mail mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/axiom-mail

--047d7b16350b411b2105033df19a--