Re: Out of memory errors at end of a run
John K <[email protected]> Wed, 2 Mar 2016 08:36:09 -0500
| Newsgroups | gmane.comp.java.grinder.user |
|---|---|
| Message-ID | <[email protected]> |
--===============1879539239103400084== Content-Type: multipart/alternative; boundary="_533fc763-7ba0-4817-9c00-e8e2d329e48d_" --_533fc763-7ba0-4817-9c00-e8e2d329e48d_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Gary=2C Thank you for the reply. I can do as you suggest and break up my Grinder wo= rkers but I'm only seeing this OutOfMemory error when my run is coming to a= n end based on grinder.duration. While it is running=2C everything is fine = so I think that what's happening is that when the Grinder is done and tries= to tally up some information=2C it runs out of memory. Since I am gathering metrics another way and not relying on any information= from Grinder (I've set all my logging to report only errors and have the d= ata log set to /dev/null to save disk space)=2C I was wondering if there is= a way to disable Grinder statistics=2C as from looking at the error stack= =2C it is crashing when trying to report some statistics that I don't use. From: [email protected] Date: Wed=2C 2 Mar 2016 13:21:36 +0000 To: [email protected] Subject: Re: [Grinder-use] Out of memory errors at end of a run John=2C I usually allocate about 1GB of RAM per 100 threads=2C but it is of course = very dependent on the complexity of your Jython scripts. Check out the following article on GC logging which might provide more info= rmation on when you're using your memory: http://grinder.sourceforge.net/g3/garbage-collection.html For performance and GC pause reasons I'd consider running multiple Grinder = instances with less threads=2C e.g. try 2 x 300 or 4 x 150. Regards=2CGary On 2 March 2016 at 12:56=2C John K <[email protected]> wrote: =0A= =0A= =0A= Hello all=2C I'm running Grinder 3.11 on RHEL 7.2 with the following JDK: java version "1.7.0" Java(TM) SE Runtime Environment (build pxa6470sr6fp1-20140108_01(SR6 FP1)) IBM J9 VM (build 2.6=2C JRE 1.7.0 Linux amd64-64 Compressed References 2014= 0106_181350 (JIT enabled=2C AOT enabled) J9VM - R26_Java726_SR6_20140106_1601_B181350 JIT - r11.b05_20131003_47443.02 GC - R26_Java726_SR6_20140106_1601_B181350_CMPRSS J9CL - 20140106_181350) JCL - 20140103_01 based on Oracle 7u51-b11 My Python script is just HTTP posts to a servlet and itself is very simple.= I'm running this for a 4 hour duration with 7 processes and 600 threads. I= n my grinder.properties=2C I have this set "grinder.jvm.arguments=3D-mx2048= M". When my run ends=2C I get lots of heap dumps and error stacks like the belo= w. I already have the memory for each worker thread set to 2GB=2C do I need= to set it even higher? I'm hoping not as I'm reaching the limit in memory = on my machine. I'm not using the Grinder statistics for anything so if I ca= n disable them to avoid this error=2C I'm find with that. Any ideas on how to proceed? 2016-03-01 18:28:35=2C512 ERROR worker-bootstrap: Error running worker proc= ess java.lang.OutOfMemoryError: Java heap space at java.lang.StringBuffer.ensureCapacityImpl(StringBuffer.java:338)= ~[na:1.7.0] at java.lang.StringBuffer.append(StringBuffer.java:204) ~[na:1.7.0] at java.io.StringWriter.write(StringWriter.java:123) ~[na:1.7.0] at java.io.PrintWriter.write(PrintWriter.java:467) ~[na:1.7.0] at java.io.PrintWriter.write(PrintWriter.java:484) ~[na:1.7.0] at java.io.PrintWriter.print(PrintWriter.java:614) ~[na:1.7.0] at net.grinder.statistics.StatisticsTable$1.next(StatisticsTable.ja= va:150) ~[grinder-core-3.11.jar:na] at net.grinder.statistics.TestStatisticsMap$ForEach.iterate(TestSta= tisticsMap.java:355) ~[grinder-core-3.11.jar:na] at net.grinder.statistics.StatisticsTable.print(StatisticsTable.jav= a:146) ~[grinder-core-3.11.jar:na] at net.grinder.engine.process.GrinderProcess.run(GrinderProcess.jav= a:554) ~[grinder-core-3.11.jar:na] at net.grinder.engine.process.WorkerProcessEntryPoint.run(WorkerPro= cessEntryPoint.java:86) [grinder-core-3.11.jar:na] at net.grinder.engine.process.WorkerProcessEntryPoint.main(WorkerPr= ocessEntryPoint.java:59) [grinder-core-3.11.jar:na] =0A= ---------------------------------------------------------------------------= --- =0A= Site24x7 APM Insight: Get Deep Visibility into Application Performance =0A= APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month =0A= Monitor end-to-end web transactions and take corrective actions now =0A= Troubleshoot faster and improve end-user experience. Signup Now! =0A= http://pubads.g.doubleclick.net/gampad/clk?id=3D272487151&iu=3D/4140 _______________________________________________ =0A= grinder-use mailing list =0A= [email protected] =0A= https://lists.sourceforge.net/lists/listinfo/grinder-use =0A= =0A= ---------------------------------------------------------------------------= ---=0A= Site24x7 APM Insight: Get Deep Visibility into Application Performance=0A= APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month=0A= Monitor end-to-end web transactions and take corrective actions now=0A= Troubleshoot faster and improve end-user experience. Signup Now!=0A= http://pubads.g.doubleclick.net/gampad/clk?id=3D272487151&iu=3D/4140 _______________________________________________=0A= grinder-use mailing list=0A= [email protected]=0A= https://lists.sourceforge.net/lists/listinfo/grinder-use = --_533fc763-7ba0-4817-9c00-e8e2d329e48d_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <style><!-- .hmmessage P { margin:0px=3B padding:0px } body.hmmessage { font-size: 12pt=3B font-family:Calibri } --></style></head> <body class=3D'hmmessage'><div dir=3D'ltr'>Hi Gary=2C<br><br>Thank you for = the reply. I can do as you suggest and break up my Grinder workers but I'm = only seeing this OutOfMemory error when my run is coming to an end based on= grinder.duration. While it is running=2C everything is fine so I think tha= t what's happening is that when the Grinder is done and tries to tally up s= ome information=2C it runs out of memory.<br><br>Since I am gathering metri= cs another way and not relying on any information from Grinder (I've set al= l my logging to report only errors and have the data log set to /dev/null t= o save disk space)=2C I was wondering if there is a way to disable Grinder = statistics=2C as from looking at the error stack=2C it is crashing when try= ing to report some statistics that I don't use.<br><br><div><hr id=3D"stopS= pelling">From: [email protected]<br>Date: Wed=2C 2 Mar 2016 13:21:36 = +0000<br>To: [email protected]<br>Subject: Re: [Grinder-use= ] Out of memory errors at end of a run<br><br><div dir=3D"ltr"><div>John=2C= </div><div><br></div><div>I usually allocate about 1GB of RAM per 100 threa= ds=2C but it is of course very dependent on the complexity of your Jython s= cripts.</div><div><br></div>Check out the following article on GC logging w= hich might provide more information on when you're using your memory:<div><= br></div><blockquote style=3D"border:none=3Bpadding:0px=3B"><div><a href=3D= "http://grinder.sourceforge.net/g3/garbage-collection.html" target=3D"_blan= k">http://grinder.sourceforge.net/g3/garbage-collection.html</a></div></blo= ckquote><div><br></div><div>For performance and GC pause reasons I'd consid= er running multiple Grinder instances with less threads=2C e.g. try 2 x 300= or 4 x 150.</div><div><br></div><div>Regards=2C</div><div>Gary</div></div>= <div class=3D"ecxgmail_extra"><br><div class=3D"ecxgmail_quote">On 2 March = 2016 at 12:56=2C John K <span dir=3D"ltr"><=3B<a href=3D"mailto:raven566@= hotmail.com" target=3D"_blank">[email protected]</a>>=3B</span> wrote:= <br><blockquote class=3D"ecxgmail_quote" style=3D"border-left:1px #ccc soli= d=3Bpadding-left:1ex=3B">=0A= =0A= =0A= <div><div dir=3D"ltr">Hello all=2C<br><br>I'm running Grinder 3.11 on RHEL = 7.2 with the following JDK:<br><br>java version "1.7.0"<br>Java(TM) SE Runt= ime Environment (build pxa6470sr6fp1-20140108_01(SR6 FP1))<br>IBM J9 VM (bu= ild 2.6=2C JRE 1.7.0 Linux amd64-64 Compressed References 20140106_181350 (= JIT enabled=2C AOT enabled)<br>J9VM - R26_Java726_SR6_20140106_1601_B181350= <br>JIT =3B - r11.b05_20131003_47443.02<br>GC =3B =3B - R26_Jav= a726_SR6_20140106_1601_B181350_CMPRSS<br>J9CL - 20140106_181350)<br>JCL - 2= 0140103_01 based on Oracle 7u51-b11<br><br>My Python script is just HTTP po= sts to a servlet and itself is very simple. I'm running this for a 4 hour d= uration with 7 processes and 600 threads. In my grinder.properties=2C I hav= e this set "grinder.jvm.arguments=3D-mx2048M".<br><br>When my run ends=2C I= get lots of heap dumps and error stacks like the below. I already have the= memory for each worker thread set to 2GB=2C do I need to set it even highe= r? I'm hoping not as I'm reaching the limit in memory on my machine. I'm no= t using the Grinder statistics for anything so if I can disable them to avo= id this error=2C I'm find with that.<br><br>Any ideas on how to proceed?<br= ><br>2016-03-01 18:28:35=2C512 ERROR worker-bootstrap: Error running worker= process<br>java.lang.OutOfMemoryError: Java heap space<br> =3B =3B=  =3B =3B =3B =3B =3B at java.lang.StringBuffer.ensureCa= pacityImpl(StringBuffer.java:338) ~[na:1.7.0]<br> =3B =3B =3B&n= bsp=3B =3B =3B =3B at java.lang.StringBuffer.append(StringBuffe= r.java:204) ~[na:1.7.0]<br> =3B =3B =3B =3B =3B =3B=  =3B at java.io.StringWriter.write(StringWriter.java:123) ~[na:1.7.0]<b= r> =3B =3B =3B =3B =3B =3B =3B at java.io.Print= Writer.write(PrintWriter.java:467) ~[na:1.7.0]<br> =3B =3B =3B&= nbsp=3B =3B =3B =3B at java.io.PrintWriter.write(PrintWriter.ja= va:484) ~[na:1.7.0]<br> =3B =3B =3B =3B =3B =3B&nbs= p=3B at java.io.PrintWriter.print(PrintWriter.java:614) ~[na:1.7.0]<br>&nbs= p=3B =3B =3B =3B =3B =3B =3B at net.grinder.statist= ics.StatisticsTable$1.next(StatisticsTable.java:150) ~[grinder-core-3.11.ja= r:na]<br> =3B =3B =3B =3B =3B =3B =3B at net.gr= inder.statistics.TestStatisticsMap$ForEach.iterate(TestStatisticsMap.java:3= 55) ~[grinder-core-3.11.jar:na]<br> =3B =3B =3B =3B =3B=  =3B =3B at net.grinder.statistics.StatisticsTable.print(Statistics= Table.java:146) ~[grinder-core-3.11.jar:na]<br> =3B =3B =3B&nbs= p=3B =3B =3B =3B at net.grinder.engine.process.GrinderProcess.r= un(GrinderProcess.java:554) ~[grinder-core-3.11.jar:na]<br> =3B =3B=  =3B =3B =3B =3B =3B at net.grinder.engine.process.Work= erProcessEntryPoint.run(WorkerProcessEntryPoint.java:86) [grinder-core-3.11= .jar:na]<br> =3B =3B =3B =3B =3B =3B =3B at net= .grinder.engine.process.WorkerProcessEntryPoint.main(WorkerProcessEntryPoin= t.java:59) [grinder-core-3.11.jar:na]<br> </div></div>=0A= <br>-----------------------------------------------------------------------= -------<br>=0A= Site24x7 APM Insight: Get Deep Visibility into Application Performance<br>= =0A= APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month<br>=0A= Monitor end-to-end web transactions and take corrective actions now<br>=0A= Troubleshoot faster and improve end-user experience. Signup Now!<br>=0A= <a href=3D"http://pubads.g.doubleclick.net/gampad/clk?id=3D272487151&=3B= iu=3D/4140" rel=3D"noreferrer" target=3D"_blank">http://pubads.g.doubleclic= k.net/gampad/clk?id=3D272487151&=3Biu=3D/4140</a><br>___________________= ____________________________<br>=0A= grinder-use mailing list<br>=0A= <a href=3D"mailto:[email protected]">[email protected]= ceforge.net</a><br>=0A= <a href=3D"https://lists.sourceforge.net/lists/listinfo/grinder-use" rel=3D= "noreferrer" target=3D"_blank">https://lists.sourceforge.net/lists/listinfo= /grinder-use</a><br>=0A= <br></blockquote></div><br></div>=0A= <br>-----------------------------------------------------------------------= -------=0A= Site24x7 APM Insight: Get Deep Visibility into Application Performance=0A= APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month=0A= Monitor end-to-end web transactions and take corrective actions now=0A= Troubleshoot faster and improve end-user experience. Signup Now!=0A= http://pubads.g.doubleclick.net/gampad/clk?id=3D272487151&=3Biu=3D/4140<= br>_______________________________________________=0A= grinder-use mailing list=0A= [email protected]=0A= https://lists.sourceforge.net/lists/listinfo/grinder-use</div> <= /div></body> </html>= --_533fc763-7ba0-4817-9c00-e8e2d329e48d_-- --===============1879539239103400084== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 --===============1879539239103400084== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ grinder-use mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/grinder-use --===============1879539239103400084==--