Re: Is a client robot an OS thread?
Alberto Klocker <alberto.klocker-A5pq3itJMno0n/[email protected]> Tue, 10 Sep 2013 15:59:42 +1000
| Newsgroups | gmane.comp.web.web-polygraph.user |
|---|---|
| Message-ID | <CAP7CNSWuKJGULkdUO-2sGm0_qsJtnJdvyL0dyRVjOQGEXKba=w@mail.gmail.com> |
--===============1399297289== Content-Type: multipart/alternative; boundary=047d7b3a99fc30864404e60137f3 --047d7b3a99fc30864404e60137f3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks for all the help! You can really see a lot of hard work and love has gone into Web-Polygraph. I've written a rails front end to handle the launching, gather all the files and host the finished report so your instructions fit perfectly. *Alberto Klocker* | Software Developer | www.bluereef.com.au *_________________________________________________________________* *D: *+61 3 9895 8006 | *T*: +61 3 9898 8000 | @AlbertoKlocker Please consider the environment before printing this email On Tue, Sep 10, 2013 at 2:53 AM, Pavel Kazlenka < [email protected]> wrote: > Hi Alberto, > > On 09/09/2013 04:40 AM, Alberto Klocker wrote: > > Hi, > > We run tests on a 10 Gb network environment with Polygraph as one of the = benchmarking tools but have found we are hitting 100% CPU usage on both the= poly-client and poly-server which would indicate we need to spread the pro= cesses and systems out. > > How well does the reporting engine handle log files from multiple sources= ? > > Say I were to script a system that launches two or three instances of the= client and server, will the reporting engine be capable of combining the r= esults into one report? > > All the polygraph reporting tools (like polygraph-lx, polygraph-ltrace, > polygraph-reporter) support multiple logs aggregation. E.g. if you have > logs from three client workers named clt.1.log, clt.2.log, clt.3.log you > can extract statistics from them simply using > $ polygraph-lx clt.1.log clt.2.log clt.3.log. Same for others. However, > there is a minor bug that prevents from ltrace tool extracting logs from > both server and client sides in one run: > https://bugs.launchpad.net/polygraph/+bug/1204983 > > Would it be better I write something that "merges" the results together? > > As a result of written above, you don't have to write anything. > > I'm curious to see what others have done to ramp Polygraph up above its = single core limits. > > We can get 10 Gb throughput with large files but my goal is to find maxim= um, stable concurrent connections. > > Any help is much appreciated! > > Sure Alberto. There is a number of actions that could help you increase > polygraph performance: > > > 1) Run multiple instances of workers and bind them to different cpu cores= . > E.g if you have 2 cpu cores on machine that runs polygraph-client, run tw= o > clients like: > taskset -c 0 polygraph-client --log clt.1.log ... > taskset -c 1 polygraph-client --log clt.2.log ... > > 2) Bind tune cpu affinity for NIC interrupts to cores different from the > ones that already occupied by polygraph workers (clients or servers). E.g= . > if you have 4 cpu cores, consider to run three polygraph-client instances > bound to cores 0-2 and bind NIC interrupts to core 3. This article should > help you with SMP affinity in linux: > http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-= linux. There is no good formula to say what should be ratio between workers= ' and > NIC's cores, but top output could give you the clues. > > 3) If you have 'virtual' (hyper-threading provided) cores, it's not a goo= d > idea to bind polygraph workers to both 'real' and 'virtual' instances of > the same core. > > 4) If you have enough RAM, consider writing test results on in-memory FS > like tmpfs (https://www.kernel.org/doc/Documentation/filesystems/tmpfs.tx= t > ). > > 5) Tuning linux TCP stack for 10G networking is important part too. I'd > use this document as a guide: > http://landley.net/kdocs/ols/2009/ols2009-pages-169-184.pdf . > > Best wishes, > Pavel > > Kevin Xie1 <Kevin_Xie1 at symantec.com <http://www.web-polygraph.org/ma= ilman/listinfo/users>> writes: > > >* Thanks for your quick response, Dmitry.*>**>* However, I don't underst= and how a single thread can simulate multiple*>* concurrent "users" submitt= ing request to web servers. Is it that the*>* single thread iterates all op= en connections one by one (submitting a*>* request or processing response o= ne at a time)?* > Sort of, though details are more complex than that. If you are > interested, search for I/O multiplexing. > > >* bound by CPU since just 1 core is used at a time, or by slow*>* connec= tions ..., do I miss something here, or my understanding is*>* fundamentall= y wrong?*>** > The fact that a single polygraph process can use only a single core may > be a limitation indeed. But moving each simulated agent to a separate > OS thread is not the solution. > > You can run multiple Polygraph client or server processes on a single > system. But that makes it harder to manage the test and usually > requires some auxiliary scripts to start all processes. > > It would be nice to have SMP support in Polygraph and we have some ideas > about it. But so far there has not been enough interest in such > project. Patches or sponsorships are welcome. > > In any case, please do not make conclusions about Polygraph performance > from the fact that it is single-threaded. Polygraph is successfully > used for high performance tests in 10Gbit networks (though it may > require multiple Polygraph processes and/or systems). > > Regards, > Dmitry > > >* Appreciated for any light shed!*>**>* Kevin*>**>* -----Original Messag= e-----*>* From: Dmitry Kurochkin [mailto:dmitry.kurochkin at measurement-fa= ctory.com <http://www.web-polygraph.org/mailman/listinfo/users>] *>* Sent: = May-03-12 3:19 PM*>* To: Kevin Xie1*>* Cc: users at web-polygraph.org <http= ://www.web-polygraph.org/mailman/listinfo/users>*>* Subject: Re: Is a clien= t robot an OS thread?*>**>* Hi Kevin.*>**>* Kevin Xie1 <Kevin_Xie1 at syman= tec.com <http://www.web-polygraph.org/mailman/listinfo/users>> writes:*>**>= >* Hi,*>>**>>* Is a client robot a real OS thread? I'm using web polygrah i= n Linux,*>>* but I doesn't see multiple threads in the OS during a test wit= h 500*>>* robots.*>>**>**>* Polygraph client and server processes are singl= e-threaded. All Robot*>* and Server agents are running in a single thread.= *>**>* Regards,*>* Dmitry*>**>>* Thanks,*>>* Kevin* > > *Alberto Klocker* | Software Developer | www.bluereef.com.au > *_________________________________________________________________* > *D: *+61 3 9895 8006 | *T*: +61 3 9898 8000 | @AlbertoKlocker > Please consider the environment before printing this email > > > _______________________________________________ > Users mailing listUsers-mN9fGWdlm5pku/f+YMZH/[email protected]://www.web-polygraph.org/mai= lman/listinfo/users > > > --047d7b3a99fc30864404e60137f3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Thanks for all the help! You can really see a lot of hard = work and love has gone into Web-Polygraph.<div><br></div><div>I've writ= ten a rails front end to handle the launching, gather all the files and hos= t the finished report so your instructions fit perfectly.</div> </div><div class=3D"gmail_extra"><br clear=3D"all"><div><div dir=3D"ltr"><s= pan style=3D"background-color:rgb(255,255,255);border-collapse:separate;col= or:rgb(34,34,34);font-size:medium;font-family:Helvetica"><div style=3D"word= -wrap:break-word"> <span style=3D"border-collapse:separate"><div style=3D"word-wrap:break-word= "><span style=3D"border-collapse:separate"><div style=3D"margin:0cm 0cm 0.0= 001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang=3D"EN-US" s= tyle=3D"font-size:8pt;font-family:Verdana,sans-serif;color:rgb(54,95,145)">= <b>Alberto Klocker</b>=A0| Software Developer</span><span style=3D"font-siz= e:8pt;font-family:Verdana,sans-serif;color:rgb(54,95,145)">=A0|=A0</span><s= pan lang=3D"EN-US"><a href=3D"http://www.bluereef.com.au/" style=3D"color:b= lue" target=3D"_blank"><span lang=3D"EN-AU" style=3D"font-size:8pt;font-fam= ily:Verdana,sans-serif;color:rgb(54,95,145);text-decoration:none">www.bluer= eef.com.au</span></a></span><span lang=3D"EN-US" style=3D"font-size:8pt;col= or:rgb(54,95,145)">=A0</span><span style=3D"font-size:8pt;color:rgb(54,95,1= 45)"></span></div> <div style=3D"margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sa= ns-serif"><u><span style=3D"font-size:8pt;color:rgb(54,95,145)">___________= ______________________________________________________</span></u></div></sp= an></div> </span></div></span><span style=3D"background-color:rgb(255,255,255);color:= rgb(34,34,34);font-size:15px;font-family:Calibri,sans-serif"><span lang=3D"= EN-US" style=3D"font-size:8pt;font-family:Verdana,sans-serif;color:rgb(54,9= 5,145)"><b>D:=A0</b><a value=3D"+61398958020" style=3D"color:rgb(17,85,204)= ">+61 3 9895 8006</a>=A0|=A0</span></span><span style=3D"background-color:r= gb(255,255,255);border-collapse:separate;color:rgb(34,34,34);font-size:medi= um;font-family:Helvetica"><span style=3D"font-size:15px;font-family:Calibri= ,sans-serif"><b><span lang=3D"EN-US" style=3D"font-size:8pt;font-family:Ver= dana,sans-serif;color:rgb(54,95,145)">T</span></b></span><span style=3D"fon= t-size:15px;font-family:Calibri,sans-serif"><span style=3D"font-size:8pt;fo= nt-family:Verdana,sans-serif;color:rgb(54,95,145)">:=A0<a value=3D"+6139898= 8000" style=3D"color:rgb(17,85,204)">+61 3 9898 8000</a>=A0</span></span><s= pan style=3D"font-size:15px;font-family:Calibri,sans-serif"><span style=3D"= font-size:8pt;font-family:Verdana,sans-serif;color:rgb(54,95,145)">|=A0</sp= an></span><a value=3D"+61398988000" style=3D"font-family:Verdana,sans-serif= ;font-size:11px;color:rgb(17,85,204)">@AlbertoKlocker</a><div style=3D"word= -wrap:break-word"> <span style=3D"border-collapse:separate"><div style=3D"word-wrap:break-word= "><span style=3D"border-collapse:separate"><span style=3D"border-collapse:s= eparate"><span style=3D"font-family:'Lucida Grande'"><div style=3D"= margin:0cm 0cm 0.0001pt;font-size:12pt;font-family:'Times New Roman'= ;,serif"> <span style=3D"border-collapse:separate;font-size:medium;font-family:Calibr= i"><span style=3D"font-family:'Times New Roman',serif;font-size:16p= x"><span style=3D"border-collapse:separate;font-size:medium;font-family:Cal= ibri"><div style=3D"margin:0cm 0cm 0.0001pt;font-size:12pt;font-family:'= ;Times New Roman',serif"> <span lang=3D"EN-US" style=3D"font-size:8pt;font-family:Verdana,sans-serif;= color:rgb(30,147,102)">Please consider the environment before printing this= email</span></div></span></span></span></div></span></span></span></div></= span></div> </span></div></div> <br><br><div class=3D"gmail_quote">On Tue, Sep 10, 2013 at 2:53 AM, Pavel K= azlenka <span dir=3D"ltr"><<a href=3D"mailto:pavel.kazlenka@measurement-= factory.com" target=3D"_blank">[email protected]</a>&g= t;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> =20 =20 =20 <div bgcolor=3D"#FFFFFF" text=3D"#000000"><div class=3D"im"> <div>Hi Alberto,<br> <br> On 09/09/2013 04:40 AM, Alberto Klocker wrote:<br> </div> </div><div class=3D"im"><blockquote type=3D"cite"> <div dir=3D"ltr"> <pre style>Hi,</pre> <pre style>We run tests on a 10 Gb network environment with Polygra= ph as one of the benchmarking tools but have found we are hitting 100% CPU = usage on both the poly-client and poly-server which would indicate we need = to spread the processes and systems out.</pre> <pre style>How well does the reporting engine handle log files from= multiple sources?=A0</pre> <pre style><span style=3D"font-family:arial">Say I were to script a= system that launches two or three instances of the client and server, will= the reporting engine be capable of combining the results into one report?<= /span> </pre> </div> </blockquote></div><div class=3D"im"> All the polygraph reporting tools (like polygraph-lx, polygraph-ltrace, polygraph-reporter) support multiple logs aggregation. E.g. if you have logs from three client workers named clt.1.log, clt.2.log, clt.3.log you can extract statistics from them simply using <br> $ polygraph-lx clt.1.log clt.2.log clt.3.log. Same for others. However, there is a minor bug that prevents from ltrace tool extracting logs from both server and client sides in one run: <a href= =3D"https://bugs.launchpad.net/polygraph/+bug/1204983" target=3D"_blank">ht= tps://bugs.launchpad.net/polygraph/+bug/1204983</a> </div><blockquote type=3D"cite"> <div dir=3D"ltr"> <pre style><span style=3D"font-family:arial">Would it be better I w= rite something that "merges" the results together?</span></pre> </div> </blockquote> As a result of written above, you don't have to write anything. <blockquote type=3D"cite"> <div dir=3D"ltr"> <pre style>I'm curious to see what others have done to ramp Pol= ygraph up above its single core limits.</pre><div class=3D"im"> <pre style>We can get 10 Gb throughput with large files but my goal= is to find maximum, stable concurrent connections.</pre> <pre style>Any help is much appreciated!</pre> </div></div> </blockquote> Sure Alberto. There is a number of actions that could help you increase polygraph performance:<div class=3D"im"><br> <br> 1) Run multiple instances of workers and bind them to different cpu cores. E.g if you have 2 cpu cores on machine that runs polygraph-client, run two clients like:<br> taskset -c 0 polygraph-client --log clt.1.log ...<br> taskset -c 1 polygraph-client --log clt.2.log ...<br> <br> 2) Bind tune cpu affinity for NIC interrupts to cores different from the ones that already occupied by polygraph workers (clients or servers). E.g. if you have 4 cpu cores, consider to run three polygraph-client instances bound to cores 0-2 and bind NIC interrupts to core 3. This article should help you with SMP affinity in linux: <a href=3D"http://www.alexonlinux.com/smp-affinity-and-proper= -interrupt-handling-in-linux" target=3D"_blank">http://www.alexonlinux.com/= smp-affinity-and-proper-interrupt-handling-in-linux</a> . There is no good formula to say what should be ratio between workers' and NIC's cores, but top output could give you the clu= es.<br> <br> 3) If you have 'virtual' (hyper-threading provided) cores, it&#= 39;s not a good idea to bind polygraph workers to both 'real' and 'v= irtual' instances of the same core.<br> <br> 4) If you have enough RAM, consider writing=A0 test results on in-memory FS like tmpfs (<a href=3D"https://www.kernel.org/doc/Document= ation/filesystems/tmpfs.txt" target=3D"_blank">https://www.kernel.org/doc/D= ocumentation/filesystems/tmpfs.txt</a>).<br> <br></div> 5) Tuning linux TCP stack for 10G networking is important part too. I'd use this document as a guide: <a href=3D"http://landley.net/kdocs/ols/2009/ols2009-pages-169-184.pdf"= target=3D"_blank">http://landley.net/kdocs/ols/2009/ols2009-pages-169-184.= pdf</a> .<br> <br> Best wishes,<br> Pavel <blockquote type=3D"cite"><div><div class=3D"h5"> <div dir=3D"ltr"> <pre style> </pre> <pre style>Kevin Xie1 <<a href=3D"http://www.web-polygraph.org/m= ailman/listinfo/users" target=3D"_blank">Kevin_Xie1 at symantec.com</a>>= writes: ><i> Thanks for your quick response, Dmitry. </i>><i> </i>><i> However, I don't understand how a single thread can simulat= e multiple </i>><i> concurrent "users" submitting request to web servers.= Is it that the </i>><i> single thread iterates all open connections one by one (submitt= ing a </i>><i> request or processing response one at a time)? </i> Sort of, though details are more complex than that. If you are interested, search for I/O multiplexing. ><i> bound by CPU since just 1 core is used at a time, or by slow </i>><i> connections ..., do I miss something here, or my understanding = is </i>><i> fundamentally wrong? </i>><i> </i> The fact that a single polygraph process can use only a single core may be a limitation indeed. But moving each simulated agent to a separate OS thread is not the solution. You can run multiple Polygraph client or server processes on a single system. But that makes it harder to manage the test and usually requires some auxiliary scripts to start all processes. It would be nice to have SMP support in Polygraph and we have some ideas about it. But so far there has not been enough interest in such project. Patches or sponsorships are welcome. In any case, please do not make conclusions about Polygraph performance from the fact that it is single-threaded. Polygraph is successfully used for high performance tests in 10Gbit networks (though it may require multiple Polygraph processes and/or systems). Regards, Dmitry ><i> Appreciated for any light shed! </i>><i> </i>><i> Kevin </i>><i> </i>><i> -----Original Message----- </i>><i> From: Dmitry Kurochkin [mailto:<a href=3D"http://www.web-polygr= aph.org/mailman/listinfo/users" target=3D"_blank">dmitry.kurochkin at measu= rement-factory.com</a>]=20 </i>><i> Sent: May-03-12 3:19 PM </i>><i> To: Kevin Xie1 </i>><i> Cc: <a href=3D"http://www.web-polygraph.org/mailman/listinfo/us= ers" target=3D"_blank">users at web-polygraph.org</a> </i>><i> Subject: Re: Is a client robot an OS thread? </i>><i> </i>><i> Hi Kevin. </i>><i> </i>><i> Kevin Xie1 <<a href=3D"http://www.web-polygraph.org/mailman/= listinfo/users" target=3D"_blank">Kevin_Xie1 at symantec.com</a>> writes= : </i>><i> </i>>><i> Hi, </i>>><i> </i>>><i> Is a client robot a real OS thread? I'm using web polyg= rah in Linux, </i>>><i> but I doesn't see multiple threads in the OS during a t= est with 500 </i>>><i> robots. </i>>><i> </i>><i> </i>><i> Polygraph client and server processes are single-threaded. All= Robot </i>><i> and Server agents are running in a single thread. </i>><i> </i>><i> Regards, </i>><i> Dmitry </i>><i> </i>>><i> Thanks, </i>>><i> Kevin</i></pre> <div> <div dir=3D"ltr"><span style=3D"border-collapse:separate;color:rg= b(34,34,34);font-size:medium;font-family:Helvetica"> <div style=3D"word-wrap:break-word"><span style=3D"border-col= lapse:separate"> <div style=3D"word-wrap:break-word"> <span style=3D"border-collapse:separate"> <div style=3D"margin:0cm 0cm 0.0001pt;font-size:11pt;= font-family:Calibri,sans-serif"><span style=3D"font-size:8pt;font-family:Ve= rdana,sans-serif;color:rgb(54,95,145)" lang=3D"EN-US"><b>Alberto Klocker</b= >=A0| Software Developer</span><span style=3D"font-size:8pt;font= -family:Verdana,sans-serif;color:rgb(54,95,145)">=A0|=A0</span><span lang= =3D"EN-US"><a href=3D"http://www.bluereef.com.au/" style=3D"color:blue" tar= get=3D"_blank"><span style=3D"font-size:8pt;font-family:Verdana,sans-serif;= color:rgb(54,95,145);text-decoration:none" lang=3D"EN-AU">www.bluereef.com.= au</span></a></span><span style=3D"font-size:8pt;color:rgb(54,95,145)" lang= =3D"EN-US">=A0</span><span style=3D"font-size:8pt;color:rgb(54,95,145)"></s= pan></div> <div style=3D"margin:0cm 0cm 0.0001pt;font-size:11pt;= font-family:Calibri,sans-serif"><u><span style=3D"font-size:8pt;color:rgb(5= 4,95,145)">________________________________________________________________= _</span></u></div> </span></div> </span></div> </span><span style=3D"color:rgb(34,34,34);font-size:15px;font-f= amily:Calibri,sans-serif"><span style=3D"font-size:8pt;font-family:Verdana,= sans-serif;color:rgb(54,95,145)" lang=3D"EN-US"><b>D:=A0</b><a value=3D"+61= 398958020" style=3D"color:rgb(17,85,204)">+61 3 9895 8006</a>=A0|=A0</span></span><span style=3D"border= -collapse:separate;color:rgb(34,34,34);font-size:medium;font-family:Helveti= ca"><span style=3D"font-size:15px;font-family:Calibri,sans-serif"><b><span = style=3D"font-size:8pt;font-family:Verdana,sans-serif;color:rgb(54,95,145)"= lang=3D"EN-US">T</span></b></span><span style=3D"font-size:15px;font-famil= y:Calibri,sans-serif"><span style=3D"font-size:8pt;font-family:Verdana,sans= -serif;color:rgb(54,95,145)">:=A0<a value=3D"+61398988000" style=3D"color:r= gb(17,85,204)">+61 3 9898 8000</a>=A0</span></span><span style=3D"font-size= :15px;font-family:Calibri,sans-serif"><span style=3D"font-size:8pt;font-fam= ily:Verdana,sans-serif;color:rgb(54,95,145)">|=A0</span></span><a value=3D"= +61398988000" style=3D"font-family:Verdana,sans-serif;font-size:11px;color:= rgb(17,85,204)">@AlbertoKlocker</a> <div style=3D"word-wrap:break-word"> <span style=3D"border-collapse:separate"> <div style=3D"word-wrap:break-word"><span style=3D"border= -collapse:separate"><span style=3D"border-collapse:separate"><span style=3D= "font-family:'Lucida Grande'"> <div> <span style=3D"border-collapse:separate;font-si= ze:medium;font-family:Calibri"><span><span style=3D"border-collapse:separat= e;font-size:medium;font-family:Calibri"> <div> <span style=3D"font-size:8pt;font-famil= y:Verdana,sans-serif;color:rgb(30,147,102)" lang=3D"EN-US">Please consider = the environment before printing this email</span></div> </span></span></span></div> </span></span></span></div> </span></div> </span></div> </div> </div> <br> <fieldset></fieldset> <br> </div></div><div class=3D"im"><pre>__________________________________= _____________ Users mailing list <a href=3D"mailto:Users-mN9fGWdlm5pku/f+YMZH/di2O/[email protected]" target=3D"_blank">Users@web-poly= graph.org</a> <a href=3D"http://www.web-polygraph.org/mailman/listinfo/users" target=3D"_= blank">http://www.web-polygraph.org/mailman/listinfo/users</a> </pre> </div></blockquote> <br> </div> </blockquote></div><br></div> --047d7b3a99fc30864404e60137f3-- --===============1399297289== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Users mailing list Users-mN9fGWdlm5pku/f+YMZH/di2O/[email protected] http://www.web-polygraph.org/mailman/listinfo/users --===============1399297289==--