Re: Beowulf Cluster is very slow. Suggestions needed to increase the speed.

--e89a8f3bae2f9c26080505093a36
Content-Type: text/plain; charset=UTF-8

Hi,

The link you have provided me was very helpful. I haven't prepared myself
before starting clusters. Few years back i did clustering with three system
using PVM for different application i.e protein ligand docking. With three
systems it took some 2 hours to finish a particular library screening.
Since, it is few hours i was happy and haven't checked the speed of the
clusters. I thought to myself i learned a clustering technique. Therefore,
this time i started blindly before preparing clustering for different
applications which takes months to complete a specific job. Therefore, now
i learned depending upon the purpose of clustering requirement of things
will be different.

I learned that speed of the network is a biggest bottleneck for clustering
especially for our need. In our lab I have found a 1000Mbs switch which
presently reduces some time compared to 100Mbs however it is not efficient.
Still it takes a lot of time. I assume we need 10gigabit switch. I was not
aware of the price these "Gigabit routing switch supporting layer 3". It
seems to me that even 8-port 10 gigabit switch cost approx. 800$. I am
still hesitant to ask my mentor for the 10 gigabit switch. Since, I dont
have no experience i had a thought what if i am wrong somewhere although
gigabit switch have to work. Now, I am looking for 10 gigabit switch in
near by labs so that i will connect and check whether it sufficiently
efficient for our job and then i can ask my mentor with some confidence.

i also learned from someone we can use cluster OS specialized designed for
this clustering purpose for instance Pelican cluster
http://pareto.uab.es/mcreel/PelicanHPC/. I will install that cluster os and
check the performance too. I am reading materials to increase the speed and
performance of cluster in terms of hardware as well as the software's. I
will tune our application according to our need.

And, also in our lab room we have dual boot systems for 8 people (windows
with either linux mint, ubuntu, fedora). In given time those systems will
be either in windows or linux environment depending upon our work. Daily 6
hours (nights) and on Sundays our system will be idle. I am not capable to
use those free computer timing for our advantage due to my skill and also
time. I guess we can use these computer time if we have gigabit switch with
networking skills. If someone did similar stuffs please document either in
your blogs or in email lists so that users like me might get benefited.

Although, it took time i learned a lot during this clustering. Thank you
for your time.


regards
Suresh




On Wed, Oct 8, 2014 at 7:38 AM, Tomcio <[email protected]> wrote:

> Hello,
>
> been here since years, but very low number of questions so I'm so happy to
> see new thread here ;)
>
> I'm not boewulf cluster admin but..
>
> I belive Your network is slow because this routing device is internally
> limited to low throughput, You need Gigabit-capable routing device (read
> this as: Gigabit routing switch supporting layer 3).
> I suspect all Your class rooms are in different IP subnets?
> If yes then the routing device is bottleneck..
>
> Even modern Core i5 based linux router will create additional delay times
> in routing when it comes to cluster network, of course under high load at
> high throughput (what clustering requires). So simple, raw networking is
> best option.
>
> Example capable switch models that can do routing are:
> Cisco SG500X
> Cisco Catalyst series
> Dell PowerConnect 7000 series (or lower)
> Of course, they could be too expensive so just look around on switch that
> can do IP routing. There are even 8-port Gigabit Managed switches available
> on market.
>
> If Your sys admin isn't blocking other network subnets You could set
> static IP on all Your cluster nodes to different network subnet (let's
> say.. 192.168.99.1-192.168.99.10 with netmask 255.255.255.0) and check if
> they see each other (probably this isn't possible) and use one of them to
> provide Internet access (master node?) with 2 Gigabit cards.
>
> USB Gigabit cards are most up to ~480Mbit in throughput since they're USB
> 2.0. It's half of needed performance and! be aware of missing Linux
> drivers.. ;)
> I see all Your hardware is Gigabit, so I belive You need Layer 3 Gigabit
> switch. Also, please check are Your NICs supporting offload functions, then
> can help with high network load.
> I personally these in /etc/rc.local
> echo "Setting offload functions on Intel PRO/1000 NICs..."
> ethtool -K eth1 rx on tx on sg on gso on gro on tso on
> ethtool -K eth2 rx on tx on sg on gso on gro on tso on
>
> And there are tips:
> http://cs.boisestate.edu/~amit/research/beowulf/beowulf-setup.pdf
> See the section: 1.2 Networking Hardware
>
> Hope this helps.
>
> Cheers,
> TooMeeK
>
>
>
>
> W dniu 2014-10-07 05:50, suresh kannan pisze:
>
>  I am an Indian student in suwon, korea. I built a Beowulf cluster
>> (system information below) with four systems in our lab for our
>> simulation work with the help of good tutorials. In those tutorials they
>> have mentioned all the system should have static ip addresses.
>> Unfortunately, in all our labs we have been provided with dynamic ip
>> address[5 ips for 15 members in three separate labs]. I have requested
>> four more ip's from our university system admin. Due to the language
>> problems, i conveyed the requirement through my korean lab mate and i
>> dont know the reason why he denied us the static ip. So i found another
>> way to skip this procedure
>> http://www.reddit.com/r/linuxquestions/comments/
>> 2gubad/why_static_ip_address_is_necessary_for_linux/.
>>
>>
>> Someone suggested to use a router (one static ip) and set static ip for
>> the four computers through a router. I did that and it worked. However,
>> the cluster is very slow. For instance If i submit my simulation job in
>> a single computer [4 core processor], it takes 2 months to complete a
>> specific job. Although, if i connect 4 systems it shows it take 6 months
>> to complete the same job. It is actually using 10 core processor
>> [3,3,2,2-100% each]. I used TOP command to see how much processor the
>> head and other nodes are using. I have used openMPI to do parallel the
>> systems. I am using GROMACS (Parallelization based on MPI has been part
>> of this software). I followed a parallel configuration for the Gromacs
>> with the help of this tutorial
>> http://flakrat.blogspot.kr/2013/04/how-to-compile-
>> gromacs-461-with-openmpi.html.
>> After reading few posts
>> http://www.reddit.com/r/linuxquestions/comments/
>> 2gbgbg/what_would_be_the_best_linux_distro_for_folding/
>> i suspected the network router might be an issue.
>>
>> Can you suggest me how can i troubleshoot this problem? Some one
>> suggested to use 2 network ports and make linux as a router and use a
>> gigabitswitch to get the speed. However, we dont have 2 network ports
>> system. If this is compulsory i can buy network ports (USB one).
>>
>> Where do i start now?
>>
>> Can i make my head node as a router and use USB network port (for the
>> second network port) and connect to a gigabitswitch (any model
>> suggestion?) to connect other nodes. I dont know much about networking
>> stuffs. It will be helpful if any experts can suggest to troubleshoot
>> this issue.
>>
>> Thank you for your time.
>>
>> regards
>>
>> Suresh
>>
>>
>> System Informations
>>
>> Head node Processor : Intel core i3
>> RAM : 1 GB
>> No. of processor : 4
>> Network cards : 03:00.0 Ethernet controller: Realtek Semiconductor Co.,
>> Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
>> System company : Samsung
>> Architecture : x86_64
>> OS flavour : Linux Mint 17 Qiana
>>
>> Node1 Processor : Intel Quad core
>> RAM : 3 GB
>> No. of processor : 4
>> Network cards : 03:00.0 Ethernet controller: Realtek Semiconductor Co.,
>> Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 02)
>> System company : TG DREAMSYS
>> Architecture : x86_64
>> OS flavour : Linux Mint 17 Qiana
>>
>> Node2 Processor : Intel core i3
>> RAM : 1 GB
>> No. of processor : 4
>> Network cards : 03:00.0 Ethernet controller: Realtek Semiconductor Co.,
>> Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
>> System company : Samsung Architecture : x86_64
>> OS flavour : Linux Mint 17 Qiana
>>
>> Node3 Processor : Intel core i3
>> RAM : 1 GB
>> No. of processor : 2 Network cards : 02:00.0 Ethernet controller :
>> Qualcomm Atheros Attansic L2 Fast Ethernet (rev a0)
>> System company : JOOYONTECH
>> Architecture : x86_64
>> OS flavour : Linux Mint 17 Qiana
>>
>> Router Company : ipTIME N604R
>> Maximum speed : 160Mbps (LAN to WAN)
>>
>
>
> --
> To UNSUBSCRIBE, email to [email protected]
> with a subject of "unsubscribe". Trouble? Contact
> [email protected]
> Archive: https://lists.debian.org/[email protected]
>
>

--e89a8f3bae2f9c26080505093a36
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div>Hi,<br><br></div>The link you have provided=
 me was very helpful. I haven&#39;t prepared myself before starting cluster=
s. Few years back i did clustering with three system using PVM for differen=
t application i.e protein ligand docking. With three systems it took some 2=
 hours to finish a particular library screening. Since, it is few hours i w=
as happy and haven&#39;t checked the speed of the clusters. I thought to my=
self i learned a clustering technique. Therefore, this time i started blind=
ly before preparing clustering for different applications which takes month=
s to complete a specific job. Therefore, now i learned depending upon the p=
urpose of clustering requirement of things will be different.<br><br>I lear=
ned that speed of the network is a biggest bottleneck for clustering especi=
ally for our need. In our lab I have found a 1000Mbs switch which presently=
 reduces some time compared to 100Mbs however it is not efficient. Still it=
 takes a lot of time. I assume we need 10gigabit switch. I was not aware of=
 the price these &quot;Gigabit routing switch supporting layer 3&quot;. It =
seems to me that even 8-port 10 gigabit switch cost approx. 800$. I am stil=
l hesitant to ask my mentor for the 10 gigabit switch. Since, I dont have n=
o experience i had a thought what if i am wrong somewhere although gigabit =
switch have to work. Now, I am looking for 10 gigabit switch in near by lab=
s so that i will connect and check whether it sufficiently efficient for ou=
r job and then i can ask my mentor with some confidence.<br><br>i also lear=
ned from someone we can use cluster OS specialized designed for this cluste=
ring purpose for instance Pelican cluster <a href=3D"http://pareto.uab.es/m=
creel/PelicanHPC/">http://pareto.uab.es/mcreel/PelicanHPC/</a>. I will inst=
all that cluster os and check the performance too. I am reading materials t=
o increase the speed and performance of cluster in terms of hardware as wel=
l as the software&#39;s. I will tune our application according to our need.=
<br><br>And, also in our lab room we have dual boot systems for 8 people (w=
indows with either linux mint, ubuntu, fedora). In given time those systems=
 will be either in windows or linux environment depending upon our work. Da=
ily 6 hours (nights) and on Sundays our system will be idle. I am not capab=
le to use those free computer timing for our advantage due to my skill and =
also time. I guess we can use these computer time if we have gigabit switch=
 with networking skills. If someone did similar stuffs please document eith=
er in your blogs or in email lists so that users like me might get benefite=
d.<br><br>Although, it took time i learned a lot during this clustering. Th=
ank you for your time.</div></div><div><br><br></div><div>regards<br></div>=
<div>Suresh<br></div><br><div><br><br></div></div><div class=3D"gmail_extra=
"><br><div class=3D"gmail_quote">On Wed, Oct 8, 2014 at 7:38 AM, Tomcio <sp=
an dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"_blank">to=
[email protected]</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" s=
tyle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hell=
o,<br>
<br>
been here since years, but very low number of questions so I&#39;m so happy=
 to see new thread here ;)<br>
<br>
I&#39;m not boewulf cluster admin but..<br>
<br>
I belive Your network is slow because this routing device is internally lim=
ited to low throughput, You need Gigabit-capable routing device (read this =
as: Gigabit routing switch supporting layer 3).<br>
I suspect all Your class rooms are in different IP subnets?<br>
If yes then the routing device is bottleneck..<br>
<br>
Even modern Core i5 based linux router will create additional delay times i=
n routing when it comes to cluster network, of course under high load at hi=
gh throughput (what clustering requires). So simple, raw networking is best=
 option.<br>
<br>
Example capable switch models that can do routing are:<br>
Cisco SG500X<br>
Cisco Catalyst series<br>
Dell PowerConnect 7000 series (or lower)<br>
Of course, they could be too expensive so just look around on switch that c=
an do IP routing. There are even 8-port Gigabit Managed switches available =
on market.<br>
<br>
If Your sys admin isn&#39;t blocking other network subnets You could set st=
atic IP on all Your cluster nodes to different network subnet (let&#39;s sa=
y.. 192.168.99.1-192.168.99.10 with netmask 255.255.255.0) and check if the=
y see each other (probably this isn&#39;t possible) and use one of them to =
provide Internet access (master node?) with 2 Gigabit cards.<br>
<br>
USB Gigabit cards are most up to ~480Mbit in throughput since they&#39;re U=
SB 2.0. It&#39;s half of needed performance and! be aware of missing Linux =
drivers.. ;)<br>
I see all Your hardware is Gigabit, so I belive You need Layer 3 Gigabit sw=
itch. Also, please check are Your NICs supporting offload functions, then c=
an help with high network load.<br>
I personally these in /etc/rc.local<br>
echo &quot;Setting offload functions on Intel PRO/1000 NICs...&quot;<br>
ethtool -K eth1 rx on tx on sg on gso on gro on tso on<br>
ethtool -K eth2 rx on tx on sg on gso on gro on tso on<br>
<br>
And there are tips:<br>
<a href=3D"http://cs.boisestate.edu/~amit/research/beowulf/beowulf-setup.pd=
f" target=3D"_blank">http://cs.boisestate.edu/~<u></u>amit/research/beowulf=
/beowulf-<u></u>setup.pdf</a><br>
See the section: 1.2 Networking Hardware<br>
<br>
Hope this helps.<br>
<br>
Cheers,<br>
TooMeeK<br>
<br>
<br>
<br>
<br>
W dniu 2014-10-07 05:50, suresh kannan pisze:<div class=3D"HOEnZb"><div cla=
ss=3D"h5"><br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
I am an Indian student in suwon, korea. I built a Beowulf cluster<br>
(system information below) with four systems in our lab for our<br>
simulation work with the help of good tutorials. In those tutorials they<br=
>
have mentioned all the system should have static ip addresses.<br>
Unfortunately, in all our labs we have been provided with dynamic ip<br>
address[5 ips for 15 members in three separate labs]. I have requested<br>
four more ip&#39;s from our university system admin. Due to the language<br=
>
problems, i conveyed the requirement through my korean lab mate and i<br>
dont know the reason why he denied us the static ip. So i found another<br>
way to skip this procedure<br>
<a href=3D"http://www.reddit.com/r/linuxquestions/comments/2gubad/why_stati=
c_ip_address_is_necessary_for_linux/" target=3D"_blank">http://www.reddit.c=
om/r/<u></u>linuxquestions/comments/<u></u>2gubad/why_static_ip_address_<u>=
</u>is_necessary_for_linux/</a>.<br>
<br>
<br>
Someone suggested to use a router (one static ip) and set static ip for<br>
the four computers through a router. I did that and it worked. However,<br>
the cluster is very slow. For instance If i submit my simulation job in<br>
a single computer [4 core processor], it takes 2 months to complete a<br>
specific job. Although, if i connect 4 systems it shows it take 6 months<br=
>
to complete the same job. It is actually using 10 core processor<br>
[3,3,2,2-100% each]. I used TOP command to see how much processor the<br>
head and other nodes are using. I have used openMPI to do parallel the<br>
systems. I am using GROMACS (Parallelization based on MPI has been part<br>
of this software). I followed a parallel configuration for the Gromacs<br>
with the help of this tutorial<br>
<a href=3D"http://flakrat.blogspot.kr/2013/04/how-to-compile-gromacs-461-wi=
th-openmpi.html" target=3D"_blank">http://flakrat.blogspot.kr/<u></u>2013/0=
4/how-to-compile-<u></u>gromacs-461-with-openmpi.html</a>.<br>
After reading few posts<br>
<a href=3D"http://www.reddit.com/r/linuxquestions/comments/2gbgbg/what_woul=
d_be_the_best_linux_distro_for_folding/" target=3D"_blank">http://www.reddi=
t.com/r/<u></u>linuxquestions/comments/<u></u>2gbgbg/what_would_be_the_best=
_<u></u>linux_distro_for_folding/</a><br>
i suspected the network router might be an issue.<br>
<br>
Can you suggest me how can i troubleshoot this problem? Some one<br>
suggested to use 2 network ports and make linux as a router and use a<br>
gigabitswitch to get the speed. However, we dont have 2 network ports<br>
system. If this is compulsory i can buy network ports (USB one).<br>
<br>
Where do i start now?<br>
<br>
Can i make my head node as a router and use USB network port (for the<br>
second network port) and connect to a gigabitswitch (any model<br>
suggestion?) to connect other nodes. I dont know much about networking<br>
stuffs. It will be helpful if any experts can suggest to troubleshoot<br>
this issue.<br>
<br>
Thank you for your time.<br>
<br>
regards<br>
<br>
Suresh<br>
<br>
<br>
System Informations<br>
<br>
Head node Processor : Intel core i3<br>
RAM : 1 GB<br>
No. of processor : 4<br>
Network cards : 03:00.0 Ethernet controller: Realtek Semiconductor Co.,<br>
Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)<br>
System company : Samsung<br>
Architecture : x86_64<br>
OS flavour : Linux Mint 17 Qiana<br>
<br>
Node1 Processor : Intel Quad core<br>
RAM : 3 GB<br>
No. of processor : 4<br>
Network cards : 03:00.0 Ethernet controller: Realtek Semiconductor Co.,<br>
Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 02)<br>
System company : TG DREAMSYS<br>
Architecture : x86_64<br>
OS flavour : Linux Mint 17 Qiana<br>
<br>
Node2 Processor : Intel core i3<br>
RAM : 1 GB<br>
No. of processor : 4<br>
Network cards : 03:00.0 Ethernet controller: Realtek Semiconductor Co.,<br>
Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)<br>
System company : Samsung Architecture : x86_64<br>
OS flavour : Linux Mint 17 Qiana<br>
<br>
Node3 Processor : Intel core i3<br>
RAM : 1 GB<br>
No. of processor : 2 Network cards : 02:00.0 Ethernet controller :<br>
Qualcomm Atheros Attansic L2 Fast Ethernet (rev a0)<br>
System company : JOOYONTECH<br>
Architecture : x86_64<br>
OS flavour : Linux Mint 17 Qiana<br>
<br>
Router Company : ipTIME N604R<br>
Maximum speed : 160Mbps (LAN to WAN)<br>
</blockquote>
<br>
<br></div></div><span class=3D"HOEnZb"><font color=3D"#888888">
-- <br>
To UNSUBSCRIBE, email to <a href=3D"mailto:[email protected]=
ian.org" target=3D"_blank">debian-beowulf-REQUEST@lists.<u></u>debian.org</=
a><br>
with a subject of &quot;unsubscribe&quot;. Trouble? Contact <a href=3D"mail=
to:[email protected]" target=3D"_blank">[email protected]=
rg</a><br>
Archive: <a href=3D"https://lists.debian.org/[email protected]" target=
=3D"_blank">https://lists.debian.org/<u></u>[email protected]</a><br>
<br>
</font></span></blockquote></div><br></div>

--e89a8f3bae2f9c26080505093a36--