Re: [Openmosix-Devel] Recommendations
"Wes Wagner" <[email protected]> Wed, 18 Jul 2007 07:24:22 -0700
| Newsgroups | gmane.linux.cluster.openmosix.general,gmane.linux.cluster.openmosix.devel |
|---|---|
| Message-ID | <[email protected]> |
--===============0470836023== Content-Type: multipart/alternative; boundary="----=_Part_53448_25932886.1184768662188" ------=_Part_53448_25932886.1184768662188 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline What we are using openmosix for is genetic programming, which is cpu intensive, not quite so memory intensive, and produces lots of single threaded processes since genetic programing is inherently tons more simple serialized rather than multi-threaded. It is better to have one thread per organism and just increase your population sizes to achieve the results you want. Now one could load all this into a queuing program - and run it via remote shell, but the other problem is that your processes run for a long time. This presents real problems if you want to take a node offline near immediately, but you don't want to abort the run. The other virtue of openmosix is the ability to PXE boot/CD boot random systems all over the network and borrow them, plus later gracefully shut them down. Overall, I evaluated alot of solutions and openmosix made the most sense because it was a set and forget type infrastructure. Plus with tools such as omuscd, you can get very lightweight workload balancing. Though queuing systems can balance how much workload they start on a system, major changes have to be made to your coding methods to provide for process migration. Sincerely, Wes Wagner On 7/18/07, Moshe Bar <[email protected]> wrote: > > Paul, Wes, all > > I actually meant to write to the list before the week-end, but this is > a good opportunity. > > It's great to see that a few people have jumped into action (incl. > Florian) to try to finish the port of oM to 2.6. > > As to the reasons for my decision: > > It is true that time and resource constraints have played a role, but > only a minor one. After all, I have been able to keep the project > going while I started no less than 5 companies and keep them all > running, am involved with several others, wrote two books, and did > several of the other items on my list "things to do before I die". > > The real reasons for my decisions are indeed tied more to the > long-term vision and feasibility of the project. openMosix is about > high performance computing, in other words let CPU (and to some > extent) RAM intensive apps run as fast as possible without hindrance > from the limitation of resources on local computers. We have all seen > that there are severe limitations to the SSI model due to the extreme > high impact of network latency, and due to the infeasible cost of > fully virtualizing the memory model (ie DSM), and the OS control block > across several nodes. > > It is tempting to think that with massive multi-cores, the advantage > of an SSI cluster stays because then you have "even more CPUs to play > with". That is not so. The migration from a multi-core system where > multi-threaded applications (as all apps are currently being > re-written as) thrive and performe well to another such node, implies > a DSM model capable to relocate threads either in gangs or single > across nodes. In other words multi-cores are changing the application > landscape towards a massively threaded one, and these apps will not > perform well in a DSM cluster by definition due to very high true and > false sharing. > > Tests we performed at my company Qlusters with very low latency > inter-connects (< 1microsecond round-trip latency) with a very > efficient implementation of DSM have shown an regular performance > impacts of 100x to 1000x. > > Even the sgi Altix let's threads run in the same SMP quadrant. > > > The other big reason is that I have over the last two years > intensively looked for a successor and was not able to find somebody > with the willingness, stamina, maturity, leadership, and coding > expertise to take over. > > If this announcement brings forward somebody to deal with this latter > issue, that doesn't solve the prior issue of multi-core application > models. > > Finally, another reason is that user numbers have been going down very > quickly and mostly these were users with, say, 4 or 5 nodes with a > total of 4 to 6 CPUs. Well, you can get all of that with much simpler > ease of use with a dual socket, quad core system today. > > Please hammer away with comments. :-) > > Moshe > > > > On 7/18/07, Wes Wagner <[email protected]> wrote: > > Paul, > > > > I am working with the author of omuscd on a 2.6 openmosix port to the > 2.6.22 > > Kernel... I need to talk with him some more, but you should expect that > the > > openmosix cluster will be picked up by another team. > > > > I just need until this weekend to get things organized. I believe Moshe > > underestimates the need for using multicore systems in a massive set and > > forget mode so people can do without the overhead of being concerned > with > > queue and batch management. > > > > We have such a need and are currently building a 40 core farm expected > to > > grow to the thousands of cores. It is my personal opinion that this > > announcement came mostly due to time and resource constraints, not the > lack > > of a need for SSI style clustering. > > > > Please watch this space in the coming weeks: > > > > https://sourceforge.net/projects/om-2-6 > > > > Our intent was originally to provide a source tree where you could > download > > an x386 x686 x64_86 source that included the omuscd management daemon > and > > went clean onto a vanilla kernel. I expect that the project will expand. > > > > Sincerely, > > Wes Wagner > > > > > > > > > > > > > > On 7/18/07, Paul Bennett <[email protected] > wrote: > > > > > > Hi List, > > > > > > I just found out that openMosix was getting end-of-lifed. Does anyone > > have any recommendations for another clustering option? > > > > > > I would like to go with something based on the 2.6 kernel and amd_64. > > > > > > http://sourceforge.net/forum/forum.php?forum_id=715406 > > > > > > Thanks, > > > > > > Paul > > > > > > > > > ------------------------------------------------------------------------- > > > This SF.net email is sponsored by DB2 Express > > > Download DB2 Express C - the FREE version of DB2 express and take > > > control of your XML. No limits. Just data. Click to get it now. > > > http://sourceforge.net/powerbar/db2/ > > > _______________________________________________ > > > openMosix-devel mailing list > > > openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org > > > > > https://lists.sourceforge.net/lists/listinfo/openmosix-devel > > > > > > > > > > > > > > -- > > Wes Wagner > > > > Join a libertarian network of person-to-person lending on Prosper: > > > > > http://www.prosper.com/groups/group_home.aspx?group_short_name=freelibertarians&referrer=AiriusTorpora&utm_source=referrer-AiriusTorpora&utm_medium=referral-link&utm_content=join_my_group-160x33&utm_campaign=referrals-group > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by DB2 Express > > Download DB2 Express C - the FREE version of DB2 express and take > > control of your XML. No limits. Just data. Click to get it now. > > http://sourceforge.net/powerbar/db2/ > > _______________________________________________ > > openMosix-devel mailing list > > openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org > > https://lists.sourceforge.net/lists/listinfo/openmosix-devel > > > > > -- Wes Wagner Join a libertarian network of person-to-person lending on Prosper: http://www.prosper.com/groups/group_home.aspx?group_short_name=freelibertarians&referrer=AiriusTorpora&utm_source=referrer-AiriusTorpora&utm_medium=referral-link&utm_content=join_my_group-160x33&utm_campaign=referrals-group ------=_Part_53448_25932886.1184768662188 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline What we are using openmosix for is genetic programming, which is cpu intensive, not quite so memory intensive, and produces lots of single threaded processes since genetic programing is inherently tons more simple serialized rather than multi-threaded. It is better to have one thread per organism and just increase your population sizes to achieve the results you want. <br><br>Now one could load all this into a queuing program - and run it via remote shell, but the other problem is that your processes run for a long time. This presents real problems if you want to take a node offline near immediately, but you don't want to abort the run. The other virtue of openmosix is the ability to PXE boot/CD boot random systems all over the network and borrow them, plus later gracefully shut them down. <br><br>Overall, I evaluated alot of solutions and openmosix made the most sense because it was a set and forget type infrastructure. Plus with tools such as omuscd, you can get very lightweight workload balancing. Though queuing systems can balance how much workload they start on a system, major changes have to be made to your coding methods to provide for process migration. <br><br>Sincerely,<br>Wes Wagner<br><br><br><div><span class="gmail_quote">On 7/18/07, <b class="gmail_sendername">Moshe Bar</b> <<a href="mailto:[email protected]">[email protected]</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Paul, Wes, all<br><br>I actually meant to write to the list before the week-end, but this is<br>a good opportunity.<br><br>It's great to see that a few people have jumped into action (incl.<br>Florian) to try to finish the port of oM to 2.6.<br><br>As to the reasons for my decision:<br><br>It is true that time and resource constraints have played a role, but<br>only a minor one. After all, I have been able to keep the project<br>going while I started no less than 5 companies and keep them all <br>running, am involved with several others, wrote two books, and did<br>several of the other items on my list "things to do before I die".<br><br>The real reasons for my decisions are indeed tied more to the<br> long-term vision and feasibility of the project. openMosix is about<br>high performance computing, in other words let CPU (and to some<br>extent) RAM intensive apps run as fast as possible without hindrance<br>from the limitation of resources on local computers. We have all seen <br>that there are severe limitations to the SSI model due to the extreme<br>high impact of network latency, and due to the infeasible cost of<br>fully virtualizing the memory model (ie DSM), and the OS control block<br>across several nodes. <br><br>It is tempting to think that with massive multi-cores, the advantage<br>of an SSI cluster stays because then you have "even more CPUs to play<br>with". That is not so. The migration from a multi-core system where <br>multi-threaded applications (as all apps are currently being<br>re-written as) thrive and performe well to another such node, implies<br>a DSM model capable to relocate threads either in gangs or single<br>across nodes. In other words multi-cores are changing the application <br>landscape towards a massively threaded one, and these apps will not<br>perform well in a DSM cluster by definition due to very high true and<br>false sharing.<br><br>Tests we performed at my company Qlusters with very low latency <br>inter-connects (< 1microsecond round-trip latency) with a very<br>efficient implementation of DSM have shown an regular performance<br>impacts of 100x to 1000x.<br><br>Even the sgi Altix let's threads run in the same SMP quadrant. <br><br><br>The other big reason is that I have over the last two years<br>intensively looked for a successor and was not able to find somebody<br>with the willingness, stamina, maturity, leadership, and coding<br>expertise to take over. <br><br>If this announcement brings forward somebody to deal with this latter<br>issue, that doesn't solve the prior issue of multi-core application<br>models.<br><br>Finally, another reason is that user numbers have been going down very <br>quickly and mostly these were users with, say, 4 or 5 nodes with a<br>total of 4 to 6 CPUs. Well, you can get all of that with much simpler<br>ease of use with a dual socket, quad core system today.<br><br>Please hammer away with comments. :-) <br><br>Moshe<br><br><br><br>On 7/18/07, Wes Wagner <<a href="mailto:[email protected]">[email protected]</a>> wrote:<br>> Paul,<br>><br>> I am working with the author of omuscd on a 2.6 openmosix port to the 2.6.22<br>> Kernel... I need to talk with him some more, but you should expect that the<br>> openmosix cluster will be picked up by another team.<br>><br>> I just need until this weekend to get things organized. I believe Moshe <br>> underestimates the need for using multicore systems in a massive set and<br>> forget mode so people can do without the overhead of being concerned with<br>> queue and batch management.<br>><br>> We have such a need and are currently building a 40 core farm expected to <br>> grow to the thousands of cores. It is my personal opinion that this<br>> announcement came mostly due to time and resource constraints, not the lack<br>> of a need for SSI style clustering.<br>><br>> Please watch this space in the coming weeks: <br>><br>> <a href="https://sourceforge.net/projects/om-2-6">https://sourceforge.net/projects/om-2-6</a><br>><br>> Our intent was originally to provide a source tree where you could download<br>> an x386 x686 x64_86 source that included the omuscd management daemon and <br>> went clean onto a vanilla kernel. I expect that the project will expand.<br>><br>> Sincerely,<br>> Wes Wagner<br>><br>><br>><br>><br>><br>><br>> On 7/18/07, Paul Bennett <<a href="mailto:[email protected]"> [email protected]</a> > wrote:<br>> ><br>> > Hi List,<br>> ><br>> > I just found out that openMosix was getting end-of-lifed. Does anyone<br>> have any recommendations for another clustering option? <br>> ><br>> > I would like to go with something based on the 2.6 kernel and amd_64.<br>> ><br>> > <a href="http://sourceforge.net/forum/forum.php?forum_id=715406">http://sourceforge.net/forum/forum.php?forum_id=715406 </a><br>> ><br>> > Thanks,<br>> ><br>> > Paul<br>> ><br>> ><br>> -------------------------------------------------------------------------<br>> > This SF.net email is sponsored by DB2 Express <br>> > Download DB2 Express C - the FREE version of DB2 express and take<br>> > control of your XML. No limits. Just data. Click to get it now.<br>> > <a href="http://sourceforge.net/powerbar/db2/">http://sourceforge.net/powerbar/db2/ </a><br>> > _______________________________________________<br>> > openMosix-devel mailing list<br>> > <a href="mailto:openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org">openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org</a><br> > ><br>> <a href="https://lists.sourceforge.net/lists/listinfo/openmosix-devel">https://lists.sourceforge.net/lists/listinfo/openmosix-devel</a><br>> ><br>> ><br>><br>><br>><br>> --<br>> Wes Wagner <br>><br>> Join a libertarian network of person-to-person lending on Prosper:<br>><br>> <a href="http://www.prosper.com/groups/group_home.aspx?group_short_name=freelibertarians&referrer=AiriusTorpora&utm_source=referrer-AiriusTorpora&utm_medium=referral-link&utm_content=join_my_group-160x33&utm_campaign=referrals-group"> http://www.prosper.com/groups/group_home.aspx?group_short_name=freelibertarians&referrer=AiriusTorpora&utm_source=referrer-AiriusTorpora&utm_medium=referral-link&utm_content=join_my_group-160x33&utm_campaign=referrals-group </a><br>> -------------------------------------------------------------------------<br>> This SF.net email is sponsored by DB2 Express<br>> Download DB2 Express C - the FREE version of DB2 express and take<br>> control of your XML. No limits. Just data. Click to get it now. <br>> <a href="http://sourceforge.net/powerbar/db2/">http://sourceforge.net/powerbar/db2/</a><br>> _______________________________________________<br>> openMosix-devel mailing list<br>> <a href="mailto:openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org"> openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org</a><br>> <a href="https://lists.sourceforge.net/lists/listinfo/openmosix-devel">https://lists.sourceforge.net/lists/listinfo/openmosix-devel</a><br>><br>><br></blockquote></div> <br><br clear="all"><br>-- <br>Wes Wagner<br><br>Join a libertarian network of person-to-person lending on Prosper:<br> <br><a href="http://www.prosper.com/groups/group_home.aspx?group_short_name=freelibertarians&referrer=AiriusTorpora&utm_source=referrer-AiriusTorpora&utm_medium=referral-link&utm_content=join_my_group-160x33&utm_campaign=referrals-group"> http://www.prosper.com/groups/group_home.aspx?group_short_name=freelibertarians&referrer=AiriusTorpora&utm_source=referrer-AiriusTorpora&utm_medium=referral-link&utm_content=join_my_group-160x33&utm_campaign=referrals-group </a> ------=_Part_53448_25932886.1184768662188-- --===============0470836023== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ --===============0470836023== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ openMosix-general mailing list openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org https://lists.sourceforge.net/lists/listinfo/openmosix-general --===============0470836023==--