Re: End-of-line or just carriage-return
"Wes Wagner" <[email protected]> Wed, 18 Jul 2007 11:13:57 -0700
| Newsgroups | gmane.linux.cluster.openmosix.devel |
|---|---|
| Message-ID | <[email protected]> |
--===============1919912589== Content-Type: multipart/alternative; boundary="----=_Part_56101_26092380.1184782437805" ------=_Part_56101_26092380.1184782437805 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline I live in Portland, Oregon and if you are willing to discuss a continuation of the open mosix project in another form I would be very happy to meet with you and anyone else here who is interested. -Wes Wagner On 7/18/07, Jonathan Day <imipak-/[email protected]> wrote: > > First off, I'd like to thank Moshe Bar and all the > other programmers involved in openMosix - I firmly > believe that it is the best of the SSI solutions out > there at present. > > I also believe that SSI is going to be more important, > not less, on the next generation of machines. The next > version of PCI Express supports multiple masters on a > single network of PCI switches. However, QoS seems to > be limited to bandwidth reservation for a master to be > established at setup. There is no way of dynamically > configuring the next generation PCI-e switches > according to varying demands. > > How does this impact SSI? Well, if you can't move the > allocation, move the process. If you can migrate > between masters, you can load-balance a PCI-e system > in a way that the hardware does not support (by > design). > > There is also the case that programs are usually > smaller than the data sets being manipulated. In a > general-purpose cluster, it is sometimes the case that > shifting the program image is vastly less expensive > than shifting the data. SIMD would seem to be the > perfect market for this, because you're guaranteed to > be running the same code on all nodes, but each node > will have its own data - which it would therefore need > to collect, rather than having a central collection > point. > > DSM is a problem - and a problem I have worked hard on > for the last few years. Part of the problem is that > when you copy to N nodes using a unicast delivery > system, you must transmit the data N times. That is > slow. NACK-Oriented Reliable Multicast would reduce > the copies needed, cutting bandwidth requirements and > the total elapsed time between start of first packet > to end of last packet. > > RDMA (Remote Direct Memory Access) is also an > interesting technology, allowing one node to copy from > local memory to remote memory, or remote memory to > local memory, in a single operation without > intervention by the kernel on either side. However, > most machines are not provided with RNICs or > Infiniband cards. Pity. > > A project of mine has been to extend RDMA to support > multicasting - something the specs don't currently > allow for - so that you can transfer to multiple > memories in one go. Devising a protocol that the RDMA > groups can sign on to is proving difficult. There are > a large number of corner cases and the more I fix, the > more I find. If RDMA proves impossible, then any > zero-copy kernel-bypass protocol would work in > principle. This would avoid having specialized > hardware but you'd need an emulation layer to provide > compatibility to the bulk of interesting software. > > Multicast with zero-copy does not totally solve the > problem of DSM, but it reduces some of the issues. You > shed a whole bunch of context switches, stack > latencies, buffer copies, backoffs caused by switch > flooding, etc. It's hard to quantify, but my > calculations repeatedly show that the latency is > roughly 5.6 microseconds plus the per hop latency on > the longest path for every 4K, regardless of the > number of nodes actually delivered to. > > Another networking technology I have been taking a > close look at is Active Messaging. There's a project - > GAMMA - that implements AM on Linux. Again, the idea > is to reduce the wasted cycles involved. AM looks very > interesting, but the implementation seems to be very > card-specific and doesn't play nice with other > protocols. To use AM in a general-purpose cluster > would need a fair amount of work put into it. > > TIPC is also worthy of examination, as anything that > reduces what a clustering solution needs to do to > cluster transparently is a Good Thing. > > Ok, I've now completely flogged, beaten and trampled > this poor dead horse into the ground, I'll mutilate a > few other things, the first of which is the SSI > concept. > > Strictly speaking, openMosix does not need to be SSI. > Indeed, it would make some sense to use the IBCS > patches to load and start applications for a range of > Intel-based *nixes and then migrate the process space > to an instance of the actual OS it is intended for. We > now have a multi system image clustering technology. > > The second issue is with real-time support. Real-time, > in this context, means that if process X is guaranteed > a timeslice of Y out of a total timeslice of Z, with a > margin of error M, then no matter how time is divided > up, how busy/free the system is, whether the system > voluntarily hands back control, etc, it WILL occupy > that amount of the available time, give or take only > the margin of error, nothing more. > > This can be fun when clustering. You don't hear of > hard real-time clusters for a reason, and the reason > has nothing to do with a lack of need. What does > real-time mean for openMosix? It means that where such > state information exists, openMosix needs to copy it. > It means that processes need to migrate according to a > packing algorithm that maximizes the chances of always > meeting the timeslice goals. If real-time networking > is also being used, then you also need the packing > algorithm to consider network latencies. > > Real-time does not mean the fastest solution, which is > what most SSI strategies look for. Nearest neighbors > and all that. You will occupy a certain length of time > even if the transfer takes less, so you must place at > the greatest distance that the transfers will still > meet deadlines even under the greatest-expected > network loads. > > Last, but by no means least, if the project is to go > unmaintained, does anyone object to it being listed on > the Unmaintained Free Projects pages? > > Jonathan > > P.S. Ok, I lied, that wasn't last, this is. If there > are people from this list going to the Open Source > conference in Portland, Oregon, next week, I suggest > we arrange some sort of mini meetup. Accepting that > Moshe has found nobody with the time and ability to > run the project at this time, I propose that those of > us who know gifted coders work on finding some, and > that those who have contacts in the moneyed world see > if the project can be sponsored somehow to the point > where a team of developers can hack it to perfection > on a viable salaried basis. > > > > > > ____________________________________________________________________________________ > Need a vacation? Get great deals > to amazing places on Yahoo! Travel. > http://travel.yahoo.com/ > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > openMosix-devel mailing list > openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org > https://lists.sourceforge.net/lists/listinfo/openmosix-devel > -- Wes Wagner Join a libertarian network of person-to-person lending on Prosper: http://www.prosper.com/groups/group_home.aspx?group_short_name=freelibertarians&referrer=AiriusTorpora&utm_source=referrer-AiriusTorpora&utm_medium=referral-link&utm_content=join_my_group-160x33&utm_campaign=referrals-group ------=_Part_56101_26092380.1184782437805 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline I live in Portland, Oregon and if you are willing to discuss a continuation of the open mosix project in another form I would be very happy to meet with you and anyone else here who is interested.<br><br>-Wes Wagner<br><br> <br><div><span class="gmail_quote">On 7/18/07, <b class="gmail_sendername">Jonathan Day</b> <<a href="mailto:imipak-/[email protected]">imipak-/[email protected]</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> First off, I'd like to thank Moshe Bar and all the<br>other programmers involved in openMosix - I firmly<br>believe that it is the best of the SSI solutions out<br>there at present.<br><br>I also believe that SSI is going to be more important, <br>not less, on the next generation of machines. The next<br>version of PCI Express supports multiple masters on a<br>single network of PCI switches. However, QoS seems to<br>be limited to bandwidth reservation for a master to be <br>established at setup. There is no way of dynamically<br>configuring the next generation PCI-e switches<br>according to varying demands.<br><br>How does this impact SSI? Well, if you can't move the<br>allocation, move the process. If you can migrate <br>between masters, you can load-balance a PCI-e system<br>in a way that the hardware does not support (by<br>design).<br><br>There is also the case that programs are usually<br>smaller than the data sets being manipulated. In a <br>general-purpose cluster, it is sometimes the case that<br>shifting the program image is vastly less expensive<br>than shifting the data. SIMD would seem to be the<br>perfect market for this, because you're guaranteed to <br>be running the same code on all nodes, but each node<br>will have its own data - which it would therefore need<br>to collect, rather than having a central collection<br>point.<br><br>DSM is a problem - and a problem I have worked hard on <br>for the last few years. Part of the problem is that<br>when you copy to N nodes using a unicast delivery<br>system, you must transmit the data N times. That is<br>slow. NACK-Oriented Reliable Multicast would reduce<br> the copies needed, cutting bandwidth requirements and<br>the total elapsed time between start of first packet<br>to end of last packet.<br><br>RDMA (Remote Direct Memory Access) is also an<br>interesting technology, allowing one node to copy from <br>local memory to remote memory, or remote memory to<br>local memory, in a single operation without<br>intervention by the kernel on either side. However,<br>most machines are not provided with RNICs or<br>Infiniband cards. Pity. <br><br>A project of mine has been to extend RDMA to support<br>multicasting - something the specs don't currently<br>allow for - so that you can transfer to multiple<br>memories in one go. Devising a protocol that the RDMA <br>groups can sign on to is proving difficult. There are<br>a large number of corner cases and the more I fix, the<br>more I find. If RDMA proves impossible, then any<br>zero-copy kernel-bypass protocol would work in<br> principle. This would avoid having specialized<br>hardware but you'd need an emulation layer to provide<br>compatibility to the bulk of interesting software.<br><br>Multicast with zero-copy does not totally solve the<br> problem of DSM, but it reduces some of the issues. You<br>shed a whole bunch of context switches, stack<br>latencies, buffer copies, backoffs caused by switch<br>flooding, etc. It's hard to quantify, but my<br>calculations repeatedly show that the latency is <br>roughly 5.6 microseconds plus the per hop latency on<br>the longest path for every 4K, regardless of the<br>number of nodes actually delivered to.<br><br>Another networking technology I have been taking a<br>close look at is Active Messaging. There's a project - <br>GAMMA - that implements AM on Linux. Again, the idea<br>is to reduce the wasted cycles involved. AM looks very<br>interesting, but the implementation seems to be very<br>card-specific and doesn't play nice with other <br>protocols. To use AM in a general-purpose cluster<br>would need a fair amount of work put into it.<br><br>TIPC is also worthy of examination, as anything that<br>reduces what a clustering solution needs to do to<br>cluster transparently is a Good Thing. <br><br>Ok, I've now completely flogged, beaten and trampled<br>this poor dead horse into the ground, I'll mutilate a<br>few other things, the first of which is the SSI<br>concept.<br><br>Strictly speaking, openMosix does not need to be SSI. <br>Indeed, it would make some sense to use the IBCS<br>patches to load and start applications for a range of<br>Intel-based *nixes and then migrate the process space<br>to an instance of the actual OS it is intended for. We <br>now have a multi system image clustering technology.<br><br>The second issue is with real-time support. Real-time,<br>in this context, means that if process X is guaranteed<br>a timeslice of Y out of a total timeslice of Z, with a <br>margin of error M, then no matter how time is divided<br>up, how busy/free the system is, whether the system<br>voluntarily hands back control, etc, it WILL occupy<br>that amount of the available time, give or take only <br>the margin of error, nothing more.<br><br>This can be fun when clustering. You don't hear of<br>hard real-time clusters for a reason, and the reason<br>has nothing to do with a lack of need. What does<br>real-time mean for openMosix? It means that where such <br>state information exists, openMosix needs to copy it.<br>It means that processes need to migrate according to a<br>packing algorithm that maximizes the chances of always<br>meeting the timeslice goals. If real-time networking <br>is also being used, then you also need the packing<br>algorithm to consider network latencies.<br><br>Real-time does not mean the fastest solution, which is<br>what most SSI strategies look for. Nearest neighbors<br>and all that. You will occupy a certain length of time <br>even if the transfer takes less, so you must place at<br>the greatest distance that the transfers will still<br>meet deadlines even under the greatest-expected<br>network loads.<br><br>Last, but by no means least, if the project is to go <br>unmaintained, does anyone object to it being listed on<br>the Unmaintained Free Projects pages?<br><br>Jonathan<br><br>P.S. Ok, I lied, that wasn't last, this is. If there<br>are people from this list going to the Open Source <br>conference in Portland, Oregon, next week, I suggest<br>we arrange some sort of mini meetup. Accepting that<br>Moshe has found nobody with the time and ability to<br>run the project at this time, I propose that those of <br>us who know gifted coders work on finding some, and<br>that those who have contacts in the moneyed world see<br>if the project can be sponsored somehow to the point<br>where a team of developers can hack it to perfection <br>on a viable salaried basis.<br><br><br><br><br>____________________________________________________________________________________<br>Need a vacation? Get great deals<br>to amazing places on Yahoo! Travel.<br><a href="http://travel.yahoo.com/"> http://travel.yahoo.com/</a><br><br>-------------------------------------------------------------------------<br>This SF.net email is sponsored by DB2 Express<br>Download DB2 Express C - the FREE version of DB2 express and take <br>control of your XML. No limits. Just data. Click to get it now.<br><a href="http://sourceforge.net/powerbar/db2/">http://sourceforge.net/powerbar/db2/</a><br>_______________________________________________<br>openMosix-devel mailing list <br><a href="mailto:openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org">openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org</a><br><a href="https://lists.sourceforge.net/lists/listinfo/openmosix-devel">https://lists.sourceforge.net/lists/listinfo/openmosix-devel </a><br></blockquote></div><br><br clear="all"><br>-- <br>Wes Wagner<br><br>Join a libertarian network of person-to-person lending on Prosper:<br> <br><a href="http://www.prosper.com/groups/group_home.aspx?group_short_name=freelibertarians&referrer=AiriusTorpora&utm_source=referrer-AiriusTorpora&utm_medium=referral-link&utm_content=join_my_group-160x33&utm_campaign=referrals-group"> http://www.prosper.com/groups/group_home.aspx?group_short_name=freelibertarians&referrer=AiriusTorpora&utm_source=referrer-AiriusTorpora&utm_medium=referral-link&utm_content=join_my_group-160x33&utm_campaign=referrals-group </a> ------=_Part_56101_26092380.1184782437805-- --===============1919912589== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ --===============1919912589== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ openMosix-devel mailing list openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org https://lists.sourceforge.net/lists/listinfo/openmosix-devel --===============1919912589==--