Re: Recommendations
"Ian Latter" <[email protected]> Thu, 26 Jul 2007 15:24:09 +1000
| Newsgroups | gmane.linux.cluster.openmosix.general |
|---|---|
| Message-ID | <[email protected]> |
--===============0538389116==
Content-type: text/plain; charset="us-ascii"
To respond to my own post - this is the sort of thing I am talking
about regarding inevitability (Mr Anderson);
Intel Releases Threading Library Under GPL 2
Posted by CmdrTaco on Wednesday July 25, @10:02AM
from the of-interest-to-some-of-you dept.
http://developers.slashdot.org/developers/07/07/25/1324221.shtml
littlefoo writes
"Intel Software Dispatch have announced the availability of
the Threading Building Blocks (TBB) template library under the
GPL v2 with the run-time exception — so this previously
commercial only package is now open for all the use,
whether for open-source projects or commercial offerings
(although they are explicitly encouraging open source use).
The interface is more task-based then thread-based, but with
a somewhat different view of things than, e.g. OpenMP. From
the Intel release: 'Intel® Threading Building Blocks (TBB) offers
a rich and complete approach to expressing parallelism in a
C++ program. It is a library that helps you leverage multi-core
processor performance without having to be a threading
expert. Threading Building Blocks is not just a threads-
replacement library. It represents a higher-level, task-based
parallelism that abstracts platform details and threading
mechanism for performance and scalability.'"
[http://threadingbuildingblocks.org/]
----- Original Message -----
>From: "Ian Latter" <[email protected]>
>To: "Moshe Bar" <[email protected]>
>Subject: Re: [openMosix-general] Recommendations
>Date: Tue, 24 Jul 2007 18:29:26 +1000
>
>
> Perhaps you're showing your age, and perhaps I'm showing mine ... ;o)
>
>
> >From your remarks, I'll break my comments out into Future, Now and
> Effort;
>
> > It is tempting to think that with massive multi-cores, the advantage
> > of an SSI cluster stays because then you have "even more CPUs to play
> > with". That is not so. The migration from a multi-core system where
> > multi-threaded applications (as all apps are currently being
> > re-written as) thrive and performe well to another such node, implies
> > a DSM model capable to relocate threads either in gangs or single
> > across nodes. In other words multi-cores are changing the application
> > landscape towards a massively threaded one, and these apps will not
> > perform well in a DSM cluster by definition due to very high true and
> > false sharing.
> >
> > Tests we performed at my company Qlusters with very low latency
> > inter-connects (< 1microsecond round-trip latency) with a very
> > efficient implementation of DSM have shown an regular performance
> > impacts of 100x to 1000x.
> >
> > Even the sgi Altix let's threads run in the same SMP quadrant.
> >
>
>
> Future;
>
> I wrote a C implementation of the Bootp/DHCP RFCs two weeks ago,
> for libMidnightCode. In those RFC documents, that range from 20 to
> 10years ago, the standards' authors explicitly permitted Bootp agents
> to create UDP/IP packets with invalid checksums. The reason they did
> this was that they felt that it would be infeasible to calculate a valid
> CRC given the size and speed of "current" firmware and embedded
> processors.
> The same was true of the 25 year-old SMTP protocol - authored to
> allow parsing as characters were received because memory was
> seen as too rare or too precious to house an entire SMTP message in
> a single buffer, before it was parsed.
> Of course, I can now buy a 4Gbyte USB2.0 300+kbps MP3 playing
> wrist-watch, for AU$95.
>
> To me, technology and time are an inevitable duet. What's impossible
> today is milspec tomorrow, enterprise next week, and child's play by
> months end. This is the by-product, the waste, of the information age.
>
> At the desktop, there is sufficient industry demand to drive Terabyte
> disks and beyond. There is equally sufficient industry demand to drive
> multi-core processors, and beyond, even though the industry leader
> tried to drive the industry in other ways;
>
> http://www.theregister.co.uk/2006/06/08/intel_gelsinger_stanford/
> June, 2006
>
> [...]
>
> "A couple of years ago, I had a discussion with Bill Gates (about
> the multi-core products)," Gelsinger said. "He was just in disbelief.
> He said, 'We can't write software to keep up with that.'"
>
> Gates called for Intel to just keep making its chips faster. "No, Bill,
> it's not going to work that way," Gelsinger informed him.
>
> [...]
>
>
>
> Though despite Bill's comments to Intel, Microsoft are well aware of
> this cascading technology phenomenon too;
>
> http://www.hpcwire.com/hpc/1347210.html
> April, 2007
>
> [...]
>
> David Callahan (Microsoft Research) pointed out that exponential
> grows really fast. If we plan on doubling the number of cores on
> a chip every year or 18 months, it won't be long before we have
> hundreds or thousands of cores on a chip. Any software solution
> aimed at 16 or 32 cores will quickly become irrelevant. We'd better
> be looking at productive ways to use massively parallel systems,
> since these may well find their way into our workstations, and
> yes, even our laptops, before the end of the next decade.
>
> [...]
>
>
>
> The PPoPP (Principles and Practice of Parallel Programming) group
> have been working on this problem;
>
> http://www.hpcwire.com/hpc/1347210.html
> April, 2007
>
> [...]
>
> One of the PPoPP attendees, Prof. Rudolf Eigenmann (Purdue
> Univ.) issued an indictment, saying that we in the parallel
> programming research community should be ashamed of
> ourselves. Single-processor systems have run out of steam,
> something the parallel programming community has been
> predicting since I was a college student. Now is the time to step
> up and reap the benefits of all our past work. We've had 30
> years to study this problem and come up with a solution, but
> what's the end result? Surprise! We still have no well-accepted
> method to generate parallel applications.
>
> [...]
>
>
> And despite no solid answer in 30 years, I know this problem will
> be solved - it is inevitable (I'd put 10 bucks on it being built into a
> wrist watch one day - except I doubt either of us will be here to
> cash in on that bet).
>
>
> But that wasn't the only limitation you proposed.
>
> I do realise that your latency comments are directed to those you
> see flogging a dead architecture - but this is an evolving
> architecture; as those of us with incompatible CPU's, RAM,
> expansion cards, disks, power supplies, cases, screens, keyboards
> (and even bloody mice), can testify.
>
> It is inevitable that the bus speed through to the network speed will
> increase ... and that your latency gap will decrease over time.
>
>
>
> Now;
>
> However, I understand that this project is not academic (the future
> is too far away for those with power problems now). And here I
> see openMosix in two lights;
>
> - it is an alternative High Performacne Cluster solution, for those
> shopping around to make their out-of-the-box software run
> faster.
>
> - it is an alternative High Performance Cluster solution, for those
> choosing to write a new application, and to make it run faster
> by using multiple devices.
>
>
> From the tens of thousands (approaching hundreds of thousands)
> of downloads that I've seen of CHAOS, over the past couple of
> years, I'd suggest that openMosix fills a non-academic role, in those
> two ways, well.
>
>
>
> Effort;
>
> So, to my mind, there's no need for openMosix to justify its
> existence.
>
> Though there is still cause to see openMosix terminated while there
> is insufficient effort available to drive the project. I am in no way
> bagging out Florian. The problem is that there is only one Florian.
>
> I'm not naive enough to believe that your public comments and press
> release (all suggesting the future shutdown of openMosix) are
> anything less than a witty recruiting call; I very much hope you
> succeed.
>
>
> However, as neither of us are offering to write the code, and as
> you still seem intent on leaving the project, I'd suggest the
> programmers be given the real vote .. as it is their baby ...
>
>
>
>
> To whomever does drive this ship, I'm still here, awating a current
> kernel version to deploy as an improved CHAOS version.
>
> My reference platform runs much better than the previous CHAOS'
> did, but does not yet support all of the old features.
>
> I will continue to chip away at libMidnightCode - to incorporate better
> functionality (with higher quality code) into the platform. For example;
> There is a grealty improved multi-threaded peer-based
> communications protocol implemented there; waiting to replace Tyd
> when the time is due.
>
>
>
>
>
>
> ----- Original Message -----
> >From: "Moshe Bar" <[email protected]>
> >To: "Wes Wagner" <[email protected]>
> >Subject: Re: [Openmosix-Devel] Recommendations
> >Date: Wed, 18 Jul 2007 15:14:45 -0500
> >
> > Paul, Wes, all
> >
> > I actually meant to write to the list before the week-end, but this is
> > a good opportunity.
> >
> > It's great to see that a few people have jumped into action (incl.
> > Florian) to try to finish the port of oM to 2.6.
> >
> > As to the reasons for my decision:
> >
> > It is true that time and resource constraints have played a role, but
> > only a minor one. After all, I have been able to keep the project
> > going while I started no less than 5 companies and keep them all
> > running, am involved with several others, wrote two books, and did
> > several of the other items on my list "things to do before I die".
> >
> > The real reasons for my decisions are indeed tied more to the
> > long-term vision and feasibility of the project. openMosix is about
> > high performance computing, in other words let CPU (and to some
> > extent) RAM intensive apps run as fast as possible without hindrance
> > from the limitation of resources on local computers. We have all seen
> > that there are severe limitations to the SSI model due to the extreme
> > high impact of network latency, and due to the infeasible cost of
> > fully virtualizing the memory model (ie DSM), and the OS control block
> > across several nodes.
> >
> > It is tempting to think that with massive multi-cores, the advantage
> > of an SSI cluster stays because then you have "even more CPUs to play
> > with". That is not so. The migration from a multi-core system where
> > multi-threaded applications (as all apps are currently being
> > re-written as) thrive and performe well to another such node, implies
> > a DSM model capable to relocate threads either in gangs or single
> > across nodes. In other words multi-cores are changing the application
> > landscape towards a massively threaded one, and these apps will not
> > perform well in a DSM cluster by definition due to very high true and
> > false sharing.
> >
> > Tests we performed at my company Qlusters with very low latency
> > inter-connects (< 1microsecond round-trip latency) with a very
> > efficient implementation of DSM have shown an regular performance
> > impacts of 100x to 1000x.
> >
> > Even the sgi Altix let's threads run in the same SMP quadrant.
> >
> >
> > The other big reason is that I have over the last two years
> > intensively looked for a successor and was not able to find somebody
> > with the willingness, stamina, maturity, leadership, and coding
> > expertise to take over.
> >
> > If this announcement brings forward somebody to deal with this latter
> > issue, that doesn't solve the prior issue of multi-core application
> > models.
> >
> > Finally, another reason is that user numbers have been going down very
> > quickly and mostly these were users with, say, 4 or 5 nodes with a
> > total of 4 to 6 CPUs. Well, you can get all of that with much simpler
> > ease of use with a dual socket, quad core system today.
> >
> > Please hammer away with comments. :-)
> >
> > Moshe
> >
> >
> >
> > On 7/18/07, Wes Wagner <[email protected]> wrote:
> > > Paul,
> > >
> > > I am working with the author of omuscd on a 2.6 openmosix port to the 2.6.22
> > > Kernel... I need to talk with him some more, but you should expect that the
> > > openmosix cluster will be picked up by another team.
> > >
> > > I just need until this weekend to get things organized. I believe Moshe
> > > underestimates the need for using multicore systems in a massive set and
> > > forget mode so people can do without the overhead of being concerned with
> > > queue and batch management.
> > >
> > > We have such a need and are currently building a 40 core farm expected to
> > > grow to the thousands of cores. It is my personal opinion that this
> > > announcement came mostly due to time and resource constraints, not the lack
> > > of a need for SSI style clustering.
> > >
> > > Please watch this space in the coming weeks:
> > >
> > > https://sourceforge.net/projects/om-2-6
> > >
> > > Our intent was originally to provide a source tree where you could download
> > > an x386 x686 x64_86 source that included the omuscd management daemon and
> > > went clean onto a vanilla kernel. I expect that the project will expand.
> > >
> > > Sincerely,
> > > Wes Wagner
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 7/18/07, Paul Bennett <[email protected] > wrote:
> > > >
> > > > Hi List,
> > > >
> > > > I just found out that openMosix was getting end-of-lifed. Does anyone
> > > have any recommendations for another clustering option?
> > > >
> > > > I would like to go with something based on the 2.6 kernel and amd_64.
> > > >
> > > > http://sourceforge.net/forum/forum.php?forum_id=715406
> > > >
> > > > Thanks,
> > > >
> > > > Paul
> > > >
> > > >
> > > -------------------------------------------------------------------------
> > > > This SF.net email is sponsored by DB2 Express
> > > > Download DB2 Express C - the FREE version of DB2 express and take
> > > > control of your XML. No limits. Just data. Click to get it now.
> > > > http://sourceforge.net/powerbar/db2/
> > > > _______________________________________________
> > > > openMosix-devel mailing list
> > > > openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> > > >
> > > https://lists.sourceforge.net/lists/listinfo/openmosix-devel
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Wes Wagner
> > >
> > > Join a libertarian network of person-to-person lending on Prosper:
> > >
> > > http://www.prosper.com/groups/group_home.aspx?
> group_short_name=freelibertarians&referrer=AiriusTorpora&utm_source=referrer-
> AiriusTorpora&utm_medium=referral-link&utm_content=join_my_group-
> 160x33&utm_campaign=referrals-group
> > > -------------------------------------------------------------------------
> > > This SF.net email is sponsored by DB2 Express
> > > Download DB2 Express C - the FREE version of DB2 express and take
> > > control of your XML. No limits. Just data. Click to get it now.
> > > http://sourceforge.net/powerbar/db2/
> > > _______________________________________________
> > > openMosix-devel mailing list
> > > openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> > > https://lists.sourceforge.net/lists/listinfo/openmosix-devel
> > >
> > >
> >
> > -------------------------------------------------------------------------
> > This SF.net email is sponsored by DB2 Express
> > Download DB2 Express C - the FREE version of DB2 express and take
> > control of your XML. No limits. Just data. Click to get it now.
> > http://sourceforge.net/powerbar/db2/
> > _______________________________________________
> > openMosix-devel mailing list
> > openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> > https://lists.sourceforge.net/lists/listinfo/openmosix-devel
> >
>
>
> --
> Ian Latter
> Late night coder ..
> http://midnightcode.org/
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems? Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> openMosix-general mailing list
> openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/openmosix-general
>
--
Ian Latter
Late night coder ..
http://midnightcode.org/
--===============0538389116==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
--===============0538389116==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
openMosix-general mailing list
openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/openmosix-general
--===============0538389116==--