Re: Re: BitTorrent for input file staging on clusters?
"Chris Kim" <[email protected]>
| Newsgroups | gmane.network.bit-torrent.general |
|---|---|
| Message-ID | <001901c4d8fc$4523f880$0401000a@chris> |
please unsubscribe from this list... Thankyou ----- Original Message ----- From: Nathan Edwards To: [email protected] Sent: Thursday, December 02, 2004 7:12 AM Subject: [BitTorrent] Re: BitTorrent for input file staging on clusters? --- In [email protected], Jesus Cea <jcea@a...> wrote: > Nathan Edwards wrote: > > > > Hi, I'm playing with BitTorrent as a method for staging large input > > files to each node of a compute cluster prior to running a distributed > > job. > > If you are in a LAN, I rather suggest using a multicast distribution > procedure. Writing a full on MPI application (or similar) just for file staging feels like overkill for this problem. In fact, isn't BT just a dynamic topology multicast? Consider any chunk of the file. A node only requests it from someone if they don't have it, and doesn't forward it until it has it, and only to those who don't already have it. The path defined by any single chunk must therefore be a tree. That each chunk's tree is (potentially) different and that the chunks arrive in an arbitrary order and via different paths give a measure of fault tolerance and load balancing. Am I missing something? Cheers! nathan Yahoo! Groups Sponsor ADVERTISEMENT ------------------------------------------------------------------------------ Yahoo! Groups Links a.. To visit your group on the web, go to: http://groups.yahoo.com/group/BitTorrent/ b.. To unsubscribe from this group, send an email to: [email protected] c.. Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service. [Non-text portions of this message have been removed] ------------------------ Yahoo! Groups Sponsor --------------------~--> $9.95 domain names from Yahoo!. Register anything. http://us.click.yahoo.com/J8kdrA/y20IAA/yQLSAA/dkFolB/TM --------------------------------------------------------------------~-> Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/BitTorrent/ <*> To unsubscribe from this group, send an email to: [email protected] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/