Re: BitTorrent for input file staging on clusters?
"Nathan Edwards" <[email protected]>
| Newsgroups | gmane.network.bit-torrent.general |
|---|---|
| Message-ID | <[email protected]> |
--- In [email protected], Jesus Cea <jcea@a...> wrote: > Nathan Edwards wrote: > > > > Hi, I'm playing with BitTorrent as a method for staging large input > > files to each node of a compute cluster prior to running a distributed > > job. > > If you are in a LAN, I rather suggest using a multicast distribution > procedure. Writing a full on MPI application (or similar) just for file staging feels like overkill for this problem. In fact, isn't BT just a dynamic topology multicast? Consider any chunk of the file. A node only requests it from someone if they don't have it, and doesn't forward it until it has it, and only to those who don't already have it. The path defined by any single chunk must therefore be a tree. That each chunk's tree is (potentially) different and that the chunks arrive in an arbitrary order and via different paths give a measure of fault tolerance and load balancing. Am I missing something? Cheers! nathan ------------------------ Yahoo! Groups Sponsor --------------------~--> Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar. Now with Pop-Up Blocker. Get it for free! http://us.click.yahoo.com/L5YrjA/eSIIAA/yQLSAA/dkFolB/TM --------------------------------------------------------------------~-> Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/BitTorrent/ <*> To unsubscribe from this group, send an email to: [email protected] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/