Re: Re: BitTorrent for input file staging on clusters?

> 
> 
> 
> --- In [email protected], Jesus Cea <jcea@a...> wrote:
> > Nathan Edwards wrote:
> > > 
> > > Hi, I'm playing with BitTorrent as a method for staging large input
> > > files to each node of a compute cluster prior to running a distributed
> > > job. 
> > 
> > If you are in a LAN, I rather suggest using a multicast distribution 
> > procedure.
> 
> Writing a full on MPI application (or similar) just for file staging
> feels like overkill for this problem.
> 
> In fact, isn't BT just a dynamic topology multicast? Consider any
> chunk of the file. A node only requests it from someone if they don't
> have it,  and doesn't forward it until it has it, and only to those
> who don't already have it. The path defined by any single chunk must
> therefore be a tree. That each chunk's tree is (potentially) different
> and that the chunks arrive in an arbitrary order and via different
> paths give a measure of fault tolerance and load balancing. 
> 
> Am I missing something?

I agree actually. Multicast is pretty difficult, as you need a retry
mechanism anyway, and udp is pretty hard to get to perform well. Also
almost all LAN switches are nonblocking or close to that so peer to peer
traffic is more or less costless. And you can let machines get slightly out
of sync with a bt protocol, while multicast is entirely synchronous.

Clearly performance analysis would be helpful but I dont think multicast
is that clearcut a winner.

Justin

 
> Cheers!
> 
> nathan
> 
> 
> 
> 
> 
> 
>  
> Yahoo! Groups Links
> 
> 
> 
>  
> 
> 
> 



------------------------ Yahoo! Groups Sponsor --------------------~--> 
Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar.
Now with Pop-Up Blocker. Get it for free!
http://us.click.yahoo.com/L5YrjA/eSIIAA/yQLSAA/dkFolB/TM
--------------------------------------------------------------------~-> 

 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/BitTorrent/

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/