Re: BitTorrent for input file staging on clusters?

> 
> 
> 
> Hi, I'm playing with BitTorrent as a method for staging large input
> files to each node of a compute cluster prior to running a distributed
> job. 
> 
> I was using scp to pull the input file(s) from the job submission node
> to each cluster node, but 40 nodes each pulling 1-4GB at the same time
> via scp just isn't workable. As I started thinking about how to do it
> better (chunks, download from peers, checksums), the solution I came
> up with sounded more and more like BitTorrent.
> 
> I've been hacking at the python BitTorrent tracker server and download
> clients to make them more suitable.
> 
> In addition to the patches for track.py from Denis Ahrens, I've had to
> add a small number of features to make it suitable for this use case. 
> 
> * Connect only from IP's matching a regular expression
>   - in RawServer, for both the tracker and downloader
> * Reparse allowed torrents directory on demand:
>   - adding http://tracker/refresh_allowed
> * unescape URL character codes in tracker urls
>   - permits http://tracker/file%3Finfo_hash=... for easier scripting
> * downloader exit once file(s) is downloaded
>   - terminate once idle (download _and_ upload) for a while
> * numerous parameter tweaks to match this use case
>   - e.g. uploader client running on submission server accepts only 
>     one connection
> 
> Is anyone else working on similar applications of BitTorrent?

My intended applications are not too dissimilar, basically LAN based
applications. Aiming for 1Gb+ speeds for multiple clients. Just in the 
process of drastically breaking the client to rewrite a load of stuff
but hope to have it fixed again soon.

I wouldnt bother about limiting IPs (just filter connections if you need to).
And I am not sure you need to exit once idle - if there is nothing to do there
should be no cpu load anyway.

justin


------------------------ Yahoo! Groups Sponsor --------------------~--> 
Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar.
Now with Pop-Up Blocker. Get it for free!
http://us.click.yahoo.com/L5YrjA/eSIIAA/yQLSAA/dkFolB/TM
--------------------------------------------------------------------~-> 

 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/BitTorrent/

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/