BitTorrent for input file staging on clusters?
"Nathan Edwards" <[email protected]>
| Newsgroups | gmane.network.bit-torrent.general |
|---|---|
| Message-ID | <[email protected]> |
Hi, I'm playing with BitTorrent as a method for staging large input
files to each node of a compute cluster prior to running a distributed
job.
I was using scp to pull the input file(s) from the job submission node
to each cluster node, but 40 nodes each pulling 1-4GB at the same time
via scp just isn't workable. As I started thinking about how to do it
better (chunks, download from peers, checksums), the solution I came
up with sounded more and more like BitTorrent.
I've been hacking at the python BitTorrent tracker server and download
clients to make them more suitable.
In addition to the patches for track.py from Denis Ahrens, I've had to
add a small number of features to make it suitable for this use case.
* Connect only from IP's matching a regular expression
- in RawServer, for both the tracker and downloader
* Reparse allowed torrents directory on demand:
- adding http://tracker/refresh_allowed
* unescape URL character codes in tracker urls
- permits http://tracker/file%3Finfo_hash=... for easier scripting
* downloader exit once file(s) is downloaded
- terminate once idle (download _and_ upload) for a while
* numerous parameter tweaks to match this use case
- e.g. uploader client running on submission server accepts only
one connection
Is anyone else working on similar applications of BitTorrent?
Thanks,
nathan
---
Nathan Edwards, Ph.D.
Center for Bioinformatics and Computational Biology
University of Maryland, College Park, MD 20742-3360
Email: nedwards at umiacs.umd.edu
------------------------ Yahoo! Groups Sponsor --------------------~-->
Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar.
Now with Pop-Up Blocker. Get it for free!
http://us.click.yahoo.com/L5YrjA/eSIIAA/yQLSAA/dkFolB/TM
--------------------------------------------------------------------~->
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/BitTorrent/
<*> To unsubscribe from this group, send an email to:
[email protected]
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/