Re: BitTorrent for input file staging on clusters?

--- In [email protected], Justin Cormack <justin@s...> wrote:

> My intended applications are not too dissimilar, basically LAN based
> applications. Aiming for 1Gb+ speeds for multiple clients. Just in the 
> process of drastically breaking the client to rewrite a load of stuff
> but hope to have it fixed again soon.
> 
> I wouldnt bother about limiting IPs (just filter connections if you
need to).

At the firewall level? I don't run that side of the shop, but if I
hope to enforce the IP address thing, then I clearly need a firewall
to guard against IP spoofing. Since the tracker (and the clients) run
on somewhat arbitrary non-priviledged ports, creating a firewall rule
for it is going to be tricky and prone to cause problems with other
things.

> And I am not sure you need to exit once idle - if there is nothing
to do there
> should be no cpu load anyway.

Either the job that is waiting for the file needs to poll to find out
when the file is available, or something must trigger it. My solution
was to make a client that exits when it is done (just like scp does)
with a status code that indicates success or failure (just like scp).
The job that requires the file can then proceed with a guarantee that
the file is complete and correct (or has an error code to indicate
that this isn't the case).

Thanks for your comments!

nathan





------------------------ Yahoo! Groups Sponsor --------------------~--> 
Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar.
Now with Pop-Up Blocker. Get it for free!
http://us.click.yahoo.com/L5YrjA/eSIIAA/yQLSAA/dkFolB/TM
--------------------------------------------------------------------~-> 

 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/BitTorrent/

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/