RE: returning results for partial downloads

> -----Message d'origine-----
> De : [email protected] [mailto:[email protected]] De la part
> de Zlatin Balevsky
> Envoyé : vendredi 5 octobre 2007 20:22
> À : [email protected]
> Objet : [the_gdf] returning results for partial downloads
> 
> Christian suggested that it would help rare file distribution if
> search results are returned for partial downloads.  After a little
> deliberation, it seems like this idea is very feasible.  The results
> will have to contain the bitprint and the ranges which have been
> successfully verified.  On the receiving side, clients can use the
> range information to determine if the entire file is available over
> the network and use that to decide if the result should be displayed.
> 
> Comments?

I have also suggested this since long (about 3 years ago), it was debated
several times, but not implemented since, most probably because it caused
problems in the reliability of swarmed downloaded.

But in fact this was caused by implementation bugs, causing swarmed
downloads to fail quite often if the same source was changing during the
transfer the set of ranges it currently has for a file and if the range
supported by a source was not contiguous from the start of the file.

But if we want to support fast and reliable swarmed downloads, we must be
able to support downloading from any source not necessarily from the start
of the file, but from any position; so as long as the swarmed download does
not complete, our local file will have as many "holes" in it as the number
of active sources we have used, so our local copy will contain multiple, non
contiguous fragments.

But even in this case, we can still share the fragments we already have
downloaded provided that they have been FULLY checked according to the
fragment of TigerTree data that represents it. This is possible for each
localfile fragment that is FULLY covered by a node in the TigerTree data,
and of course, it requires having this TigerTree data ready for this check.

But the TigerTree datamay be downloaded asynchronously in a separate time
after the first fragments have been downloaded from one source (because that
source may not have TigerTree data for this file, and we need to find
another source for that TigerTree data).

Consequently, we'll have locally some fragments that are still missing, some
fragments that have been downloaded but can't be checked because they are
not fully covered by some matching TigerTree data node, and some fragments
that have been downloaded and that now match with the tigertree data. Only
these last fragments can be shared safely and immediately even if the file
is not complete.

For any fragment that was previously downloaded, but that reveal later to be
corrupted according to the TigerTree data node we have now and that covers
it completely, such fragments must be dropped and reloaded from another
source. But this will continue to work because we have still not shared it.

This scheme works only because file fragments are checked before being
shares. And it benefits not only to rare files (allowing the number of
sources supporting a rare file to augment during the download by multiple
clients that still don't have a complete copy), but for ALL files on general
More generally it will avoid one client to steal ALL the upload bandwidth
from any uploader, because the uploader will be able to distribute its own
workload to lots of clients, if it only gives them a limited fragment size
before queueing again multiple clients with ongoing transfers):

the idea is to have for example N clients with active uploads in progress
plus N (or 2N: this is tuneable) clients temporarily idled in the requeueing
list. All the other newcoming candidate clients are queued with a lower
priority with the existing mechanism. In fact I would militate for an
implementation where each file can only have at most one active upload in
progress to any source, so this upload should be as fast as possible, but
limited to a small fragment size that can be checked immediately by the
receipinet according to TreeData.

Then that client is requeued, and other clients are served, but will be able
to use also the upload resources of the first served clients even if what
they have is partial. Each client that has successfully downloaded a
fragment will need to inform as soon as possible the uploading source so
that it can inform the other candidate clients to the same file that another
supplementary source is available that will help supporting the charge.
Rapidely this system will converge to faster speed for everybody.