Re: returning results for partial downloads

I believe most of these ideas are already implemented in the current
download mesh.  What is missing are the ideas for queueing discipline.
 That is a very interesting topic, and it would be a very exciting
challenge to come up with schemes that can beat bittorrent's
tit-for-tat.

Z

On 10/5/07, Philippe Verdy <[email protected]> wrote:
>
>
>
>
>
>
>
>
>  > -----Message d'origine-----
>  > De : [email protected] [mailto:[email protected]] De la part
>  > de Zlatin Balevsky
>  > Envoyé : vendredi 5 octobre 2007 20:22
>  > À : [email protected]
>  > Objet : [the_gdf] returning results for partial downloads
>
>  >
>  > Christian suggested that it would help rare file distribution if
>  > search results are returned for partial downloads.  After a little
>  > deliberation, it seems like this idea is very feasible.  The results
>  > will have to contain the bitprint and the ranges which have been
>  > successfully verified.  On the receiving side, clients can use the
>  > range information to determine if the entire file is available over
>  > the network and use that to decide if the result should be displayed.
>  >
>  > Comments?
>
>  I have also suggested this since long (about 3 years ago), it was debated
>  several times, but not implemented since, most probably because it caused
>  problems in the reliability of swarmed downloaded.
>
>  But in fact this was caused by implementation bugs, causing swarmed
>  downloads to fail quite often if the same source was changing during the
>  transfer the set of ranges it currently has for a file and if the range
>  supported by a source was not contiguous from the start of the file.
>
>  But if we want to support fast and reliable swarmed downloads, we must be
>  able to support downloading from any source not necessarily from the start
>  of the file, but from any position; so as long as the swarmed download does
>  not complete, our local file will have as many "holes" in it as the number
>  of active sources we have used, so our local copy will contain multiple, non
>  contiguous fragments.
>
>  But even in this case, we can still share the fragments we already have
>  downloaded provided that they have been FULLY checked according to the
>  fragment of TigerTree data that represents it. This is possible for each
>  localfile fragment that is FULLY covered by a node in the TigerTree data,
>  and of course, it requires having this TigerTree data ready for this check.
>
>  But the TigerTree datamay be downloaded asynchronously in a separate time
>  after the first fragments have been downloaded from one source (because that
>  source may not have TigerTree data for this file, and we need to find
>  another source for that TigerTree data).
>
>  Consequently, we'll have locally some fragments that are still missing, some
>  fragments that have been downloaded but can't be checked because they are
>  not fully covered by some matching TigerTree data node, and some fragments
>  that have been downloaded and that now match with the tigertree data. Only
>  these last fragments can be shared safely and immediately even if the file
>  is not complete.
>
>  For any fragment that was previously downloaded, but that reveal later to be
>  corrupted according to the TigerTree data node we have now and that covers
>  it completely, such fragments must be dropped and reloaded from another
>  source. But this will continue to work because we have still not shared it.
>
>  This scheme works only because file fragments are checked before being
>  shares. And it benefits not only to rare files (allowing the number of
>  sources supporting a rare file to augment during the download by multiple
>  clients that still don't have a complete copy), but for ALL files on general
>  More generally it will avoid one client to steal ALL the upload bandwidth
>  from any uploader, because the uploader will be able to distribute its own
>  workload to lots of clients, if it only gives them a limited fragment size
>  before queueing again multiple clients with ongoing transfers):
>
>  the idea is to have for example N clients with active uploads in progress
>  plus N (or 2N: this is tuneable) clients temporarily idled in the requeueing
>  list. All the other newcoming candidate clients are queued with a lower
>  priority with the existing mechanism. In fact I would militate for an
>  implementation where each file can only have at most one active upload in
>  progress to any source, so this upload should be as fast as possible, but
>  limited to a small fragment size that can be checked immediately by the
>  receipinet according to TreeData.
>
>  Then that client is requeued, and other clients are served, but will be able
>  to use also the upload resources of the first served clients even if what
>  they have is partial. Each client that has successfully downloaded a
>  fragment will need to inform as soon as possible the uploading source so
>  that it can inform the other candidate clients to the same file that another
>  supplementary source is available that will help supporting the charge.
>  Rapidely this system will converge to faster speed for everybody.
>
>
>
>