Re: [core-dev] Downloading in 100k chunks

It's time to revisit this stuff. Currently about 100% of our downloads 
HTTP1.1, since we used having a hash as the heuristic for distinguishing 
HTTP1.1 uploaders from HTTP 1.0 uploaders.

Given that MIN_SPLIT_SIZE ==  CHUNK_SIZE we are seeing no stealing 
happening at all. For the stealing to occur, MIN_SPLIT_SIZE needs to be 
less than CHUNK_SIZE. Note that this is not a bad thing -- read on to 
see why.

Increasing the chunk size has the advantage that on 
high-latency/low-bandwidth connections we would speed up the download by 
sending less headers (note that this was much worse with HTTP 1.0 
downloads -- since if the chunk size was too small we would have to 
re-establish the TCP connection for each chunk). 

Having a very large chunk is also not the best idea -- stealing from a 
HTTP 1.1 uploader is tricky, because there is no way to tell the server 
to stop the uploading (meaning change your mind about the range you 
want) after you have made the request. Once the request is made, you 
have to download the range requested before you make the next request or 
close the connection if another worker steals a part of that chunk from 
you. So you have to predeterine what ranges you want before you make the 
request, and if the range is too big, and one downloader is going slow 
you can steal from it but at the expense of having the close that 
connection, So It's a delicate balance.

Using a heuristic like 1% may not be optimal --  for a small file, 1% 
can be much smaller than the current chunk size, and may cause far more 
http headers to be exchanged than necessary.

Comments?

Thanks,

Sumeet

Gregorio Roper wrote:

> Is it still necessary to download all files in 100k chunks? It seems a 
> little inefficient sending all these HTTP-requests/responses back and 
> forth and it definitely slows the download down if you are working 
> with a high latency connection.
> Maybe using a dynamic chunk size of 1% of the file size would be a 
> good idea.
>
> mfg
> gregorio
> _______________________________________________
> core-dev mailing list
> [email protected]
> http://www.limewire.org/mailman/listinfo/core-dev
>
> .
>


_______________________________________________
core-dev mailing list
[email protected]
http://www.limewire.org/mailman/listinfo/core-dev