Re: [core-dev] Downloading in 100k chunks

You are right. There has to be a lower limit for the chunk size which could be the 100k 
LimeWire is using at the moment.
You could also use 100k or 1% of the amount that has yet to be downloaded, which would 
probably help circumventing the problem of very low parellelism when a download is near 
complete.
It would be interesting to know what implications a larger chunk size would have on the 
download mesh.

mfg
gregorio

Sumeet Thadani wrote:

> It's time to revisit this stuff. Currently about 100% of our downloads 
> HTTP1.1, since we used having a hash as the heuristic for distinguishing 
> HTTP1.1 uploaders from HTTP 1.0 uploaders.
> 
> Given that MIN_SPLIT_SIZE ==  CHUNK_SIZE we are seeing no stealing 
> happening at all. For the stealing to occur, MIN_SPLIT_SIZE needs to be 
> less than CHUNK_SIZE. Note that this is not a bad thing -- read on to 
> see why.
> 
> Increasing the chunk size has the advantage that on 
> high-latency/low-bandwidth connections we would speed up the download by 
> sending less headers (note that this was much worse with HTTP 1.0 
> downloads -- since if the chunk size was too small we would have to 
> re-establish the TCP connection for each chunk).
> Having a very large chunk is also not the best idea -- stealing from a 
> HTTP 1.1 uploader is tricky, because there is no way to tell the server 
> to stop the uploading (meaning change your mind about the range you 
> want) after you have made the request. Once the request is made, you 
> have to download the range requested before you make the next request or 
> close the connection if another worker steals a part of that chunk from 
> you. So you have to predeterine what ranges you want before you make the 
> request, and if the range is too big, and one downloader is going slow 
> you can steal from it but at the expense of having the close that 
> connection, So It's a delicate balance.
> 
> Using a heuristic like 1% may not be optimal --  for a small file, 1% 
> can be much smaller than the current chunk size, and may cause far more 
> http headers to be exchanged than necessary.
> 
> Comments?
> 
> Thanks,
> 
> Sumeet
> 
> Gregorio Roper wrote:
> 
>> Is it still necessary to download all files in 100k chunks? It seems a 
>> little inefficient sending all these HTTP-requests/responses back and 
>> forth and it definitely slows the download down if you are working 
>> with a high latency connection.
>> Maybe using a dynamic chunk size of 1% of the file size would be a 
>> good idea.
>>
>> mfg
>> gregorio
>> _______________________________________________
>> core-dev mailing list
>> [email protected]
>> http://www.limewire.org/mailman/listinfo/core-dev
>>
>> .
>>
> 
> 
> _______________________________________________
> core-dev mailing list
> [email protected]
> http://www.limewire.org/mailman/listinfo/core-dev
> 
> 
_______________________________________________
core-dev mailing list
[email protected]
http://www.limewire.org/mailman/listinfo/core-dev