Re: [core-dev] Downloading in 100k chunks

If we decide to change that, can we make it a power of 2? It feels much better that way ;-)

---------- Original Message ----------------------------------
From: Gregorio Roper <[email protected]>
Reply-To: [email protected]
Date: Wed, 30 Jun 2004 17:17:19 +0200

>You are right. There has to be a lower limit for the chunk size which could be the 100k 
>LimeWire is using at the moment.
>You could also use 100k or 1% of the amount that has yet to be downloaded, which would 
>probably help circumventing the problem of very low parellelism when a download is near 
>complete.
>It would be interesting to know what implications a larger chunk size would have on the 
>download mesh.
>
>mfg
>gregorio
>
>Sumeet Thadani wrote:
>
>> It's time to revisit this stuff. Currently about 100% of our downloads 
>> HTTP1.1, since we used having a hash as the heuristic for distinguishing 
>> HTTP1.1 uploaders from HTTP 1.0 uploaders.
>> 
>> Given that MIN_SPLIT_SIZE ==  CHUNK_SIZE we are seeing no stealing 
>> happening at all. For the stealing to occur, MIN_SPLIT_SIZE needs to be 
>> less than CHUNK_SIZE. Note that this is not a bad thing -- read on to 
>> see why.
>> 
>> Increasing the chunk size has the advantage that on 
>> high-latency/low-bandwidth connections we would speed up the download by 
>> sending less headers (note that this was much worse with HTTP 1.0 
>> downloads -- since if the chunk size was too small we would have to 
>> re-establish the TCP connection for each chunk).
>> Having a very large chunk is also not the best idea -- stealing from a 
>> HTTP 1.1 uploader is tricky, because there is no way to tell the server 
>> to stop the uploading (meaning change your mind about the range you 
>> want) after you have made the request. Once the request is made, you 
>> have to download the range requested before you make the next request or 
>> close the connection if another worker steals a part of that chunk from 
>> you. So you have to predeterine what ranges you want before you make the 
>> request, and if the range is too big, and one downloader is going slow 
>> you can steal from it but at the expense of having the close that 
>> connection, So It's a delicate balance.
>> 
>> Using a heuristic like 1% may not be optimal --  for a small file, 1% 
>> can be much smaller than the current chunk size, and may cause far more 
>> http headers to be exchanged than necessary.
>> 
>> Comments?
>> 
>> Thanks,
>> 
>> Sumeet
>> 
>> Gregorio Roper wrote:
>> 
>>> Is it still necessary to download all files in 100k chunks? It seems a 
>>> little inefficient sending all these HTTP-requests/responses back and 
>>> forth and it definitely slows the download down if you are working 
>>> with a high latency connection.
>>> Maybe using a dynamic chunk size of 1% of the file size would be a 
>>> good idea.
>>>
>>> mfg
>>> gregorio
>>> _______________________________________________
>>> core-dev mailing list
>>> [email protected]
>>> http://www.limewire.org/mailman/listinfo/core-dev
>>>
>>> .
>>>
>> 
>> 
>> _______________________________________________
>> core-dev mailing list
>> [email protected]
>> http://www.limewire.org/mailman/listinfo/core-dev
>> 
>> 
>_______________________________________________
>core-dev mailing list
>[email protected]
>http://www.limewire.org/mailman/listinfo/core-dev
>
_______________________________________________
core-dev mailing list
[email protected]
http://www.limewire.org/mailman/listinfo/core-dev