Re: CPU usage since 8.7.1

> Am 29.10.2025 um 22:12 schrieb Dmitry Karpov via curl-library <[email protected]>:
> 
> I ran the same throttling tests on 8.16.0 that helped to find the speed throttling issue on 8.7, and I can tell that the throttling accuracy in 8.16.0 is significantly worse than in 8.6.0 and 8.7.1.
> 
> My tests use 1MB downloads on network speeds: 700mbps, 50mbps and 20mbps, with 8mbps and 16mbps throttling speeds.
> The test measures the performed transfers speeds on 10 downloads and calculates average and max deviations from the rate limits.
> 
> Here is the brief description of test results:
> 
> 8.6.0 (and 8.7.1)
> Network speed: 700mbps,  Rate limit: 8mbps, Avg deviation: -0.86%, Max deviation: -1.5%.
> Network speed: 700mbps,  Rate limit: 16mbps, Avg deviation: -1.72%, Max deviation: -2.8%.
> 
> Network speed: 50mbps,  Rate limit: 8mbps, Avg deviation: -0.37%, Max deviation: -1%.
> Network speed: 50mbps,  Rate limit: 16mbps, Avg deviation: - 1.64%, Max deviation: -3.2%.
> 
> Network speed: 20mbps,  Rate limit: 8mbps, Avg deviation: -0.6%, Max deviation: -1.1%.
> Network speed: 20mbps,  Rate limit: 16mbps, Avg deviation: -1.37%, Max deviation: -1.7%.
> 
> 8.16.0
> Network speed: 700mbps,  Rate limit: 8mbps, Avg deviation:  4.65%, Max deviation: 13.7%.
> Network speed: 700mbps,  Rate limit: 16mbps, Avg deviation: 6.69%, Max deviation: 17.7%.
> 
> Network speed: 50mbps,  Rate limit: 8mbps, Avg deviation: 7.03%, Max deviation: 10.2%.
> Network speed: 50mbps,  Rate limit: 16mbps, Avg deviation: 9.99%, Max deviation: 15%.
> 
> Network speed: 20mbps,  Rate limit: 8mbps, Avg deviation: 10.89%, Max deviation: 15.5%.
> Network speed: 20mbps,  Rate limit: 16mbps, Avg deviation: 5.34%, Max deviation: 14.5%.
> 
> As we can see, the 8.6.0 (and 8.7.1) has almost perfect rate limit accuracy with a tendency to curb the download speed a little but below the limit,
> which is a good thing that makes the download speed unlikely to go beyond the limit.
> 
> On the other hand, the 8.16.0 has wider deviations from the rate limits with a tendency to use more network bandwidth than the limit specifies.
> And the max deviations above 10% make it difficult to have reliable predictions of the throttled download speeds.
> 
> If the almost perfect accuracy of 8.6.0/8.7.1 comes with a price of much higher CPU utilization, which maybe not acceptable for everyone,
> then maybe we need to adjust the rate limiting mechanism and make it to provide just 5% accuracy with a tendency to use less network bandwidth than the rate limit,
> when the precise accuracy is not possible. 
> 
> The tendency to use more bandwidth in the current mechanism creates problems when rate limits are specifically set to prevent low priority downloads from stealing
> network bandwidth from downloads with higher priority.

We'd have to revisit rate-limited/paused transfers in the future. Your measurements show that this could be better.

Not sure when we'll find the time for that. So, I cannot promise anything.

Cheers,
Stefan

> 
> Thanks,
> Dmitry Karpov
> 
> 
> 
> 
> -----Original Message-----
> From: curl-library <[email protected]> On Behalf Of Dmitry Karpov via curl-library
> Sent: Tuesday, October 28, 2025 11:38 AM
> To: libcurl development <[email protected]>
> Cc: Dmitry Karpov <[email protected]>
> Subject: [EXTERNAL] Re: CPU usage since 8.7.1
> 
> It was me who reported it back then.
> The precision loss of the rated downloads was more than 10% in some cases, which was above our 5% tolerance.
> 
> It caused problems in multi-download prioritized scenarios where rated downloads with less priority were stealing bandwidth from the downloads with higher priority.
> 
> I am also concerned about CPU usage, but 10% rate precision deviation is too high.
> And in our case (video streaming application), the problem was that the prioritized video got worse quality and other problems which affected our users.
> 
> Ideally, the optimal solution would be to avoid busy loops and using too many CPU cycles, and provide 5% rate accuracy.
> 
> Thanks!
> Dmitry Karpov
> 
> -----Original Message-----
> From: curl-library <[email protected]> On Behalf Of Gleb Smirnoff via curl-library
> Sent: Tuesday, October 28, 2025 11:11 AM
> To: Daniel Stenberg <[email protected]>; Stefan Eissing <[email protected]>
> Cc: Gleb Smirnoff <[email protected]>; Gleb Smirnoff via curl-library <[email protected]>
> Subject: [EXTERNAL] Re: CPU usage since 8.7.1
> 
> On Tue, Oct 28, 2025 at 05:32:27PM +0100, Daniel Stenberg wrote:
> D> On Tue, 28 Oct 2025, Gleb Smirnoff via curl-library wrote:
> D> 
> D> > So here I am raising this question.  What needs to be done to get 
> D> > CPU savings of 8.6.0 level and how can we help here?
> D> 
> D> As recall things, the change that introduced the new (current) 
> D> behavior was done to enhance the accuracy of the rate limit logic, at 
> D> the expense of CPU.
> D> 
> D> The change was pushed for by someone who argued that the increased 
> D> precision was valuable for them if I remember correctly.
> D> 
> D> Can we find a way to have both ways and let the user select which? 
> D> Maybe for example by maybe something as silly and simple as doing 
> D> CURLOPT_MAX_RECV_SPEED_LARGE % 10000 and if that is non-zero, higher 
> D> precision is wanted.
> 
> I added Stefan to explicit Cc.  Since he very recently added 24badd29f titled "limit-rate revisited" (of course this conlicted with out patch again), he definitely should have an opinion on how this can be solved.
> 
> --
> Gleb Smirnoff
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette:   https://curl.se/mail/etiquette.html
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette:   https://curl.se/mail/etiquette.html
> -- 
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette:   https://curl.se/mail/etiquette.html

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html