Re: working around status code 429

On 1/31/2024 7:08 AM, Dirk Hünniger via curl-users wrote:
> sorry for the delay. I updated the output file in the file curlrun1 to 
> point to tmp, which is a path that exists. I attached a 7zip file 
> "files.7z" of the input file "curlrun1" and the output file "out".  I 
> used the following curl command:
>
> curl --retry 30 -v -w "%{http_code} %{url} %{filename_effective}\n" 
> --compressed -K ./curlrun1 --parallel -L > out 2>&1
>
> When I do
>
> cat out | grep "429 h"
>
> I get the output:
>
> 429 https://de.wikibooks.org/File:Parathion-skeletal.png /tmp/626.2.html
> 429 https://de.wikibooks.org/wiki/Wikibooks:Sammlungen/File:Betain.svg 
> /tmp/637.3.html
> and more.
>
> So to my understanding the are 429 errors, which are not retried, 
> although I used --retry in the call to curl. 


Try:
grep "Warning: Problem : HTTP error. Will retry in" out

This most recent log shows transfers are retried (you can see the 
decreasing numbers in the barrage of output). Why do you think the 
transfers are not retried? This may be an unusual case because the 
server sets the retry delay time to 1 second using the header 
`retry-after: 1` and curl complies with that directive [1], but it is 
probably an impractical time.

To address this I am proposing a new command line option that will allow 
ignoring the server's retry delay time. [2]


[1]: https://curl.se/docs/manpage.html#--retry
[2]: https://github.com/curl/curl/pull/12837

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html