Re: working around status code 429

Hi Ray,

sorry for the delay. I updated the output file in the file curlrun1 to 
point to tmp, which is a path that exists. I attached a 7zip file 
"files.7z" of the input file "curlrun1" and the output file "out".  I 
used the following curl command:

curl --retry 30 -v -w "%{http_code} %{url} %{filename_effective}\n" 
--compressed -K ./curlrun1 --parallel -L > out 2>&1

When I do

cat out | grep "429 h"

I get the output:

429 https://de.wikibooks.org/File:Parathion-skeletal.png /tmp/626.2.html
429 https://de.wikibooks.org/wiki/Wikibooks:Sammlungen/File:Betain.svg 
/tmp/637.3.html
and more.

So to my understanding the are 429 errors, which are not retried, 
although I used --retry in the call to curl.

Yours Dirk

On 26.01.24 08:28, Ray Satiro via curl-users wrote:
> On 1/25/2024 5:10 AM, Dirk Hünniger via curl-users wrote:
>>
>> I attached the output of
>>
>> curl --retry 30 -v -w "%{http_code} %{url} %{filename_effective}\n" 
>> --compressed -K ./curlrun1 --parallel -L > out 2>&1
>>
>> but the file is so large that I cannot really work with well it in 
>> gedit.
>>
>
> The output shows that the transfers that return HTTP 429 have failed 
> with curl error CURLE_WRITE_ERROR (23) and that is at least one reason 
> why they are not being retried. You can try with the latest version of 
> curl but I suspect this is not going to be a curl issue. Also, if 
> gedit isn't working for you then use something else. There's no reason 
> you shouldn't be able to work with large files like that in a text 
> editor.
>
> Here's one example. They all look the same:
>
> Warning: Failed to create the file
> Warning: /tmp/MediaWiki2LaTeXImages-3943f5a7379c6219/827.4.html: No 
> such file
> Warning: or directory
> * Failure writing output to destination
> * stopped the pause stream!
> curl: (23) Failure writing output to destination
>
>