Re: working around status code 429

Timothe Litt via curl-users <[email protected]> Thu, 25 Jan 2024 05:54:11 -0500

Newsgroups	gmane.comp.web.curl.general
Message-ID	<[email protected]>

'split' is one option (man 1 split) for cutting the log into manageable 
(and ordered) chunks.

And/or try grep -B200 -E 'retry|429' | <your editor, less, ...>

Once you identify a failing instance, you can adjust the window to move 
back in time.

It can help to number the lines before splitting - 'nl -ba' will do that.

Happy hunting.

Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.

On 25-Jan-24 05:10, Dirk Hünniger via curl-users wrote:
>
> Hi Ray,
>
> I attached the output of
>
> curl --retry 30 -v -w "%{http_code} %{url} %{filename_effective}\n" 
> --compressed -K ./curlrun1 --parallel -L > out 2>&1
>
> but the file is so large that I cannot really work with well it in gedit.
>
> Yours Dirk
>
>
> On 25.01.24 00:48, Ray Satiro via curl-users wrote:
>> On 1/24/2024 6:13 PM, Daniel Stenberg via curl-users wrote:
>>> On Wed, 24 Jan 2024, Dirk Hünniger wrote:
>>>
>>>> I got a simplified test case:
>>>
>>> It is rare to see "a simplified test case" with a 16,000 line config 
>>> file fetching 8,000 files.
>>>
>>> Sorry, but I don't know what to do with that. 
>>
>>
>> I tried it several times but I can't reproduce a 429. The way the 
>> retry code is written it should retry those transfers if no curl 
>> error occurred. For example a curl error could occur if the output 
>> path /tmp/MediaWiki2LaTeXImages-3943f5a7379c6219/ does not exist at 
>> the time the output file is opened.
>>
>> Run curl in verbose mode (curl -v) and save the output (1>out 2>&1) 
>> and search for 'retry' and '429' and maybe there is something in the 
>> verbose output that would indicate what happened.
>>
>>
>

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html

OpenPGP_signature.asc (application/pgp-signature, 495 B)