Re: Wget fails to download some URLs from www.investing.com

Tim Rühsen <[email protected]> Sun, 18 Feb 2024 13:56:11 +0100

Newsgroups	gmane.comp.web.wget.general
Message-ID	<[email protected]>

On 2/14/24 11:21, Chris Smith via Primary discussion list for GNU Wget 
wrote:
> Hi guys,
> This is not so much a bug as requests being blocked by the cloudflare server.
> 
> Checkout:
>      https://www.investing.com/crypto/bitcoin/btc-usd-historical-data

The server just gives a 403 Forbidden.

> 
> The URL works in Firefox, but fails to download using cURL or Wget.
> I have tried various user-agent strings, so that is not the problem.

Maybe the page requires a Referrer: header? In theory it could be 
anything, e.g. the page is HTTP/3 only or whatnot :|

You can start with F12 (developer console) in Firefox, load the page and 
check the request headers. Or better: there is a copy-as-curl when 
right-clicking on the request line that puts a full curl command line 
into your clipboard.

> 
> The following URL works for Firefox, cURL and Wget:
>      https://www.investing.com/equities/lloyds-banking-grp-historical-data
> 
> Kind regards,
> Chris Smith <[email protected]>
> 
> _________________________________________________________________
> ________________________________________________________
> Your E-Mail. Your Cloud. Your Office. eclipso Mail Europe. https://www.eclipso.de
> 
> 
>

OpenPGP_signature.asc (application/pgp-signature, 840 B)