Re: Forbidden requests for kernel.org/releases.json

Hey Konstantin,

On Tue, Apr 15, 2025 at 11:01:44AM +0200, Borislav Petkov wrote:
> On Fri, Apr 11, 2025 at 04:54:25PM -0400, Konstantin Ryabitsev wrote:
> > You can continue doing it -- this is a lightweight operation. Just use a HEAD
> > request instead of a GET request.
> 
> Ok, here's what I have now:
> 
>                     headers = {
>                         "User-Agent": "Boris patch massager script vp.py ([email protected])"
>                     }
>                     get = requests.head(link_url, headers=headers)
>                     print(get.headers)
> 
> and for that done on the URL:
> 
> https://lore.kernel.org/[email protected]
> 
> it returns 302 with the Location header redirecting to the same thing but in
> the /all/ range.
> 
> {'Server': 'nginx', 'Date': 'Tue, 15 Apr 2025 08:52:28 GMT', 'Content-Type': 'text/plain', 'Content-Length': '79', 'Connection': 'keep-alive', 'Age': '0', 'Location': 'http://lore.kernel.org/all/[email protected]/', 'Via': '1.1 varnish (Varnish/6.6)', 'X-Varnish': '17023135', 'X-Frame-Options': 'DENY', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Strict-Transport-Security': 'max-age=15768001', 'Content-Security-Policy': "default-src 'self'; worker-src 'self' blob:; style-src 'self' 'unsafe-inline'; img-src https:"}
> 
> Now, if I query the URL in the /all/ range, it gives 301:
> 
> {'Server': 'nginx', 'Date': 'Tue, 15 Apr 2025 08:56:39 GMT', 'Content-Type': 'text/plain', 'Content-Length': '79', 'Connection': 'keep-alive', 'Age': '0', 'Location': 'http://lore.kernel.org/all/[email protected]/', 'Via': '1.1 varnish (Varnish/6.6)', 'X-Varnish': '9754343', 'X-Frame-Options': 'DENY', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Strict-Transport-Security': 'max-age=15768001', 'Content-Security-Policy': "default-src 'self'; worker-src 'self' blob:; style-src 'self' 'unsafe-inline'; img-src https:"}
> 
> giving me the unencrypted http:// Location and if I do that it gives me 301
> again to the *encrypted* URL:
> 
> {'Server': 'nginx', 'Date': 'Tue, 15 Apr 2025 08:57:15 GMT', 'Content-Type': 'text/html', 'Content-Length': '162', 'Connection': 'keep-alive', 'Location': 'https://lore.kernel.org/all/[email protected]'}
> 
> LOL.
> 
> So, what would be the best and the lowest overhead thing to use?

do you have any guidance on this?

It seems that the AI bot protection is returning status 301 now for any URL
- valid or not:

Link: https://lore.kernel.org/all/bla
...
check_url: ERROR: URL https://lore.kernel.org/all/bla not reachable, status_code: 301 : {'Server': 'nginx', 'Date': 'Mon, 13 Oct 2025 13:12:38 GMT', 'Content-Type': 'text/plain', 'Content-Length': '48', 'Connection': 'keep-alive', 'Age': '0', 'Location': 'https://lore.kernel.org/all/bla/', 'Via': '1.1 varnish (Varnish/6.6)', 'X-Varnish': '172179071', 'X-Frame-Options': 'DENY', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Strict-Transport-Security': 'max-age=15768001', 'Content-Security-Policy': "default-src 'self'; worker-src 'self' blob:; style-src 'self' 'unsafe-inline'; img-src https:"}

after doing a HEAD request for the URL.

Should I stop checking Link URL availability in my script?

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette