Re: gitweb performance/errors

--===============8365511449959519374==
Content-Type: multipart/signed; boundary="Sig_/lelZ4iYtERI8Sk1lAJGQI4s";
 protocol="application/pgp-signature"; micalg=pgp-sha256

--Sig_/lelZ4iYtERI8Sk1lAJGQI4s
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

Fabian Keil via Privoxy-users <[email protected]> wrote on 20=
26-01-25 at 19:01:40:

> tmknight88--- via Privoxy-users <[email protected]> wrote o=
n 2026-01-25 at 12:06:35:
>=20
> > Hi, I wanted to report (assuming you are unaware) that the gitweb inter=
face
> > of privoxy.git performance has degraded since approximately 23 Jan 2026.
> > For example, the summary page can take many seconds to load and frequen=
tly
> > times out with a generic nginx error message.
>=20
> Thanks for the report.
>=20
> I was already aware of the problem but haven't had time to look into solv=
ing it.
>=20
> The robots.txt already disallows crawling of /gitweb and a bunch
> of obvious "AI" bots receive static error pages, but since about
> three days ago there are simply too many requests coming in from
> clients that ignore the robots.txt and don't use obvious "AI" bot
> User-Agent headers either.
>=20
> Looks like we need more crawler counter measures.

I've added a cookie barrier which seems to keep the crawlers at bay for now.

Fabian

--Sig_/lelZ4iYtERI8Sk1lAJGQI4s
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEETzbBfzgWkTZUoehQaRgikYuiNxwFAml7bQwACgkQaRgikYui
NxzDZg//Q4rGyl1Q2ouZGnAqhS0ggXyX0QgUdzna6XHjdkawy402kwf8ELK4XFd2
ijOfeZA99B2bELW7t8kc2jyGylff3U6a96OjWUN2/vL+gGq0fwJhxw4ro5wWzDLG
v99v72zRlWSpy+PgObhsGH1ztgDvBGXtw4jNt1UlUZ8M7fp4SrIFVgBhZrgNe6RI
0FxiR5caSPlZmcgvLCb7pZTvvcljV38kA77SLl90pReQHfcZ4+r1t9tqIs/wIKc7
b928vEsAtmQlFNf/cg2gll/1bzWmXzd/V5CgJLuJYehS18UtWIJyYG5J9XNkDOtF
WCEB2GI3GXD5JnlI1dFdlPoc9KsQDN8wK+vsGilw2vuQ9/MPj9WqgYbPmb8tx0Lk
inXxu/FI2XPlnJdxyOaDjs1jStG6h89V/Sj6cco75/Tp/fcjgtHs3GKXLJpaZuGM
FicOUfU4/WZnqiYv/3cWxWM8uMnPAQiMVJnXBr2Ift4Q8asKcgg1CQPKo7oTotOA
saVYTvIuwXCoIg+U3KNV+5ws9JWE+zAzZ/iU8DiFl8eKpAiHH3juliMTq8NKG71R
wXroI1fHkatnDYXN70FSiOB3ivgCXz8Tqn2D6IlKtttkxYqARhCmMzGukWLD/2Nf
brL6K9HpqSxcRCZFOUkF+Vyrtq20TjCQmnnLQykNxpJwMDWDerY=
=5EY0
-----END PGP SIGNATURE-----

--Sig_/lelZ4iYtERI8Sk1lAJGQI4s--

--===============8365511449959519374==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Privoxy-users mailing list
[email protected]
https://lists.privoxy.org/mailman/listinfo/privoxy-users

--===============8365511449959519374==--