Re: Update base-url for xorg downloads?

On Tue, Nov 4, 2025 at 11:05 AM "Douglas R. Reno"
<[email protected]> wrote:
>
> On 10/23/25 12:25 PM, Rainer Fiebig ([email protected] via blfs-dev
> Mailing List) wrote:
> > Am 23.10.25 um 17:40 schrieb Bruce Dubbs ([email protected] via
> > blfs-dev Mailing List):
> >> On 10/23/25 10:24 AM, Rainer Fiebig ([email protected] via blfs-dev
> >> Mailing List) wrote:
> >>> Maybe the base-url for xorg-downloads needs to be updated.
> >>>
> >>> Testing a script, I downloaded several xorg-packages, using the urls in
> >>> the latest wget-list for BLFS.  Those downloads took unusually long.
> >>>
> >>> Suspecting a gnu-like problem, I checked out x.org.  Turned out that the
> >>> current base-url for downloads is
> >>> https://www.x.org/releases/individual/
> >>>
> >>> In BLFS it is
> >>> https://www.x.org/pub/individual/
> >> I went here...
> >>
> >>> The latter redirects to
> >>> https://www.x.org/archive/individual/
> >> True.
> >>
> >>> which may be the reason for the slow downloads, I speculate.
> >>>
> >>> At any rate, after updating the urls in the wget-list, I could download
> >>> xorg packages at the usual speed.
> >> However the pages came up almost instantaneously for me and I did not
> >> see any download speed issue.
> > OK, so perhaps what I saw was just an anomaly.  But I'll leave the sed
> > to substitute the current base-url for the old (?) one in my script.  It
> > won't hurt, I think.
> >
> > I did the same with the gnu-urls because the problem with slow downloads
> > still exists, at least as of Oct. 21.  I found this helpful for chosing
> > a suitable mirror: https://download.savannah.gnu.org/mirmon/allgnu/.
> >
> > Rainer
>
> I know this is late, but as I'm just starting to return to things today
> - the reason why the downloads from x.org are slow is the same reason
> why ftp.gnu.org and freedesktop.org are struggling at times - the usage
> of AI crawlers consistently scraping their sites.
>
> As far as I'm aware there aren't mirrors for freedesktop hosted
> infrastructure (including x.org), but in the event that the site goes
> too slow or goes completely offline, users can get packages from
> https://ftp2.osuosl.org/pub/blfs/conglomeration/ - I personally still
> prefer the official sites but have had to use this recently myself!
>
> - Doug
>


On the LFS side as a preventative measure can we set up
https://anubis.techaro.lol/, it makes it very computationally
expensive for crawlers to come through but for regular users they do a
Proof of work and save it as a cookie and it never bothers them again.
It's extremely fast on any modern system and makes it so that "bad"
bots, like those used by AI search companies can't come through. Good
bots (like wget and curl) will still work without issue. From some
quick reading we should still be fine for google search indexing but
maybe disable it on the *LFS books themselves and only enable it for
Trac.
-- 
Rahul

-- 
http://lists.linuxfromscratch.org/sympa/info/blfs-dev
Unsubscribe: See the above information page