Problem downloading a website from archive.org

<[email protected]> Wed, 12 Mar 2025 22:57:05 -0000
Newsgroups gmane.comp.web.wget.general
Message-ID <01e001db93a2$361e1dd0$0200050a@tshome>
Hi Everyone

I am trying to download a complete website from archive.org using Wget =
but I have run into a problem.

If you are a human and you are exploring an old website on archive.org, =
you may notice that sometimes when you click on a link from one page on =
the website to another, the datestamp part of the URL changes. You can =
also end up on the same page as you were previously but with a different =
datestamp.

This is not much of a problem if you are a human but it a problem for =
webcrawlers such as wget because they can end up duplicating some parts =
of a website many times and not reaching other parts of a website for a =
very long time. This is the problem I am having.

Do any of you know the solution to this problem?

Thank you very much.

Kind regards

Tim

P.S.  I am sorry if this a duplicate but I previously posted it before =
subscribing so I don't know if it went through.