Problem downloading a website from archive.org
<[email protected]> Wed, 12 Mar 2025 22:57:05 -0000
| Newsgroups | gmane.comp.web.wget.general |
|---|---|
| Message-ID | <01e001db93a2$361e1dd0$0200050a@tshome> |
Hi Everyone I am trying to download a complete website from archive.org using Wget = but I have run into a problem. If you are a human and you are exploring an old website on archive.org, = you may notice that sometimes when you click on a link from one page on = the website to another, the datestamp part of the URL changes. You can = also end up on the same page as you were previously but with a different = datestamp. This is not much of a problem if you are a human but it a problem for = webcrawlers such as wget because they can end up duplicating some parts = of a website many times and not reaching other parts of a website for a = very long time. This is the problem I am having. Do any of you know the solution to this problem? Thank you very much. Kind regards Tim P.S. I am sorry if this a duplicate but I previously posted it before = subscribing so I don't know if it went through.