NLNZ 2018 domain crawl
Ben O'Brien <Ben.O'[email protected]>
| Newsgroups | gmane.org.operators.nznog |
|---|---|
| Message-ID | <4E91115DC917E64E8395E4CD9BFDE759010C1BC26C@wlgprdmbx06.dia.govt.nz> |
Hi, We at the National Library are about to begin our 2018 New Zealand domain crawl. As in previous years, the crawl is being run by Internet Archive, and some of you may have already seen the crawlers as IA go through their initial setup with some test crawls. The user agent string their crawlers will be using is: NLNZ_IAHarvester2018 While we do our best to crawl respectfully, obey robots.txt, and break out of any crawler traps before grinding too much bandwidth, if you encounter any issues from us then please get in contact. Cheers, Ben O’Brien | Digital Preservation Web Engineer | Preservation, Research and Consultancy National Library of New Zealand | Te Puna Mātauranga o Aotearoa PO Box 1467, Wellington 6140, New Zealand | www.natlib.govt.nz<http://www.natlib.govt.nz/> _______________________________________________ NZNOG mailing list [email protected] https://list.waikato.ac.nz/mailman/listinfo/nznog