Re: Digging Past Root Page Links
Jim <[email protected]>
| Newsgroups | gmane.comp.web.htdig.general |
|---|---|
| Message-ID | <[email protected]> |
On Thu, 31 Mar 2005, doug wrote: > I was able to install and setup htdig, no problem. I > ran the rundig program and it went through with no > errors, and the search functions work a-ok too. The > problem I'm having is that rundig only seems to be > indexing pages that are linked directly on the > start_url page. > > There's no robots.txt blocking access, and I haven't > changed the default hop count setting either, so I'm > stumped. > > Any suggestions would be greatly appreciated. I would start by running htdig with a few -v's and looking through the output for clues as to why documents are being dropped. Generally when a page is excluded the reason is also provided if you use a sufficiently high verbosity level. Jim ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ ht://Dig general mailing list: <[email protected]> ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-general