Re: RE: [Xaraya_users] Re: Crawlers/SE/roBots
[email protected] (mikespub) Mon, 01 Dec 2003 20:11:51 GMT
| Newsgroups | gmane.comp.cms.xaraya.knowledge-base |
|---|---|
| Organization | not much |
| Message-ID | <[email protected]> |
In article <mailman.1221.1070294063.4976.xaraya_knowledge-base-n5IRV0TL0hwRxVAL8JNkPw@public.gmane.org>, Xaraya Knowledge Base <[email protected]> wrote: >... >With some crawlers (phpDig) this creates an infinite loop (or near infinite >loop because of exponential growth of the internal links of the site) and >causes the crawler to crash and in essence your site never gets indexed. > >Solution: >Make sure all your links are fully qualified. IE never have just >"index.php/articles/" but always have "http://domain.com/index.php/articles" > >All links generated by the Xaraya API should already do this, it's only >user/admin entered URLs that can be troublesome. > Also note that search engines and crawlers should take into account the base href directive specified in page headers - this explicitly sets the base correctly for short URLs in Xaraya. So if phpDig doesn't take into account the base href, a bug report should be issued for that crawler - unless you removed the base href from your page template of course, in which case you're the one to blame :-) Mike.