RE: [Xaraya_users] Re: Crawlers/SE/roBots

>> Xaraya indexes well within Search Engines, but they need to 
>> know where the site is before they can do anything.  You 
>> probably need to get a few in bound links to your site.
>> 

An additional item that I ran into, while integrating phpDig into Xaraya was
that you need to make sure you don't have any relative links when using
Short Urls.

The reason is that _some_ search engines will do this (actual exp w/ phpDig
doing it):
* Go to homepage: http://domain.com/index.php
* Follow a link to an article: http://domain.com/index.php/articles/123
* Follow a link to a pubtype: http://domain.com/index.php/articles/pubtype/
* See a relative link, like index.php/polls/123 that has been manually
entered into the main menu.
* Follows that link, constructing the relative link based on the current
page, ie resulting in:

http://domain.com/index.php/articles/pubtype/index.php/polls/123

Xaraya does not throw an error - nor does it terminate the page.  Instead it
happily displays _something_.  Something with links to other things.  So the
crawler will attempt to follow those links as well.  Usually finding it's
way back to article & poll link already visited, resulting in another level
of depth:

http://domain.com/index.php/articles/pubtype/index.php/polls/123/index.php/p
olls/123

With some crawlers (phpDig) this creates an infinite loop (or near infinite
loop because of exponential growth of the internal links of the site) and
causes the crawler to crash and in essence your site never gets indexed.

Solution:
Make sure all your links are fully qualified.  IE never have just
"index.php/articles/" but always have "http://domain.com/index.php/articles"

All links generated by the Xaraya API should already do this, it's only
user/admin entered URLs that can be troublesome.

--
Mike C.