RE: New Spider

This message was sent from: Tips and Tricks
http://forums.oscommerce.com/viewtopic.php?p=224808#224808
----------------------------------------------------------------

Because the of the way spiders work.

There's usually 2 different spiders that actually visit.

The first one just comes to your site, and gathers as many URLs as it can find, and puts them into a 'list' of urls to later parse.

The seond one comes, and goes through the list a parses each page for each URL in the list.

Then the first one comes back, and looks for any URLs that it might have missed the first time. If it finds new ones, it adds them to the list.

Each time the "Gatherer" finds a URL with a SID, it thinks it's found a new a new url, and adds it to the list again.

This process repeats until it finds no new urls, and all urls have been parsed.

if in the middle of this process, you stop the sids from appearing in the urls for the spider, it still has to finish processing all of the urls that are already in the list. 

Therefore, depending on how long it was trapped with SIDs in the URL, you could continue to see the SIDs in the URLs that the spider is visiting until it has parsed them all. By stipping the SID from appearing, you stop any NEW urls with SID from getting put in your list of URLs the spider needs to parse.

I'm sure htis is overly simplified, but I think that its a bit easier to understnad spiders when explained this way.