Re: Identifying Known Spiders?
"Michael Crawford" <[email protected]>
| Newsgroups | gmane.comp.web.analog.general |
|---|---|
| Message-ID | <[email protected]> |
On Thu, Jul 3, 2008 at 8:37 AM, Jeremy Wadsack <[email protected]> wrote: > The robots list from which that page was built no longer exists. The group that was maintaining it decided that it didn't make sense to maintain a database of "known robots" any more as anyone can make a robot. In my personal case, it's not so much that I want to watch all the bots, as to monitor my progress at getting a new site indexed by the search engines. While Google, Yahoo and MSN together provide the vast majority of search engine referrals, there are still a few small, independent players such as JGDO. There are lots of reasons for running a bot, some good, some bad. I'd be happy if I could get a report of visits by the bots belong to, say, the top half-dozen search engines. Note that it often happens, with new sites, that a search engine spider may not visit at all for months, and even then will only fetch the home page. By creating config files for each of my pages, I hope to monitor spider visits throughout my site. If this isn't yet possible with analog, I don't think it would be hard to implement, and would be very popular, and so would get Analog a lot more users, and maybe some consulting fees for Analog experts. -- Michael David Crawford mdcrawford at gmail dot com Enjoy my art, photography, music and writing at http://www.geometricvisions.com/ --- Free Music Downloads --- +------------------------------------------------------------------------ | TO UNSUBSCRIBE from this list: | http://lists.meer.net/mailman/listinfo/analog-help | | Analog Documentation: http://analog.cx/docs/Readme.html | List archives: http://www.analog.cx/docs/mailing.html#listarchives | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general +------------------------------------------------------------------------