Re: Web crawling software?

I have compiled a list of resources for a course on Web mining, and  
the list includes pointers to a few off-the-shelf crawlers such as  
Heritrix:

	http://informatics.indiana.edu/fil/Class/b659/resources.html#sw

Best,

Filippo Menczer
================================================================
Departments of Informatics & Computer Science
Indiana University           http://informatics.indiana.edu/fil
Eigenmann Hall 909           Voice: (812) 856-1377
Bloomington, IN 47406        Fax:   (812) 856-1995
================================================================


On Feb 3, 2007, at 1:19 PM, Sergio Nunes wrote:

> Hi,
>
> I'm currently evaluating free web crawling software for several large
> crawls that I'm going to perform. Developing my own software is also a
> possibility, but I'm trying to avoid the "not invented here"  
> syndrome :)
>
> What software package do you recommend for performing a large web
> crawl on "off-the-shelf" hardware?
>
> If you really don't have time to send me a detailed answer, please
> just point to the software you've successfully used. I'll do a simple
> count.
>
> Please send your answer directly to me and I'll summarize the results
> to the list.
>
> Thanks in advance,
> Sérgio Nunes