Re: Web crawling software?

"ChaTo (Carlos Alberto-Alejandro CASTILLO-Ocaranza)" <[email protected]>
Newsgroups gmane.comp.information-retrieval.webir
Message-ID <1170539371.5126.8.camel@pitagoras> 
On Sat, 2007-02-03 at 18:19 +0000, Sergio Nunes wrote:

> I'm currently evaluating free web crawling software for several large
> crawls that I'm going to perform. Developing my own software is also a
> possibility, but I'm trying to avoid the "not invented here" syndrome :)

Nutch -- Java, written to work with Lucene, has a partnership with
Apache Foundation

Heritrix -- Java, written for the Internet Archive

WIRE -- C++, written for Web characterization studies [by our group at
CWR.cl]

Check also this list, it's quite complete:
http://en.wikipedia.org/wiki/Web_crawler#Examples_of_Web_crawlers

--
ChaTo
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.