Web crawling software - Summary

Hi,

Thanks to all who contributed to my initial question: "What software
package do you recommend for performing a large web crawl on
"off-the-shelf" hardware?"

The list of recommended software is below, with the number of
references indicated:

* (4) Heritrix - http://crawler.archive.org/
* (2) WIRE - http://www.cwr.cl/projects/WIRE/
* (1) Labrador - http://www.dcs.gla.ac.uk/~craigm/labrador/
* (1) Poacher (could not find the URL)
* (1) Larbin - http://larbin.sourceforge.net/index-eng.html
* (1) Nutch - http://lucene.apache.org/nutch/
* (1) wget - http://www.gnu.org/software/wget/
* (1) WebQL - http://www.ql2.com/academic

Below are all the answers unedited.
Thanks again to all who contributed.

Sérgio Nunes

---
From: Hemant Joshi

Have you looked at Labrador on line at
http://www.dcs.gla.ac.uk/~craigm/labrador/
--
From: Dragomir R. Radev

Have you used poacher?
--
From: haozhi Ye

Larbin is really good for high performance crawling and I have used it
to crawl 10 Million web pages with several machines in a week.

http://larbin.sourceforge.net/index-eng.html

Another one may be Heritrix, but I've never tried it. It's written in
Java so I doubt its performance. http://crawler.archive.org/
--
From: Filippo Menczer

I have compiled a list of resources for a course on Web mining, and
the list includes pointers to a few off-the-shelf crawlers such as
Heritrix: http://informatics.indiana.edu/fil/Class/b659/resources.html#sw
--
From: ChaTo (Carlos Alberto-Alejandro CASTILLO-Ocaranza)

Nutch -- Java, written to work with Lucene, has a partnership with
Apache Foundation

Heritrix -- Java, written for the Internet Archive

WIRE -- C++, written for Web characterization studies [by our group at
CWR.cl]

Check also this list, it's quite complete:
http://en.wikipedia.org/wiki/Web_crawler#Examples_of_Web_crawlers
--
From: Milad Shokouhi

You may also want to have a look at WGET:
http://www.gnu.org/software/wget/
--
From: Nicholas Kushmerick

hi Sergio, you might be interested in WebQL, an industrial-strength Web
data extraction/integration tool from QL2 Software.  Among other features,
WebQL has built-in Web crawling capabilities.

WebQL licenses are available free of charge for non-commercial academic
research/education activities -- see www.ql2.com/academic for details.
--
From: Michel Beigbeder

Never tested any one but I've heard about
 heritrix      http://crawler.archive.org/
 wire          http://www.cwr.cl/projects/WIRE/
I think that Nutch/Lucene also embed a Web crawler.

(Otherwise I used wget and curl but they have not the same goal, I
suppose you know them.)

You should have a look to Castillo's PhD:
       http://www.chato.cl/534/article-63160.html
--