Web crawling software?

Hi,

I'm currently evaluating free web crawling software for several large
crawls that I'm going to perform. Developing my own software is also a
possibility, but I'm trying to avoid the "not invented here" syndrome :)

What software package do you recommend for performing a large web
crawl on "off-the-shelf" hardware?

If you really don't have time to send me a detailed answer, please
just point to the software you've successfully used. I'll do a simple
count.

Please send your answer directly to me and I'll summarize the results
to the list.

Thanks in advance,
Sérgio Nunes