Re: Large-scale spidering

On Apr 14, 2006, at 4:26 AM, J Cook wrote:

> Plucene maybe? Its up on CPAN.

I'm intimately acquainted with Plucene.  I actually spent a week or  
two hacking on it last August before deciding that its performance  
issues could not be resolved without a complete overhaul which would  
break the API.

    http://www.rectangular.com/kinosearch/benchmarks.html

KinoSearch, like Plucene, is a text search engine library.  In order  
to write an industrial-strength spider a la Nutch, you need a lot  
more than that:  HTML::Parser, HTML::LinkExtor, LWP::RobotUA...  I've  
now discovered WWW::RobotRules::AnyDBM_File, which is going to be  
very helpful.  But there are a lot of other problems to be solved.   
Check-summing page content to eliminate duplicate documents available  
via multiple URLs.  Managing crawl depth so that a spider doesn't  
venture too deep into one domain and forget about all the others.   
Eventually, if you want to get fancy, link analysis and pagerank.

LWP::Parallel::RobotUA looks interesting.  There's a bunch of stuff  
under Bundle::LinkController, but it hasn't been updated in a while.   
What else?

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/