Re: Large-scale spidering

Plucene maybe? Its up on CPAN.

Justin

Marvin Humphrey wrote:

> Greets,
>
> I've written an industrial-strength search engine library for Perl  
> (KinoSearch), and now I have clients who want me to work on a large- 
> scale spidering app for them.  Sort of like Nutch for Perl (<http:// 
> lucene.apache.org/nutch>).  Putch.  :)
>
> What efforts have already been undertaken in this area?  A survey of  
> existing CPAN releases that I should study would be great.  I've  
> written a small-scale spider using LWP::RobotUA.  I've scanned over  
> the WWW::Mechanize docs, but don't yet grasp its full capabilities.   
> What else?
>
> Thanks,
>
> Marvin Humphrey
> Rectangular Research
> http://www.rectangular.com/
>
>