Large-scale spidering

Greets,

I've written an industrial-strength search engine library for Perl  
(KinoSearch), and now I have clients who want me to work on a large- 
scale spidering app for them.  Sort of like Nutch for Perl (<http:// 
lucene.apache.org/nutch>).  Putch.  :)

What efforts have already been undertaken in this area?  A survey of  
existing CPAN releases that I should study would be great.  I've  
written a small-scale spider using LWP::RobotUA.  I've scanned over  
the WWW::Mechanize docs, but don't yet grasp its full capabilities.   
What else?

Thanks,

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/