Re: Large-scale spidering
[email protected] (Randal L. Schwartz)
| Newsgroups | gmane.comp.lang.perl.modules.lwp |
|---|---|
| Organization | Stonehenge Consulting Services; Portland, Oregon, USA |
| Message-ID | <[email protected]> |
>>>>> "Marvin" == Marvin Humphrey <[email protected]> writes: Marvin> I've written an industrial-strength search engine library for Perl Marvin> (KinoSearch), and now I have clients who want me to work on a large- Marvin> scale spidering app for them. Sort of like Nutch for Perl (<http:// Marvin> lucene.apache.org/nutch>). Putch. :) The original Inktomi spider was written in Perl, and even available as public open source for a while. "Back in the day." Is this for a private network? Please tell me you're not trying to build Yet Another Spider to visit my site! If so, please use the Google or Yahoo API to leverage the fact that they've already done an excellent job of visiting a few thousand URLs, including 40,000 images. You don't need to come here. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 <[email protected]> <URL:http://www.stonehenge.com/merlyn/> Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!