Re: spider with 100 threads

Am Samstag, den 12.01.2008, 17:16 +0100 schrieb Peter Halacsy:
> hello
> 
> I've to download a million urls on a machine as fast as I can. I've 
> modified the spider in the example directory. But if I add more than 100 
> request to the pipeline it slows down.
> 
> What is the best mode of doing things like this?

Well, you have to check what slows it down. As I've recently written a
crawler for a search engine I know two typical problems. First, it might
be that DNS is the reason. Netclient uses synchronous DNS lookups (i.e.
it has to stop all its work while a lookup is in progress) by default,
but you can plug in asynchronous DNS resolvers (by setting the
"resolver" config). Wink (wink.com) has developed a DNS library that is
able to do async DNS lookups with the help of a real DNS server (see
netdns on oss.wink.com). If you have only a few domains to look up, you
could alternatively also cache domains.

The second thing is the postprocessing. E.g. if you write the downloaded
data to disk, this can be a serious problem, because this is also
synchronous. You can test that by turning any postprocessing off. A
solution is to move all postprocessing to a second process or thread,
and to use an async protocol for interprocess communication (e.g. using
the sunrpc stuff in ocamlnet).

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
[email protected]          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------



-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace