Re: spider with 100 threads

Gerd Stolpmann wrote:
> Am Samstag, den 12.01.2008, 17:16 +0100 schrieb Peter Halacsy:
>> hello
>>
>> I've to download a million urls on a machine as fast as I can. I've 
>> modified the spider in the example directory. But if I add more than 100 
>> request to the pipeline it slows down.
>>
>> What is the best mode of doing things like this?
> 
> Well, you have to check what slows it down. [...]

> help of a real DNS server (see
> netdns on oss.wink.com). If you have only a few domains to look up, you
> could alternatively also cache domains.
>

Well, I've cached the IP addresses and now I can download 15 pages/sec 
on my ADSL line. I'll try it on a server of my university.


> The second thing is the postprocessing. E.g. if you write the downloaded
> data to disk, this can be a serious problem, because this is also
> synchronous.

Yes, now I switched off the serialization. I'll test if I can compile 
ocamlnet on linux.

peter


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/