spider with 100 threads

hello

I've to download a million urls on a machine as fast as I can. I've 
modified the spider in the example directory. But if I add more than 100 
request to the pipeline it slows down.

What is the best mode of doing things like this?

peter


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace