Re: Limit the simultaneous twisted.web.client.downloadPage requests

[email protected] wrote:
> On 01:59 pm, [email protected] wrote:
>>> Thanks for quick and great help, Terry and Jean-Paul!
>> You're welcome. If in any doubt about conflicting advice, do what JP 
>> suggests.
>>
>> The reason I pointed you to JP's Cooperator solution was that when you
>> extract the set of links from a page that you also want to fetch, you 
>> could
>> make another generator and submit that to the same cooperator's 
>> coiterate
>> function. That would just schedule the additional getPage calls to be 
>> run.
>>
>> The only wrinkle (I think) in that is that in doing the very last 
>> getPage,
>> the cooperator will run out of more work and fire the deferred you got
>> originally. But the final page may contain more links. I'm not sure if 
>> I'm
>> thinking clearly about that final bit. That's your job :-)
> 
> cooperator and DeferredSemaphore are probably both good solutions. 
> There's not much reason to prefer one over the other. :)
> 
> Jean-Paul
> 

That list seems like far from the most active one, but the one filled 
with friendly people :) The thing I like when dealing with Twisted and 
it's documentation is that it is seen often that people who make it, and 
who use it, really love Twisted.