Re: Limit the simultaneous twisted.web.client.downloadPage requests

On Oct 24, 2009, at 6:20 AM, Igor Katson wrote:

> Is my idea about a "wrapper" practical and what's the general way to
> write it?
> On which event is it better to decrement the counter of the amount
> currently downloading files?


Another way that you might do this is by using this small snippet:

http://bitbucket.org/adroll/turtl/src/tip/turtl/engine.py

turtl is a project that you can use either as a proxy server in front
of all of your clients (and throttle the requests on a url basis) or
embedded in your system like this:

                 from turtl import engine
                 thr = engine.ThrottlingDeferred(parallelism, calls,  
interval)
                 dl = [thr.run(callable, *args, **kwargs) for args in  
self.args]
                 defer.DeferredList(dl).addBoth(lambda _: reactor.stop 
())

In this case you get a lot more control over the number of calls in a  
given
interval that you are allowed to make (for example Amazon Alexa allows
only 15 calls per second).

-- 
Valentino Volonghi aka Dialtone
Now Running MacOSX 10.6
http://www.adroll.com/