Re: render_GET and memory consumption

On 10-12-21 02:53 PM, Pedro I. Sanchez wrote:
> On 10-12-21 02:24 PM, [email protected] wrote:
>> On 07:02 pm, [email protected] wrote:
>>> On 10-12-21 12:47 PM, [email protected] wrote:
>>>> On 04:43 pm, [email protected] wrote:
>>>>>
>>>>> OK, I guess I'm being slow :-( Here's another version, same results.
>>>>>
>>>>> import os
>>>> >from twisted.internet import reactor
>>>> >from twisted.web.server import Site
>>>> >from twisted.web.resource import Resource
>>>>>
>>>>> CHUNK_SIZE = 32*1024
>>>>> data = os.urandom(10*1024*1024)
>>>>> chunks = []
>>>>>
>>>>> def make_chunks():
>>>>>        s = 0
>>>>>        for chunk in iter(lambda: data[s:s+CHUNK_SIZE], ''):
>>>>>            chunks.append(chunk)
>>>>>            s = s + CHUNK_SIZE
>>>>>
>>>>> class TestPage(Resource):
>>>>>         isLeaf = True
>>>>>
>>>>>         def render_GET(self, request):
>>>>>             for chunk in chunks:
>>>>>                 request.write(chunk)
>>>>>
>>>>> make_chunks()
>>>>> root = Resource()
>>>>> root.putChild('test', TestPage())
>>>>> reactor.listenTCP(8880, Site(root))
>>>>> reactor.run()
>>>>
>>>> This version has flat memory usage on my system - 33MB all the way
>>>> through.
>>>> Jean-Paul
>>>
>>> Did you try this running the clients on a different machine than the
>>> one
>>> running the server? I find that if I run both the server and the
>>> clients
>>> (using httperf) on the same machine things behave differently. I can
>>> only see the memory consumption problem consistently when running the
>>> clients on a separate machine (which would be my real use case).
>>>
>>> BTW, I'm running these test on Ubuntu 10.04, Python 2.6.5, twisted
>>> 10.0.0-2ubuntu2.
>>
>> Argh.  I just ran it over loopback, so I wasn't accurately testing the
>> case you described.  James Knight just reminded me that the transport
>> will still find a way to copy data in this case, so you're right that
>> this still isn't a working solution.  Instead, you have to go all the
>> way to producers/consumers, and only write more data to the transport
>> buffer when it has finished dealing with what you previously gave it.
>>
>> Request implements IConsumer, which means you can pass an IProducer
>> provider to its registerProducer method.  When the send buffer gets
>> close to empty, the IProducer's resumeProducing method will be called
>> and you can write some more bytes to the Request.
>>
>> There are some more complications and subtleties in the
>> IProducer/IConsumer interfaces, which you can read about at
>> <http://twistedmatrix.com/documents/current/core/howto/producers.html>.
>>
>> Basically, you'll end up with something like:
>>
>>       from twisted.python.log import err
>>       from twisted.protocols.basic import FileSender
>>       from twisted.web.server import NOT_DONE_YET
>>
>>       def render_GET(self, request):
>>           producer = FileSender()
>>           d = producer.beginFileTransfer(StringIO(chunk), request)
>>           def finished(ignored):
>>               request.finish()
>>           d.addErrback(err, "Streaming data to client failed")
>>           d.addCallback(finished)
>>           return NOT_DONE_YET
>>
>> Sorry for the earlier mis-information.
>>
>> Jean-Paul
>>
>> _______________________________________________
>> Twisted-web mailing list
>> [email protected]
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
>
> Thanks again Jean-Paul,
>
> This one will take me some time to digest and prototype. I'll give it a
> whirl to try to implement a solution and I'll post back my results.
>

dada! This one works!

import os
from StringIO import StringIO
from twisted.python.log import err
from twisted.internet import reactor
from twisted.web.server import Site, NOT_DONE_YET
from twisted.web.resource import Resource
from twisted.protocols.basic import FileSender

data = os.urandom(10*1024*1024)

class TestPage(Resource):
     isLeaf = True

     def _cb_chunk_done(self, ignore, request):
         request.finish()

     def render_GET(self, request):
         producer = FileSender()
         d = producer.beginFileTransfer(StringIO(data), request)
         d.addCallback(self._cb_chunk_done, request)
         d.addErrback(err, "Streaming data to client failed")
         return NOT_DONE_YET

root = Resource()
root.putChild('test', TestPage())
reactor.listenTCP(8880, Site(root))
reactor.run()

Thanks again for your time, much appreciated!

-- 
Pedro