Re: Large File Download

You can effectively stream a file byte by byte - you just need to print 
a chunk at a time and mod_perl and apache will handle it 
appropriately... I do this all the time to handle large data downloads 
(the systems I manage are backed by peta bytes of data)...

The art is often not in the output - but in the way you get and process 
data before sending it - I have code that will upload/download arbitrary 
large files (using HTML5's file objects) without using excessive amounts 
of memory... (all data is stored in chunks in a MySQL database)

Streaming has other advantages with large data - if you wait till you 
generate all the data then you will find that you often get a time out - 
I have a script which can take up to 2 hours to generate all the output 
- but it never times out as it is sending a line of data at a time.... 
and do data is sent every 5-10 seconds... and the memory footprint is 
trivial - as only data for one line of output is in memory at a time..


On 28/03/2015 16:25, John Dunlap wrote:
> sendfile sounds like its exactly what I'm looking for. I see it in the 
> API documentation for Apache2::RequestIO but how do I get a reference 
> to it from the reference to Apache2::RequestRec which is passed to my 
> handler?
>
> On Sat, Mar 28, 2015 at 9:54 AM, Perrin Harkins <[email protected] 
> <mailto:[email protected]>> wrote:
>
>     Yeah, sendfile() is how I've done this in the past, although I was
>     using mod_perl 1.x for it.
>
>     On Sat, Mar 28, 2015 at 5:55 AM, André Warnier <[email protected]
>     <mailto:[email protected]>> wrote:
>
>         Randolf Richardson wrote:
>
>                 I know that it's possible(and arguably best practice)
>                 to use Apache to
>                 download large files efficiently and quickly, without
>                 passing them through
>                 mod_perl. However, the data I need to download from my
>                 application is both
>                 dynamically generated and sensitive so I cannot expose
>                 it to the internet
>                 for anonymous download via Apache. So, I'm wondering
>                 if mod_perl has a
>                 capability similar to the output stream of a java
>                 servlet. Specifically, I
>                 want to return bits and pieces of the file at a time
>                 over the wire so that
>                 I can avoid loading the entire file into memory prior
>                 to sending it to the
>                 browser. Currently, I'm loading the entire file into
>                 memory before sending
>                 it and
>
>                 Is this possible with mod_perl and, if so, how should
>                 I go about
>                 implementing it?
>
>
>                     Yes, it is possible -- instead of loading the
>             entire contents of a file into RAM, just read blocks in a
>             loop and keep sending them until you reach EoF (End of File).
>
>                     You can also use $r->flush along the way if you
>             like, but as I understand it this isn't necessary because
>             Apache HTTPd will send the data as soon as its internal
>             buffers contain enough data.  Of course, if you can tune
>             your block size in your loop to match Apache's output
>             buffer size, then that will probably help.  (I don't know
>             much about the details of Apache's output buffers because
>             I've not read up too much on them, so I hope my
>             assumptions about this are correct.)
>
>                     One of the added benefits you get from using a
>             loop is that you can also implement rate limiting if that
>             becomes useful.  You can certainly also implement access
>             controls as well by cross-checking the file being sent
>             with whatever internal database queries you'd normally use
>             to ensure it's okay to send the file first.
>
>
>         You can also :
>         1) write the data to a file
>         2) $r->sendfile(...);
>         3) add a cleanup handler, to delete the file when the request
>         has been served.
>         See here for details :
>         http://perl.apache.org/docs/2.0/api/Apache2/RequestIO.html#C_sendfile_
>
>         For this to work, there is an Apache configuration directive
>         which must be set to "on". I believe it is called "UseSendFile".
>         Essentially what senfile() does, is to delegate the actual
>         reading and sending of the file to Apache httpd and the
>         underlying OS, using code which is specifically optimised for
>         this purpose. It is much kore efficient than doing this in a
>         read/write loop by yourself, at the cost of having less fine
>         control over the operation.
>
>
>
>
>
> -- 
> John Dunlap
> /CTO | Lariat/
> /
> /
> /*Direct:*/
> /[email protected] <mailto:[email protected]>/
> /
> *Customer Service:*/
> 877.268.6667
> [email protected] <mailto:[email protected]>



---
This email has been checked for viruses by Avast antivirus software.
http://www.avast.com



-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.