Re: Large File Download

On 28/03/2015 19:54, Issac Goldstand wrote:
> sendfile is much more efficient than that.  At the most basic level,
> sendfile allows a file to be streamed directly from the block device (or
> OS cache) to the network, all in kernel-space (see sendfile(2)).
>
> What you describe below is less effective, since you need to ask the
> kernel to read the data, chunk-by-chunk, send it to userspace, and then
> from userspace back to kernel space to be sent to the net.
>
> Beyond that, the Apache output filter stack is also spending time
> examining your data, possibly buffering it differently than you are (for
> example to make HTTP chunked-encoding) - by using sendfile, you'll be
> bypassing the output filter chain (for the request, at least;
> connection/protocol filters, such as HTTPS encryption will still get in
> the way, but you probably want that to happen :)) further optimizing the
> output.
>
> If you're manipulating data, you need to stream yourself, but if you
> have data on the disk and can serve it as-is, sendfile will almost
> always perform much, much, much better.
>
In the cases I was pointing out (in line with the original request) that 
streaming the
data is more efficient that writing it to disk and then using 
sendfile.... (reading is
more efficient than writing - from experience)

Depends on the cost of producing the file - the end time response for 
the user may
well be less by streaming it out than writing it to disk - and then 
sending it to the
user with sendfile - they will already have most of the file (network 
permitting)
before the last chunk of content is produced....

I won't re-iterate the rest of the issues with memory management that 
are achieved
but also I will also point out that if you ever write a file to disk you 
are putting your
server at risk - either from a security or dos attack point of view - so 
if you can avoid
it - do so!


---
This email has been checked for viruses by Avast antivirus software.
http://www.avast.com



-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.