fnord serves large files minus 4096 bytes

Henrik Langos <[email protected]> Mon, 23 Sep 2013 18:49:08 +0200
Newsgroups gmane.comp.web.fnord
Message-ID <[email protected]>
Hi there,

I've just found a strange behavior in fnord when serving a large file.
Here's what wget says: (domain names, IP addresses and file paths 
changed to protect the guilty ;-)

========================================


$ wget -S http://foo:[email protected]/path-to/large.zip
--2013-09-23 18:19:04--  http://foo:*password*@example.com/path-to/large.zip
Resolving example.com... 192.168.1.70
Connecting to example.com|192.168.1.70|:80... connected.
HTTP request sent, awaiting response...
   HTTP/1.0 401 Authorization Required
   WWW-Authenticate: Basic realm="example.com:80"
   Connection: close
Connecting to example.com|192.168.1.70|:80... connected.
HTTP request sent, awaiting response...
   HTTP/1.0 200 OK
   Server: fnord/1.10-4
   Content-Type: application/zip
   Content-Length: 2355798059
   Last-Modified: Fri, 06 Sep 2013 13:58:48 GMT
Length: 2355798059 (2.2G) [application/zip]
Saving to: `mdm-vm-demo.zip.2'

99% 
[===================================================================================================================================================================================================================================================> 
] 2,355,793,963 48.3M/s   in 48s

2013-09-23 18:19:52 (47.2 MB/s) - Connection closed at byte 2355793963. 
Retrying.

--2013-09-23 18:19:53--  (try: 2)  
http://foo:*password*@example.com/path-to/large.zip
Connecting to example.com|192.168.1.70|:80... connected.
HTTP request sent, awaiting response...
   HTTP/1.0 401 Authorization Required
   WWW-Authenticate: Basic realm="example.com:80"
   Connection: close
Connecting to example.com|192.168.1.70|:80... connected.
HTTP request sent, awaiting response...
   HTTP/1.0 206 Partial Content
   Server: fnord/1.10-4
   Content-Type: application/zip
   Content-Length: 4096
   Last-Modified: Fri, 06 Sep 2013 13:58:48 GMT
   Accept-Ranges: bytes
   Content-Range: bytes 2355793963-2355798058/2355798059
Length: 2355798059 (2.2G), 4096 (4.0K) remaining [application/zip]
Saving to: `mdm-vm-demo.zip.2'

100%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>] 
2,355,798,059 --.-K/s   in 0s

2013-09-23 18:19:53 (244 MB/s) - `mdm-vm-demo.zip.2' saved 
[2355798059/2355798059]

$

========================================

As you can see it seems like fnord will only send file size minus 4096 
bytes.

wget insists on getting the date that has been announced in the original 
Content-Length header, but unfortunately most browsers are not that keen 
on getting their data. Firefox for example will happily announce a 
complete download even though it is short 4096 bytes.

The behavior is normal for smaller files located at the same directory, 
meaning you get the file in one go. The fnord installation has not be 
changed in years so I an rule out recent changes it the environment.
I have not yet tried to find out the exact limit at which the behavior 
changes but my wild guess would be somewhere around 2^31.

Can anybody confirm this behavior? Or may it be a system specific thing?

The system where fnord is installed is a Debian 5.0, amd64, Xen 
paravirtualized guest(DomU).
# file fnord
fnord: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically 
linked, stripped
# uname -a
Linux example.com 2.6.26-2-xen-amd64 #1 SMP Mon Jun 13 18:44:16 UTC 2011 
x86_64 GNU/Linux
#

best regards
-henrik