Re: fetch on pkg.freebsd.org does not handle If-Modified-Since properly

Hi Ryan,

It is happier. If-Modified-Since is now properly handled. Thank you.
Would you mind sharing what you changed? I noticed that the server is now nginx instead of varnish. I don't know how Fastly works, but it would be nice if the config was in an open repository.

Regards,
Ronald.

 
Van: Ryan Steinmetz <[email protected]>
Datum: zaterdag, 10 januari 2026 17:32
Aan: Ronald Klop <[email protected]>
CC: [email protected]
Onderwerp: Re: fetch on pkg.freebsd.org does not handle If-Modified-Since properly
> 
> This should be happier now.
> 
> -r
> 
> On (01/10/26 15:10), Ronald Klop wrote:
> >Hi,
> >
> >I have a script which regularly fetch some files from pkg.freebsd.org for keeping statistics. (https://www.klop.ws/pkgstats/pkg-age.html)
> >
> >The command it executes is: "fetch -vv -o /home/ronald/verify_pkgs.sh/downloads/FreeBSD:13:armv7_latest_data.pkg --if-modified-since=/home/ronald/verify_pkgs.sh/downloads/FreeBSD:13:armv7_latest_data.pkg https://pkg.freebsd.org/FreeBSD:13:armv7/latest/data.pkg"
> >
> >Mind that the request header If-Modified-Since has the exact same time as the response header Last-Modified.
> >If-Modified-Since: Sun, 05 Oct 2025 04:47:06 GMT
> >Last-Modified: Sun, 05 Oct 2025 04:47:06 GMT
> >
> >Still the response says HTTP/1.1 200 OK instead of 304 Not Modified.
> >
> >I've had this in my projects if the File.lastModified() returns a date with milliseconds, while the http-request time always uses 0 for milliseconds. In that case the file is always newer.
> >NB: not all fetch requests have this behaviour. Some give a 304 response.
> >
> >Anyway, maybe the cause is something else. Would be nice to fix this optimization.
> >
> >Here the verbose output of the fetch command:
> >scheme:   "https"
> >user:     ""
> >password: ""
> >host:     "pkg.freebsd.org"
> >port:     "0"
> >document: "/FreeBSD:13:armv7/latest/data.pkg"
> >---> pkg.freebsd.org:443
> >resolving server address: pkg.freebsd.org:443
> >SSL options: 82004850
> >Peer verification enabled
> >Using OpenSSL default CA cert file and path
> >Verify hostname
> >TLSv1.3 connection established using TLS_AES_128_GCM_SHA256
> >Certificate subject: /CN=pkg.freebsd.org
> >Certificate issuer: /C=US/O=Certainly/CN=Certainly Intermediate R1
> >requesting https://pkg.freebsd.org/FreeBSD:13:armv7/latest/data.pkg
> >>>>GET /FreeBSD:13:armv7/latest/data.pkg HTTP/1.1
> >If-Modified-Since: Sun, 05 Oct 2025 04:47:06 GMT
> >>>>If-Modified-Since: Sun, 05 Oct 2025 04:47:06 GMT
> >>>>Host: pkg.freebsd.org
> >>>>Accept: */*
> >>>>User-Agent: fetch libfetch/2.0
> >>>>Connection: close
> >>>>
> ><<< HTTP/1.1 200 OK
> >
> ><<< Connection: close
> >
> ><<< Content-Length: 9137918
> >
> ><<< Server: nginx
> >
> >content length: [9137918]
> ><<< Content-Type: application/octet-stream
> >
> ><<< Last-Modified: Sun, 05 Oct 2025 04:47:06 GMT
> >
> ><<< ETag: "68e1f84a-8b6efe"
> >
> >last modified: [2025-10-05 04:47:06]
> ><<< X-Content-Type-Options: nosniff
> >
> ><<< X-XSS-Protection: 1; mode=block
> >
> ><<< X-Frame-Options: SAMEORIGIN
> >
> ><<< Accept-Ranges: bytes
> >
> ><<< Via: 1.1 varnish, 1.1 varnish
> >
> ><<< Date: Sat, 10 Jan 2026 11:04:02 GMT
> >
> ><<< X-Served-By: cache-chi-kigq8000150-CHI, cache-ams2100142-AMS
> >
> ><<< X-Cache: MISS, MISS
> >
> ><<< X-Cache-Hits: 0, 0
> >
> ><<< X-Timer: S1768043043.639628,VS0,VE106
> >
> ><<< Cache-Control: no-cache,no-store,max-age=0
> >
> ><<<
> >
> >offset 0, length -1, size -1, clength 9137918
> >local size / mtime: 9137918 / 1759639626
> >remote size / mtime: 9137918 / 1759639626
> >/home/ronald/verify_pkgs.sh/downloads/FreeBSD:13:armv7_latest_d        8923 kB   20 MBps    01s
> >
> >
> >Regards,
> >Ronald.
> >
> 
> -- 
> Ryan Steinmetz
> PGP: 4143 0BDC 21B0 B3D9 6D47  F0DA 504D C825 7FAD 8C58
> 
> 
>