Question: Do APT support streaming download for metadata/deb?
陈 溢飞 <[email protected]>
| Newsgroups | gmane.linux.debian.apt.devel |
|---|---|
| Message-ID | <SY6PR01MB7691E2AA0B82FC6DE1969EF98410A@SY6PR01MB7691.ausprd01.prod.outlook.com> |
Hi, I have been designing an HTTP APT repository server. My `Release` and `Packages` files are automatically generated when an HTTP GET request is received and are sent via streaming without a `Content-Length` header. Currently, cURL can fetch the files successfully, but APT cannot. To investigate, I deployed a simple HTTP server that forwards requests to/from <https://deb.debian.org>, while removing the `Content-Length` header. In this case, both cURL and Wget can fetch the content successfully, but APT still fails. It seems that APT may not support streaming downloads. cURL's output: ```shell $ curl http://127.0.0.1:8080/dists/frankenstein/Release Origin: <Origin> Label: <Label> Suite: unstable Codename: frankenstein Changelogs: https://example.com Date: Sat, 20 Sep 2025 14:58:08 UTC Valid-Until: Sat, 27 Sep 2025 14:58:08 UTC Acquire-By-Hash: no No-Support-for-Architecture-all: Packages Architectures: amd64 amd64 Components: core extra Description: <Desc> ``` APT's output: ``` root@Linux:/# apt update Ign:1 http://127.0.0.1:8080 frankenstein InRelease Get:2 http://127.0.0.1:8080 frankenstein Release [340 B] Ign:3 http://127.0.0.1:8080 frankenstein Release.gpg Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Get:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Get:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Get:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Get:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Err:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages 404 Not Found [IP: 127.0.0.1 8080] Undetermined Error [IP: 127.0.0.1 8080] Fetched 340 B in 8s (42 B/s) All packages are up to date. Warning: No Hash entry in Release file /var/lib/apt/lists/partial/127.0.0.1:8080_dists_frankenstein_Release Warning: Failed to fetch http://127.0.0.1:8080/dists/frankenstein/core/binary-amd64/Packages Undetermined Error [IP: 127.0.0.1 8080] Warning: Some index files failed to download. They have been ignored, or old ones used instead. Notice: Some sources can be modernized. Run 'apt modernize-sources' to do so. ``` Regarding the line `Get:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages`, it seems APT received the data but did not process it correctly. For comparison, I saved the auto-generated files on a static HTTP server for testing. This time, APT updated successfully: ``` Ign:1 http://127.0.0.1:8080 frankenstein InRelease Get:2 http://127.0.0.1:8080 frankenstein Release [319 B] Ign:3 http://127.0.0.1:8080 frankenstein Release.gpg Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Ign:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Get:4 http://127.0.0.1:8080 frankenstein/core amd64 Packages [111 kB] Ign:5 http://127.0.0.1:8080 frankenstein/core Translation-en Fetched 111 kB in 0s (3242 kB/s) 14 packages can be upgraded. Run 'apt list --upgradable' to see them. Warning: No Hash entry in Release file /var/lib/apt/lists/partial/127.0.0.1:8080_dists_frankenstein_Release Notice: Some sources can be modernized. Run 'apt modernize-sources' to do so. ``` So, does APT not support streaming downloads? If not, is adding streaming downloads support a good idea? Best regards,