[RFC/PATCH 0/1] Extract uncompressed data.tar with copy_file_range()

Hi,

I would appreciate feedback on whether this approach would be considered at
all.  The optimization requires changing the private IPC architecture between
dpkg and dpkg-deb, so it is more intrusive than the copy operation itself.
Before refining it further, I would like to know whether that tradeoff is
acceptable to the dpkg maintainers, or whether a different architectural
boundary would be preferred.

The primary motivation is building many OS images from the same set of
uncompressed packages.  When the package archive and image roots reside on a
reflink-capable filesystem, copy_file_range() can let the extracted files
share their underlying extents with the data.tar member in the package
archive.  An image builder repeatedly installing the same packages into many
roots can then avoid storing another physical copy of all those files for
every image.  Depending on the workload, this has the potential to reduce
disk usage massively, in addition to avoiding userspace copying and reducing
the time spent extracting packages.

Realizing the reflink benefit also requires regular-file payloads to be
aligned suitably in the uncompressed tar stream.  The proposed GNU tar
--align=SIZE support provides that archive layout by padding PAX extended
headers so file data starts at the requested boundary:

  https://lists.gnu.org/archive/html/bug-tar/2026-07/msg00003.html

The dpkg copy_file_range() support does not depend on that option for
correctness, but unaligned archives will generally miss the intended extent
sharing optimization.  Once the GNU tar patch is merged, dpkg will therefore
also need to expose the alignment control when creating packages, so builders
can request appropriately aligned, uncompressed data.tar members.

To make this possible, the patch adds a private protocol through which
dpkg-deb can expose a seekable, bounded range for an uncompressed data.tar
member.  dpkg tracks that bound while parsing the tar stream and attempts
copy_file_range() only when installing a regular file directly.  Compressed
archives, non-seekable input, unsupported filesystems, and older dpkg-deb
implementations continue to use the existing streaming path.

In particular, feedback would be welcome on:

  * whether optimizing the uncompressed-package image-building use case is
    something dpkg would be willing to support;
  * whether extending the private dpkg/dpkg-deb IPC in this way is acceptable;
    and
  * whether there is a preferred alternative for exposing the raw data.tar
    member without adding this protocol.

The patch includes fallback, compatibility, malformed-member, protocol, and
end-to-end tests.

Daan De Meyer (1):
  dpkg: Extract uncompressed tar members with copy_file_range

 configure.ac               |   1 +
 lib/dpkg/Makefile.am       |   1 +
 lib/dpkg/fdio.c            |  85 ++++++++++++++
 lib/dpkg/fdio.h            |  19 +++
 lib/dpkg/libdpkg.map       |   1 +
 lib/dpkg/t/t-fdio.c        | 232 +++++++++++++++++++++++++++++++++++++
 src/Makefile.am            |   2 +
 src/at/deb-streaming.at    |  53 +++++++++
 src/at/local.at            |   1 +
 src/common/fsys-tarfile.h  |  28 +++++
 src/deb/extract.c          |  76 +++++++++++-
 src/main/archives.c        | 116 ++++++++++++++++---
 src/main/archives.h        |  11 +-
 src/main/unpack.c          | 140 +++++++++++++++++++++-
 tests/t-filtering/Makefile |  60 ++++++++++
 15 files changed, 800 insertions(+), 26 deletions(-)
 create mode 100644 lib/dpkg/t/t-fdio.c
 create mode 100644 src/common/fsys-tarfile.h