[PATCH v2 0/8] bootm: size the noload buffer from the compressor header

This is v2 of "bootm: size the noload decompression buffer from the
compressor header". Tom pushed back on v1
(https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/)
on two grounds:

  1. No concrete problem report driving the change.
  2. ~1297 platforms grew by ~170-400 bytes; the change is not
     opt-in, so the size cost falls on everyone.

On the first point, Nora Schiffer replied with a concrete use case
(EFI-in-FIT plus padded loaders such as shim, systemd-boot, and
OpenWrt's lzma-loader can produce compression ratios that outrun the
8x heuristic), and mentioned this is on the road map for TQ-Systems
standard BSPs.

On the second point, v2 reworks the implementation to cut the size
cost, measures it across the format and architecture buckets, and
splits the work per format so each decompressor's support can be
taken or dropped on its own.

Background: for a compressed kernel_noload image, bootm_load_os()
sizes the decompression buffer as ALIGN(image_len * 8, SZ_1M). The
8x heuristic works for typical kernels, but any well-compressed
payload can exceed it, and no fixed multiplier is safe against
arbitrarily compressible input.

Each implementation patch adds a small static header-parse helper in
bootm.c (no new public API) and wires it into a size-hint switch;
helper and switch case are only compiled when the matching
decompressor is enabled, so boards that do not build a format pay no
code for it. gzip's ISIZE is a fixed trailer read, lzma's size a
fixed header read, lz4 mirrors ulz4fn()'s frame-header validation,
and zstd asks zstd_get_frame_header(), whose frame-parsing code
already ships with the zstd decompressor. The header-recorded value
is attacker-controlled, so it is capped at CONFIG_SYS_BOOTM_LEN, and
it is only an allocation hint: the decoder stays authoritative
during the actual decompression.

Text size deltas of the u-boot ELF (size(1), distro gcc 13.3 cross
toolchains); data/bss are unchanged everywhere. To make the columns
directly comparable, the v1 column is v1's implementation commit
cherry-picked onto this series' base, so both columns share one
baseline:

  board          arch     decompressors        v1        v2
  qemu_arm       arm      gzip                +160      +104
  qemu-ppce500   powerpc  gzip                +176      +112
  mt7623n_bpir2  arm      gzip+lzma           +184      +128
  qemu_arm64     arm64    gzip+lzma+lz4       +384      +368
  qemu-riscv64   riscv64  gzip+lzma+lz4       +332      +352
  th1520_lpi4a   riscv64  all four            +412      +404
  am62x_evm_a53  arm64    all four, LTO       +0 *    +8192 *
  qemu-x86       x86      none                +108        -2

  * am62x_evm_a53's number is dominated by the Cortex-A53 erratum
    843419 linker workaround (default-enabled in distro binutils for
    aarch64): symbol-level code growth (nm -S) is +392 for v1 and
    +452 for v2, but those bytes shift which ADRP instructions land
    at the erratum's page offsets, and ld pads each inserted veneer
    to a full 4 KiB page. v2 happens to trigger two such pages here;
    v1 triggered the same two on its own original base and none on
    this one. See the world-build note below.

To see how much each bucket weighs, I configured all 1550 defconfigs
and sorted them by which decompressors they enable next to bootm:

  858  gzip only (722 of them arm, essentially the 32-bit boards)
  530  gzip+lzma+lz4 (494 arm, mostly arm64, plus 36 riscv)
   38  gzip+lzma
   31  bootm with no decompressor at all
   27  gzip+lzma+lz4+zstd
   19  gzip+lz4
   15  gzip+zstd
    6  other combinations
   26  do not link bootm at all

To measure at the same scale as the original objection, I also ran a
full world build (buildman, all 1550 defconfigs, distro plus
kernel.org toolchains, gcc 13.3/14.2) over one branch holding the
base, the v1 implementation, its revert, and this series. 1496
boards built on all four commits with the revert reproducing the
base sizes exactly (44 boards did not build on every commit, and 10
built nondeterministically; both sets were excluded). Of those 1496,
the same 1375 change under either version and the rest are
untouched, including every board without bootm or without a
decompressor:

                                        v1            v2
  mean delta over all boards          +194 B        +166 B
  median delta (changed boards)       +160 B         +96 B
  median, 856 gzip-only boards        +112 B         +80 B
  median, 517 gzip+lzma+lz4 boards    +392 B        +376 B
  boards cheaper with v2                   -          1222
  boards costlier with v2                  -            74

The world build also puts the am62x footnote in proportion: 53
boards under v1 and 49 under v2 (28 in both sets), all arm64, show
size(1) jumps of one or two 4 KiB pages in either direction (min
-8192, max +8192). The mechanism is the Cortex-A53 erratum 843419
linker workaround: when a code change shifts which ADRP instructions
land at page offsets 0xff8/0xffc, ld materialises a 16-byte veneer
and pads it to a full 4 KiB page so the page offsets of all
downstream code stay unchanged. Any few-hundred-byte change re-rolls
which boards are affected, in both directions; symbol-level growth
on every such board I checked matches the byte ranges above.

Since Tom noted the higher growth in his run was on multi-algorithm
platforms: building each patch in sequence on a gzip+lzma+lz4 board
(qemu_arm64) and an all-four board (th1520_lpi4a) gives the
per-format cost directly, in bytes:

               qemu_arm64    th1520_lpi4a
  gzip             +112             +86
  zstd               +0             +62
  lz4              +176            +176
  lzma              +80             +80

(zstd is +0 on qemu_arm64 because that board does not enable it, so
the guard really does compile the helper out.) lz4 is the most
expensive parser because it mirrors ulz4fn()'s frame validation;
zstd is the cheapest because zstd_get_frame_header() already ships
with the decompressor. Since the series is split per format, if the
multi-algorithm cost still looks too high, dropping the lz4 patch
alone would cut the 517-board gzip+lzma+lz4 bucket from a median of
+376 to roughly +200; lz4 images then simply keep the 8x fallback.

In the v1 thread Simon suggested recording the uncompressed size as
a FIT property instead. As discussed there, the two compose: a FIT
property could be layered on top later, with bootm preferring the
property, then the stream header, then the 8x fallback. This series
provides the part that works for every existing image and for the
legacy uImage form of kernel_noload.

Series layout, one decompressor at a time:

  1. gzip helper + wiring
  2. gzip pytests (lying-header overflow, header-sized, boundary)
  3. zstd helper
  4. zstd pytest (guarded by requiredtool zstd)
  5. lz4 helper
  6. lz4 pytest (guarded by requiredtool lz4)
  7. lzma helper
  8. lzma pytests (real size patched into the header field, plus the
     "unknown" size marker fallback; needs no external tool since
     Python's lzma module is in the standard library)

Every patch builds in isolation on sandbox_defconfig and
qemu_arm_defconfig; the seven kernel_noload_decomp pytests and the
full test_fit class pass on sandbox.

Changes in v2:
- split the single implementation patch into per-format patches
  (gzip, zstd, lz4, lzma), each acceptable or droppable on its own
- make the helpers static in bootm.c, compiled only when the
  matching decompressor is enabled, instead of one always-built
  public image_decomp_get_uncompressed_size() in boot/image.c; this
  removes the cost from boards without the formats entirely
- regroup the pytests per format, next to the patch they exercise
- add runtime lzma coverage (header-recorded size and the "unknown"
  marker fallback) in place of v1's parser-only C unit test, which
  cannot reach a static helper
- measure the size cost on a common base across all 1550 boards and
  document the per-bucket numbers and the arm64 erratum-843419 page
  effect in this cover letter

Aristo Chen (8):
  bootm: size the noload gzip decompression buffer from ISIZE
  test: fit: cover the kernel_noload gzip header-size and lying-header
    paths
  bootm: size the noload zstd decompression buffer from
    Frame_Content_Size
  test: fit: cover the kernel_noload zstd header-size path
  bootm: size the noload lz4 decompression buffer from Content_Size
  test: fit: cover the kernel_noload lz4 header-size path
  bootm: size the noload lzma decompression buffer from the header
  test: fit: cover the kernel_noload lzma header-size and unknown-size
    paths

 boot/bootm.c              | 144 ++++++++++++++++++++++-
 test/py/tests/test_fit.py | 282 ++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 400 insertions(+), 26 deletions(-)

--
2.43.0