Re: lzma running out of memory on qemu-user (#846564)

Hi Guillem,

On Thu, 2024-09-26 at 13:20 +0200, Guillem Jover wrote:
> On Thu, 2024-09-26 at 10:31:55 +0200, John Paul Adrian Glaubitz wrote:
> > I'm using qemu-user with a Debian unstable sh4 chroot on a 40-core Xeon machine
> > and running dpkg-deb fails with the same issue reported in #846564 [1]:
> > 
> > dpkg-deb: building package 'gcc-snapshot' in '../gcc-snapshot_20240922-1+sh4.1_sh4.deb'.
> > dpkg-deb: building package 'gcc-snapshot-dbgsym' in '../gcc-snapshot-dbgsym_20240922-1+sh4.1_sh4.deb'.
> > dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate memory
> > dpkg-deb: error: <compress> from tar -cf subprocess returned error exit status 2
> > dh_builddeb: error: dpkg-deb --root-owner-group --build debian/.debhelper/gcc-snapshot/dbgsym-root .. returned exit code 2
> > dh_builddeb: error: Aborting due to earlier error
> 
> Hmm, the heuristics in the code should in theory prevent this, so it
> would be nice to know what is going wrong with that, in case it could
> be improved. See the filter_xz_get_memlimit() and
> filter_xz_get_cputhreads() functions in lib/dpkg/.

You should be able to reproduce it with qemu-user on a machine with many cores.
Any emulated 32-bit architecture should work.

> > Is there a way to tune lzma such that it reduces memory consumption in this case?
> 
> You should be able to globally reduce the amount of threads used with
> the DPKG_DEB_THREADS_MAX envvar (as the --threads-max option would
> need to be passed from debian/rules or debhelper or similar).

Thanks a lot for the suggestion. For anyone else running into this problem,
the following workaround added to debian/rules did the trick to me:

# Workaround for #846564
override_dh_builddeb:
        dh_builddeb -- --threads-max=16

Thanks,
Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913