Re: Bug#876416: libzstd FTBFS on mips(el): Allocation error : not enough memory

James Cowgill <[email protected]> Fri, 22 Sep 2017 11:16:52 +0100

Newsgroups	gmane.linux.debian.ports.mips
Message-ID	<[email protected]>

Hi,

On 22/09/17 07:36, Andreas Tille wrote:
> control: tags -1 help
> 
> Hi Graham,
> 
> since I have no idea how to deal with this I'm forwarding the issue
> to debian-mips list asking desperately for help.
[...]
>> /*stdin*\            :  9.40%   (6000000000 => 563851359 bytes, /*stdout*\)
>> roundTripTest: ./datagen -g1500000000 -P97 |  ../programs/zstd -v1
>> -T999 |  ../programs/zstd -d
>>
>> *** zstd command line interface 32-bits v1.3.1, by Yann Collet ***
>> zstd: error 11 : Allocation error : not enough memory

I think the problem is using too many threads with big stacks. When you
use "zstd -T999", zstd will try to create 256 threads (the maximum
amount). Since the default stack size is 8M, this will try to allocate
2G of virtual memory. On most 32-bit architectures which have 3G address
space boundaries, there is just enough memory to do this. On mips and
mipsel, the boundary is at 2G so there will not be enough space to
allocate all these stacks along with any existing memory.

To fix this you could:

Use "ulimit -s 4096" to half the default stack size. This is only a
workaround for the Debian packaging though - you can't expect users to
do this.

Reduce the value of ZSTDMT_NBTHREADS_MAX. In my testing, using 253
threads worked so you may not need to reduce it much (but maybe to a
more rounder number). You could limit it to mips if you want, but please
only limit it to 32-bit mips:
 #if defined(__mips__) && !defined(__mips64)

Thanks,
James

signature.asc (application/pgp-signature, 833 B)