Re: How to estimate the upper bound of the peak memory consumption of cryptsetup itself?

On Fri, Jun 24, 2022 at 11:14:37AM +0200, Milan Broz wrote:
>On 20/06/2022 02:19, Coiby Xu wrote:
>>On Sat, Jun 18, 2022 at 05:12:56PM +0200, Milan Broz wrote:
>>>On 16/06/2022 06:43, Coiby Xu wrote:
>>>>Hi,
>>>>
>>>>Recently, I notice cryptsetup itself consumes significant amount of
>>>>memory (~256M) when estimating the memory requirement for dumping vmcore
>>>>to a LUKS-encrypted disk,
>>>>
>>>>$ time -v cryptsetup luksOpen encrypted.img volume --key-file mykey.keyfile | grep "Maximum resident set size"
>>>>          Maximum resident set size (kbytes): 1309828
>>>>$ cryptsetup luksDump encrypted.img
>>>>...
>>>>Keyslots:
>>>>    0: luks2
>>>>          PBKDF:      argon2id
>>>>          Memory:     1048576
>>>>          ...
>>>>
>>>>
>>>>So is there a way to estimate the upper bound of the peak memory
>>>>consumption of cryptsetup itself without running cryptsetup?
>>>
>>>As you already found, the major memory consumption is by memory-hard KDF.
>>>But this memory is used only while calculating keyslot encryption key,
>>>it is released immediately after the Argon call is finished.
>>>I do not think we have better estimation here.
>>
>>Thanks for the reply! Sorry I meant the way to estimate the overhead of
>>crypsetup itself i.e. ~256M in the above example. Previously I only take
>>the memory consumption by memory-hard KDF into consideration and
>>neglected the memory consumption of cryptsetup itself. This obviously
>>leads to an underestimation of the memory requirement of cryptsetup. I
>>need to overestimate the memory requirement a bit to make sure OOM won't
>>happen that's why I am asking if there is a way to estimate the
>>upper bound of memory requirement of cryptsetup itself.
>
>There is no generic way to get a number - it depends on configuration
>of the distro, libc, translations, everything that is locked including
>shared libraries.
>
>If it is about RHEL, you can perhaps know exact configuration - please
>ask people in Red Hat.

Provided the configuration, is there an golden algorithm or a formula to
get the number? If it doesn't exist and I need to do some tests to get
an empirical number, are there any big factors I need to be aware of?

>
>>>(Another story is locking all memory, including big areas used by libc,
>>>but that should not be problem here, I hope.)
>>
>>Do you mean locking all memory first in order to know the memory
>>requirement?
>
>We use (mlockall(MCL_CURRENT | MCL_FUTURE) so it locks all used memory
>+ all future allocated memory.
>
>Today, it is not the best option and we will probably lock only specific
>region with stored keys in the future.

Thanks for the explanation! 

>
>Milan
>

-- 
Best regards,
Coiby