Re: Fwd: Memory problems with dar

Hi,

I've created a source tarball for convenience, available at
https://dar.edrusb.org/dar.linux.free.fr/Specific_builds/

Cheers,
Denis

Le 18/09/2022 à 19:19, Denis Corbin a écrit :
> Hi Per,
> 
> Some news:
> 
> You will find on github on the branch named "early_mem_release" a
> modified version of release 2.7.7 that uses much less memory in
> sequential read mode as what it used to do so far.
> 
> What the code change brings is to release memory used to store directory
> content once the reading process exits from that directory, thus the dar
> process heap stays lower. There is not visibile performance penalty or
> gain, but a huge memory gain when the directory tree is wide. At the
> opposite if the directory tree is a chain of nested subdirectories, the
> gain should be quite inexistant.
> 
> During these tests, I could get an approximative measure of the amount
> of memory used to store metadata inode: 889696 KB for 1151448 inodes for
> the whole dar process which lead to a majorant of 792 bytes per inode.
> This is not far from the 850 bytes mentionned in the documentation, but
> thing may vary depending on the number of Extended Attributes saved
> (none in my case) and FSA (linux FSA in my testings).
> 
> When using early memory release (with sequential read mode) the dar
> process on this data set only consumes 119 MB instead of 889 MB.
> 
> There is some room to improve that, but think that is will still depend
> on the shape of the directory tree.
> 
> This code will be merged later either in the master trunk for
> availability with release 2.8.0 or maybe if backward compatibility tests
> passed, into 2.7.x branch, thus for a future 2.7.8 or 2.7.9 release.
> 
> In the hope this will help in your context.
> 
> Regards,
> Denis
> 
> Le 20/08/2022 à 15:17, Petr Skoda a écrit :
>> Hi Denis and all,
>>
>> I have searched details about a dar memory requirements and I am not
>> sure I know the answer.
>> In the FAQ there is a link to script dar_rqck.bash for computation of
>> amount of memory based on equation one file=1300bytes.
>> In the Limitations file you write one file requires 650bytes (and 2x
>> more for differential backup) - I guess that the 1300 in the script
>> comes from this logic. But you write that is is already 850 for later
>> dar versions.
>>
>> However, it is not clear whether this is about infinit, 64 or 32 bit
>> integers.   I could not find a table about requirements of modern dars
>> in all this modes .   Furthermore in the FAQ you write that to overcome
>> the memory problem you may compile with 64 or 32 bit integers. But this
>> is already the default how the dar is compiled today !
>>
>> So as I understand - my dar is 64 and 16GB of RAM in computer is not
>> sufficient for extracting catalogue from the dar file I have created as
>> described yesterday. if I count files I have 23901645 of them which
>> would suggest 23901645*1700 (I take 850x2) =
>> 38GB !!!  (I have only 16) .
>> So I am hopeless - and I am not allowed to activate swap (admins has
>> good reasons for it ... to keep fast array of disk)
>>
>> But is the in memory catalog needed only for extracting it (by -C ) ? 
>> It seems I am not able to test archive as well.
>> Only listing in serial mode     seems to be working   (cat
>> betelgeuse-data.1.dar|dar  -l - -0)
>>
>> So I would ask you for a table of memory needed for all bit modes in
>> modern dars and is there a way how to do catalog extraction or testing
>> in some "serial" slow mode ...
>> I see only way now to send my 3TB files to the original computer which
>> has 128GB of RAM or do the extraction of catalogue again as a part of
>> dar creation on the original computer before sending it through nc to
>> the lto machne (lto has a LTO6 tape but only 16GB of RAM).
>>
>> Best regards,
>>
>> Petr
>>
>>
>>
>>
>>
>> ---------- Původní e-mail ----------
>> Od: Petr Skoda <[email protected]>
>> Komu: [email protected]
>> Datum: 19. 8. 2022 21:10:52
>> Předmět: [Dar-support] Memory problems with dar
>>
>>
>>     Hi Denis,
>>
>>     I have another problems with my backup (as described recently Ihave
>>     transfered over nc from one machine to another
>>     about 3TB dar  file.  But I am not able to extract catalog or test
>>     it on 16GB machine. After some time dar it is killed with memory
>>     overflow. The machine may not have swap intentionally. I am using
>>     dar 2.7.7RC1  . The dar -V says
>>       Integer size used            : 64 bits.
>>     (default compilation) . In FAQ you write that the compilation should
>>     be --enable-mode=64.   Isn't it that 64 bit shown on dar -V?
>>
>>     Is is possible to extract the catalogue or test file in some serial
>>     mode (-0) ? That would not cosume a lot of memory ?
>>     If I will have more slices - would it help when transforming it in
>>     serial mode ?
>>
>>     I have seen there is now 2.7.7 available ... If I compile it somehow
>>     to spare memory, can I use it on the file created with 2.7.7 RC1 as
>>     above or do I need it resent  again from the original server ?
>>
>>     Thanks a lot
>>
>>
>>
>>
>>
>>