Re: Fwd: Memory problems with dar

Hi Per,

Some news:

You will find on github on the branch named "early_mem_release" a
modified version of release 2.7.7 that uses much less memory in
sequential read mode as what it used to do so far.

What the code change brings is to release memory used to store directory
content once the reading process exits from that directory, thus the dar
process heap stays lower. There is not visibile performance penalty or
gain, but a huge memory gain when the directory tree is wide. At the
opposite if the directory tree is a chain of nested subdirectories, the
gain should be quite inexistant.

During these tests, I could get an approximative measure of the amount
of memory used to store metadata inode: 889696 KB for 1151448 inodes for
the whole dar process which lead to a majorant of 792 bytes per inode.
This is not far from the 850 bytes mentionned in the documentation, but
thing may vary depending on the number of Extended Attributes saved
(none in my case) and FSA (linux FSA in my testings).

When using early memory release (with sequential read mode) the dar
process on this data set only consumes 119 MB instead of 889 MB.

There is some room to improve that, but think that is will still depend
on the shape of the directory tree.

This code will be merged later either in the master trunk for
availability with release 2.8.0 or maybe if backward compatibility tests
passed, into 2.7.x branch, thus for a future 2.7.8 or 2.7.9 release.

In the hope this will help in your context.

Regards,
Denis

Le 20/08/2022 à 15:17, Petr Skoda a écrit :
> Hi Denis and all,
> 
> I have searched details about a dar memory requirements and I am not
> sure I know the answer.
> In the FAQ there is a link to script dar_rqck.bash for computation of
> amount of memory based on equation one file=1300bytes.
> In the Limitations file you write one file requires 650bytes (and 2x
> more for differential backup) - I guess that the 1300 in the script
> comes from this logic. But you write that is is already 850 for later
> dar versions.
> 
> However, it is not clear whether this is about infinit, 64 or 32 bit
> integers.   I could not find a table about requirements of modern dars
> in all this modes .   Furthermore in the FAQ you write that to overcome
> the memory problem you may compile with 64 or 32 bit integers. But this
> is already the default how the dar is compiled today !
> 
> So as I understand - my dar is 64 and 16GB of RAM in computer is not
> sufficient for extracting catalogue from the dar file I have created as
> described yesterday. if I count files I have 23901645 of them which
> would suggest 23901645*1700 (I take 850x2) =
> 38GB !!!  (I have only 16) .
> So I am hopeless - and I am not allowed to activate swap (admins has
> good reasons for it ... to keep fast array of disk)
> 
> But is the in memory catalog needed only for extracting it (by -C ) ? 
> It seems I am not able to test archive as well.
> Only listing in serial mode     seems to be working   (cat
> betelgeuse-data.1.dar|dar  -l - -0)
> 
> So I would ask you for a table of memory needed for all bit modes in
> modern dars and is there a way how to do catalog extraction or testing
> in some "serial" slow mode ...
> I see only way now to send my 3TB files to the original computer which
> has 128GB of RAM or do the extraction of catalogue again as a part of
> dar creation on the original computer before sending it through nc to
> the lto machne (lto has a LTO6 tape but only 16GB of RAM).
> 
> Best regards,
> 
> Petr
> 
> 
> 
> 
> 
> ---------- Původní e-mail ----------
> Od: Petr Skoda <[email protected]>
> Komu: [email protected]
> Datum: 19. 8. 2022 21:10:52
> Předmět: [Dar-support] Memory problems with dar
> 
> 
>     Hi Denis,
> 
>     I have another problems with my backup (as described recently Ihave
>     transfered over nc from one machine to another
>     about 3TB dar  file.  But I am not able to extract catalog or test
>     it on 16GB machine. After some time dar it is killed with memory
>     overflow. The machine may not have swap intentionally. I am using
>     dar 2.7.7RC1  . The dar -V says
>       Integer size used            : 64 bits.
>     (default compilation) . In FAQ you write that the compilation should
>     be --enable-mode=64.   Isn't it that 64 bit shown on dar -V?
> 
>     Is is possible to extract the catalogue or test file in some serial
>     mode (-0) ? That would not cosume a lot of memory ?
>     If I will have more slices - would it help when transforming it in
>     serial mode ?
> 
>     I have seen there is now 2.7.7 available ... If I compile it somehow
>     to spare memory, can I use it on the file created with 2.7.7 RC1 as
>     above or do I need it resent  again from the original server ?
> 
>     Thanks a lot
> 
> 
> 
> 
>