Re: Fwd: Memory problems with dar

Le 20/08/2022 à 15:17, Petr Skoda a écrit :
> Hi Denis and all,

Hi Petr,

> 
> I have searched details about a dar memory requirements and I am not
> sure I know the answer.
> In the FAQ there is a link to script dar_rqck.bash for computation of
> amount of memory based on equation one file=1300bytes.
> In the Limitations file you write one file requires 650bytes (and 2x
> more for differential backup) - I guess that the 1300 in the script
> comes from this logic. But you write that is is already 850 for later
> dar versions.

At that time it was easy to have a relatively precise estimation of the
memory requirement, but since then, the catalogue holds in memory
variable number per file of extended attributes, Filesystem specific
attributes, which add on top of the metadata that is always present for
a file in memory (filename, permissions, ownership, dates).

> 
> However, it is not clear whether this is about infinit, 64 or 32 bit
> integers.   I could not find a table about requirements of modern dars
> in all this modes .   Furthermore in the FAQ you write that to overcome
> the memory problem you may compile with 64 or 32 bit integers. But this
> is already the default how the dar is compiled today !

stepping backward to 32bits integer would bring a limit on archive size,
file size and dates... this is no more a solution. I will update the FAQ.

> 
> So as I understand - my dar is 64 and 16GB of RAM in computer is not
> sufficient for extracting catalogue from the dar file I have created as
> described yesterday. if I count files I have 23901645 of them which
> would suggest 23901645*1700 (I take 850x2) =
> 38GB !!!  (I have only 16) .
> So I am hopeless - and I am not allowed to activate swap (admins has
> good reasons for it ... to keep fast array of disk)
> 
> But is the in memory catalog needed only for extracting it (by -C ) ? 

Unfortunately no, it is needed for all operations relative to the backup
content (this is not the case for dar_xform or dar_slave that just plays
at the slice layer).

> It seems I am not able to test archive as well.
> Only listing in serial mode     seems to be working   (cat
> betelgeuse-data.1.dar|dar  -l - -0)

I guess you have not let the operation reach its end, right?

> 
> So I would ask you for a table of memory needed for all bit modes in
> modern dars and is there a way how to do catalog extraction or testing
> in some "serial" slow mode ...
> I see only way now to send my 3TB files to the original computer which
> has 128GB of RAM or do the extraction of catalogue again as a part of
> dar creation on the original computer before sending it through nc to
> the lto machne (lto has a LTO6 tape but only 16GB of RAM).

if you just want to extract the catalogue, you can avoid transferring
the whole backup back to the 128GB RAM server:

Either the backup is already sliced, in that case you will only need the
last slice (assuming it entirely contains the catalogue), else you can
re-slice the backup using dar_xform and have dar_xform removing the
slices as soon as they are created except the last ones (the two last
ones for example):

  dar_xform -s <size> -E 'rm -f %p/%b.$((%N-2)).%e' 3TB_bkp tmp_bkp

the slice <size> should be large enough to contain the whole catalogue.
You can also use "N-3" instead of N-2 above if you want to keep only the
3 last slices at anytime of the dar_xform copy operation.

If it is not clear what it does, the -E option run the command it has in
argument, each time a new slice of tmp_bkp is generated, which number is
%N. This command remove the slice %N-2, so you do not need 3 TB of
additional disk space, but just the disk space for 2 or 3 slices of
let's say 40 GB each (assuming you passed '-s 40G' to dar_xform). But
yes you need to read the whole 3 TB of data... and if you have SSDs this
would implies a lot of useless write I/O...

[...]

> 
>     I have another problems with my backup (as described recently Ihave
>     transfered over nc from one machine to another
>     about 3TB dar  file.  But I am not able to extract catalog or test
>     it on 16GB machine. After some time dar it is killed with memory
>     overflow. The machine may not have swap intentionally. I am using
>     dar 2.7.7RC1  . The dar -V says
>       Integer size used            : 64 bits.
>     (default compilation) . In FAQ you write that the compilation should
>     be --enable-mode=64.   Isn't it that 64 bit shown on dar -V?

yes it is.

> 
>     Is is possible to extract the catalogue or test file in some serial
>     mode (-0) ? That would not cosume a lot of memory ?
>     If I will have more slices - would it help when transforming it in
>     serial mode ?

Today no unfortunately, the slicing is completely independent from the
backup content.

But this makes me thing of some possible memory optimization in the
special case of sequential reading mode... I must look forward whether
this would be feasible and if so how simple or not it would be...

> 
>     I have seen there is now 2.7.7 available ... If I compile it somehow
>     to spare memory, can I use it on the file created with 2.7.7 RC1 as
>     above or do I need it resent  again from the original server ?

there is no problem, the 2.7.7.RC1 is a release candidate for 2.7.7 if
there is differences between the two they only concern bugs, but not the
archive format of available feature. This is not different than passing
from 2.7.6 to 2.7.7. And to reassure you, there has been more than 12
versions of archive format since the beginning and you can still read
archive format "01" with current 2.7.7 (this is part of non-regression
tests).

Looking at git, from 2.7.7.RC1 to 2.7.7 there is only documentation
update and minor code modification to have dar more tolerant to the
fadvise() system call failure.


> 
>     Thanks a lot
> 

Best Regards,
Denis