Re: 2.7 pre-release: Virtual memory exploding and dar locks up

Hi Carsten

Well, the problem may be related to the data under backup. While I have
not that high volume of data to test as you did, I get more than 85 MiB
written to disk (making a backup of 17 GiB that completed without
problem using your [2] command-line, modified only about the --fs-root
option and path of the resulting backup). The memory usage I observed
was stable as you can see below:

ps aum
[...]
denis     2578  318  0.7 993752 59204 pts/2    -    21:31  53:18 [dar]
denis        - 17.4    -      -     - -        Rl+  21:31   2:55 -
denis        -  0.0    -      -     - -        Sl+  21:48   0:00 -
denis        -  0.0    -      -     - -        Rl+  21:48   0:00 -
denis        -  0.0    -      -     - -        Rl+  21:48   0:00 -
denis        -  0.0    -      -     - -        Rl+  21:48   0:00 -
denis        -  0.0    -      -     - -        Rl+  21:48   0:00 -
denis        -  0.0    -      -     - -        Rl+  21:48   0:00 -
denis        -  0.0    -      -     - -        Rl+  21:48   0:00 -
denis        -  0.0    -      -     - -        Rl+  21:48   0:00 -
denis        -  0.0    -      -     - -        Rl+  21:48   0:00 -
denis        -  0.0    -      -     - -        Rl+  21:48   0:00 -
denis        -  0.0    -      -     - -        Rl+  21:48   0:00 -
denis        -  0.0    -      -     - -        Rl+  21:48   0:00 -
denis        -  0.0    -      -     - -        Rl+  21:48   0:00 -

can you produce the output of 'dar -V' to let me check the option that
where activated at compilation time?

What is strange is in what you describe is the fact dar virtual memory
footprints increased dramatically while it produces nothing or quite
nothing on disk.... Just a guess, could you try recompiling dar without
optimization (export CXXFLAGS ; export LDFLAGS; ./configure ; ...) and
see whether the behavior changes?

What type of data is it:
- a lot of small files? (what is their size then more or less?)
- a mix of small and larger files?
- only files large files?

the data set I used contained files of a size of from a few hundred
kilobytes to 5 MiB with an average of 4,3 MiB (18 GiB for 4282 files).

see below other question inlined...

On 31/03/2021 09:36, Carsten Aulbert wrote:
> Hi
> 
> On 30.03.21 21:48, Denis Corbin wrote:

[...]

> yeah, for that reason my pipeline was actually
> 
> storage box of origin:
> tar c | mbuffer --TCP-->
> 
> powerful CPU box:
> mbuffer | xz -9 [...] | mbuffer --TCP-->
> 
> storage box for tarballs:
> mbuffer | split --> local storage
> 
> which in principle would still work except the new requirement of quasi
> random access to few files within the end results which tar would not
> allow, hence: *dar* :)

:)

[...]

>>
>> I will try to reproduce your context and see whether I also met this
>> problem so I can fix it. Thanks for feedback!
> 
> If I can assist in any way, feel free to contact me on or off-list. If
> you cannot reproduce the issue I can try to organize access to a machine
> where I see this - but that would probably mean me going to the data
> center once again and single out a machine for you (some effort but
> doable).

another question, is the problem systematic or frequent but not
systematic, if so then when it fails, do you always end with the same
amount of data written do disk (same resulting backup size)?

Thanks for the possibility to access your environment, we will come to
that point only if necessary, but that's good to have this last resort
solution, thank you for that.

> 
> Cheers
> 
> Carsten
> 
>

Cheers,
Denis


[1] dar --create test --verbose --fs-root /data/test-dataset
--alter=binary --slice=100G --min-digits=4 --compression=gzip
--slice-mode=0600 --hash=sha512

[2] /opt/dar/bin/dar --create test-parallel --verbose --fs-root=
/data/test-dataset --alter=binary --slice=100G --min-digits=4
--compression=zstd:20:100k -G 1,12 --slice-mode=0600 --hash=sha512

_______________________________________________
Dar-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-support