Re: 2.7 pre-release: Virtual memory exploding and dar locks up

On 30/03/2021 09:46, Carsten Aulbert wrote:
> Hi,

Hi,

> 
> I am very new to dar as I need/want to create archives which are more or
> less randomly accessible unlike tar.xz which I have been using until now.
> 
> One of my current tasks is to put quite large datasets into an archive
> format, thus I tried dar 2.6.2 on Debian buster but the compression
> performance is a bit lacking:

pay attention to the fact tar uses a default compression ratio (usually
6 depending on the compression binary) while dar use the maximum
available (9 or more depending on the algo used). You can tune that with
dar (not with tar AFAIK) thanks to the -z option (see details in man page).

> 
> A test date set (1,540,323MByte according to du) is being archived into
> 100GByte slices for the past ~24h and is currently working on the 11th
> slice[1].
> 
> From previous tests with tar, usually the compression stage was
> responsible for throughput limitations, therefore, I turned to 2.7.0pre4
> yesterday for some evaluation. But with a command like [2] it reliably
> locks up after reaching a 256GByte virtual memory footprint.
> 
> While running, RSS memory usage stays quite (a few 100 MBytes at most)
> low but VSS explodes fast and then locks up, i.e. nothing happens
> anymore and needs to be killed:
> 
> ps aum
> [...]
> root     30370 24.9  0.1 268795548 338880 pts/3 -   07:37   0:54
> /opt/dar/bin/dar [options from [2]]
> root         -  1.3    -      -     - -        SLl+ 07:37   0:02 -
> root         -  0.0    -      -     - -        SLl+ 07:38   0:00 -
> root         -  0.0    -      -     - -        SLl+ 07:38   0:00 -
> root         -  0.0    -      -     - -        SLl+ 07:38   0:00 -
> root         -  0.0    -      -     - -        SLl+ 07:38   0:00 -
> root         -  0.0    -      -     - -        SLl+ 07:38   0:00 -
> root         -  0.0    -      -     - -        SLl+ 07:38   0:00 -
> root         -  0.0    -      -     - -        SLl+ 07:38   0:00 -
> root         -  0.0    -      -     - -        SLl+ 07:38   0:00 -
> root         -  0.0    -      -     - -        SLl+ 07:38   0:00 -
> root         -  0.0    -      -     - -        SLl+ 07:38   0:00 -
> [...]
> 
> on disk 85MByte are written but that's all.
> 
> Is there anything at fault in my command or can I assist in fixing this
> bug - if this is one?

nothing at fault on your command, dar/libdar should tell it else, most
probably a bug I have not yet met in the non-regression tests currently
running...

I will try to reproduce your context and see whether I also met this
problem so I can fix it. Thanks for feedback!

> 
> Cheers
> 
> Carsten
> 
> [1] dar --create test --verbose --fs-root /data/test-dataset
> --alter=binary --slice=100G --min-digits=4 --compression=gzip
> --slice-mode=0600 --hash=sha512
> 
> [2] /opt/dar/bin/dar --create test-parallel --verbose --fs-root=
> /data/test-dataset --alter=binary --slice=100G --min-digits=4
> --compression=zstd:20:100k -G 1,12 --slice-mode=0600 --hash=sha512
> 

Cheers,
Denis

_______________________________________________
Dar-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-support