Re: 2.7 pre-release: Virtual memory exploding and dar locks up

On 01/04/2021 19:08, Carsten Aulbert wrote:
> Hi

Hi,

> 
> On 31.03.21 22:34, Denis Corbin wrote:
>> can you produce the output of 'dar -V' to let me check the option that
>> where activated at compilation time?
> 
> 
> /opt/dar/bin/dar -V
> 
>  dar version 2.7.0.pre4, Copyright (C) 2002-2021 Denis Corbin
>    Long options support         : YES
> 
>  Using libdar 6.3.0 built with compilation time options:
>    gzip compression (libz)      : YES
>    bzip2 compression (libbzip2) : YES
>    lzo compression (liblzo2)    : YES
>    xz compression (liblzma)     : YES
>    zstd compression (libzstd)   : YES
>    lz4 compression (liblz4)     : YES
>    Strong encryption (libgcrypt): YES
>    Public key ciphers (gpgme)   : YES
>    Extended Attributes support  : YES
>    Large files support (> 2GB)  : YES
>    ext2fs NODUMP flag support   : YES
>    Integer size used            : 64 bits
>    Thread safe support          : YES
>    Furtive read mode support    : YES
>    Linux ext2/3/4 FSA support   : YES
>    Mac OS X HFS+ FSA support    : NO
>    Linux statx() support        : YES
>    Detected system/CPU endian   : little
>    Posix fadvise support        : YES
>    Large dir. speed optimi.     : YES
>    Timestamp read accuracy      : 1 nanosecond
>    Timestamp write accuracy     : 1 nanosecond
>    Restores dates of symlinks   : YES
>    Multiple threads (libthreads): YES (1.3.0)

please use libthreadar 1.3.1. Release 1.3.0 has a bug fixes that might
well be the cause of the problem you observe... I bet this is the cause
of the problem seen the symptoms! :)

>    Delta compression (librsync) : YES
>    Remote repository (libcurl)  : NO
>    argon2 hashing (libargon2)   : NO
> 
>  compiled the Mar 29 2021 with GNUC version 8.3.0
>  dar is part of the Disk ARchive suite (Release 2.7.0.pre4)
>  dar comes with ABSOLUTELY NO WARRANTY; for details
>  type `dar -W'. This is free software, and you are welcome
>  to redistribute it under certain conditions; type `dar -L | more'
>  for details.
> 
> 
>> What is strange is in what you describe is the fact dar virtual memory
>> footprints increased dramatically while it produces nothing or quite
>> nothing on disk.... Just a guess, could you try recompiling dar without
>> optimization (export CXXFLAGS ; export LDFLAGS; ./configure ; ...) and
>> see whether the behavior changes?
> 
> Let's see (I hope -O2 won't spoil the build).>
> (1) I reinstalled libthreadad 1.3.0 to /opt/libthreadar just to be sure
> nothing from my earlier tests intervenes. Also uninstalled/removed all
> traces from former installs under /usr/local/

you can use "make uninstall" (after ./configure --prefix=...) both with
libthreadar and dar/libdar to make the cleanup for you. Just be sure to
give the same path to the --prefix= option as the one that was used to
build the software.

> 
> (2) cd libthreadar-1.3.0; ./configure --prefix=/opt/libthreadar; make -j
> 12; make install

As mentionned earlier, please use libthreadar-1.3.1 I'm pretty sure the
problem comes from the 1.3.0 release which has the following bug:

From release 1.3.0 to 1.3.1 (extracted from libthreadar Changelog)
  fixed bug in thread class that leads the pthread_join() call not to
  be executed when join() method was invoked and the thread had
  completed, leading to memory saturation passed a certain number of
  thread created

> 
> (3) first failed try to build dar without opt:
> tar xfz dar-2.7.0.pre4.tar.gz
> cd dar-2.7.0.pre4
> unset LD_LIBRARY_PATH; export
> PKG_CONFIG_PATH=/opt/libthreadar/lib/pkgconfig; export CXXFLAGS ; export
> LDFLAGS; ./configure --prefix=/opt/dar-no-opt
> 
> this failed as configure thought libthreadar was not operable

OK I agree, this is normal as you have installed libthreadar under a
specific directory, unless you have either LD_LIBRARY_PATH set or
/etc/ld.so.conf including /opt/libthreadar/lib and run ldconfig afterward.

> 
> (4) New try:
> export LD_LIBRARY_PATH=/opt/libthreadar/lib/; export
> PKG_CONFIG_PATH=/opt/libthreadar/lib/pkgconfig; export CXXFLAGS ; export
> LDFLAGS; ./configure --prefix=/opt/dar-no-opt
> make -j 12
> make install
> export LD_LIBRARY_PATH=/opt/libthreadar/lib/:/opt/dar-no-opt/lib
> /opt/dar-no-opt/bin/dar -V
> 
>  dar version 2.7.0.pre4, Copyright (C) 2002-2021 Denis Corbin
>    Long options support         : YES
> 
>  Using libdar 6.3.0 built with compilation time options:
>    gzip compression (libz)      : YES
>    bzip2 compression (libbzip2) : YES
>    lzo compression (liblzo2)    : YES
>    xz compression (liblzma)     : YES
>    zstd compression (libzstd)   : YES
>    lz4 compression (liblz4)     : YES
>    Strong encryption (libgcrypt): YES
>    Public key ciphers (gpgme)   : YES
>    Extended Attributes support  : YES
>    Large files support (> 2GB)  : YES
>    ext2fs NODUMP flag support   : YES
>    Integer size used            : 64 bits
>    Thread safe support          : YES
>    Furtive read mode support    : YES
>    Linux ext2/3/4 FSA support   : YES
>    Mac OS X HFS+ FSA support    : NO
>    Linux statx() support        : YES
>    Detected system/CPU endian   : little
>    Posix fadvise support        : YES
>    Large dir. speed optimi.     : YES
>    Timestamp read accuracy      : 1 nanosecond
>    Timestamp write accuracy     : 1 nanosecond
>    Restores dates of symlinks   : YES
>    Multiple threads (libthreads): YES (1.3.0)
>    Delta compression (librsync) : YES
>    Remote repository (libcurl)  : NO
>    argon2 hashing (libargon2)   : NO
> 
>  compiled the Apr  1 2021 with GNUC version 8.3.0
>  dar is part of the Disk ARchive suite (Release 2.7.0.pre4)
>  dar comes with ABSOLUTELY NO WARRANTY; for details
>  type `dar -W'. This is free software, and you are welcome
>  to redistribute it under certain conditions; type `dar -L | more'
>  for details.
> 
> Unfortunately, no change whatsoever a lock up after about 6400 lines of
> output, i.e. about 3200 files added and saved. I made a small screen
> cast of htop while running:

OK this was good to know there was not an compiler optimization problem
as it has arrived in the past.

> 
> https://wolke7.aei.mpg.de/s/jFPxg6kkHe45FGi
> 
> (apologies to archive readers in the future, the 'movie' will expire at
> the end of July 2021)
> 
>> What type of data is it:
>> - a lot of small files? (what is their size then more or less?)
>> - a mix of small and larger files?
>> - only files large files?
>>
> 
> It's a user's project directory of one of his many projects. It starts
> of with a largish python venv and contains otherwise many hdf5 files of
> various files. If necessary, I can try to generate a file size spectrum.

For now this is not necessary, unless the track of libthreadar 1.3.1 has
 been invalidated...

> 
>> another question, is the problem systematic or frequent but not
>> systematic, if so then when it fails, do you always end with the same
>> amount of data written do disk (same resulting backup size)?
>>
> 
> It's totally reproducible with this dataset:

OK, that's good to know as it becomes much easier to troubleshoot a
problem when it is reproducible

> 
> stat dartest1.0001.dar
>   File: dartest1.0001.dar
>   Size: 88090117        Blocks: 172393     IO Block: 131072 regular file
> Device: 36h/54d Inode: 8           Links: 1
> Access: (0600/-rw-------)  Uid: ( 5001/ carsten)   Gid: ( 5100/     lvc)
> Access: 2021-04-01 16:57:22.079594066 +0000
> Modify: 2021-04-01 16:57:44.791764531 +0000
> Change: 2021-04-01 16:57:44.791764531 +0000
>  Birth: -
> 
> rm dartest1.0001.dar*
> 
> /opt/dar-no-opt/bin/dar --create dartest1 --verbose [...] | pv -labert >
> /dev/null
> 
> Arguments read from /opt/dar-no-opt/etc/darrc :
> 
> ^C6.43k 0:00:29 [0.00 /s] [ 221 /s]
> Received signal: Interrupt
> Archive delayed termination engaged
> Disabling signal handler, the next time this signal is received the
> program will abort immediately
> ^C
> 
> stat dartest1.0001.dar
>   File: dartest1.0001.dar
>   Size: 88090117        Blocks: 172357     IO Block: 131072 regular file
> Device: 36h/54d Inode: 772         Links: 1
> Access: (0600/-rw-------)  Uid: ( 5001/ carsten)   Gid: ( 5100/     lvc)
> Access: 2021-04-01 17:04:51.510967270 +0000
> Modify: 2021-04-01 17:05:14.163137290 +0000
> Change: 2021-04-01 17:05:14.163137290 +0000
>  Birth: -
> 
>> Thanks for the possibility to access your environment, we will come to
>> that point only if necessary, but that's good to have this last resort
>> solution, thank you for that.
> 
> Yeah, it will be a kind of last resort as it take quite a bit of effort
> - plus I would need to get the ok from the user to make his dataset
> available for testing by external people. Until then, please remote
> control me ;-)

Let's hope then the 1.3.1 release of libthreadar will fix the issue!

> 
> Cheers
> 
> Carsten
> 

Cheers,
Denis

_______________________________________________
Dar-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-support