Re: 2.7 pre-release: Virtual memory exploding and dar locks up

Hi

On 31.03.21 22:34, Denis Corbin wrote:
> can you produce the output of 'dar -V' to let me check the option that
> where activated at compilation time?


/opt/dar/bin/dar -V

  dar version 2.7.0.pre4, Copyright (C) 2002-2021 Denis Corbin
    Long options support         : YES

  Using libdar 6.3.0 built with compilation time options:
    gzip compression (libz)      : YES
    bzip2 compression (libbzip2) : YES
    lzo compression (liblzo2)    : YES
    xz compression (liblzma)     : YES
    zstd compression (libzstd)   : YES
    lz4 compression (liblz4)     : YES
    Strong encryption (libgcrypt): YES
    Public key ciphers (gpgme)   : YES
    Extended Attributes support  : YES
    Large files support (> 2GB)  : YES
    ext2fs NODUMP flag support   : YES
    Integer size used            : 64 bits
    Thread safe support          : YES
    Furtive read mode support    : YES
    Linux ext2/3/4 FSA support   : YES
    Mac OS X HFS+ FSA support    : NO
    Linux statx() support        : YES
    Detected system/CPU endian   : little
    Posix fadvise support        : YES
    Large dir. speed optimi.     : YES
    Timestamp read accuracy      : 1 nanosecond
    Timestamp write accuracy     : 1 nanosecond
    Restores dates of symlinks   : YES
    Multiple threads (libthreads): YES (1.3.0)
    Delta compression (librsync) : YES
    Remote repository (libcurl)  : NO
    argon2 hashing (libargon2)   : NO

  compiled the Mar 29 2021 with GNUC version 8.3.0
  dar is part of the Disk ARchive suite (Release 2.7.0.pre4)
  dar comes with ABSOLUTELY NO WARRANTY; for details
  type `dar -W'. This is free software, and you are welcome
  to redistribute it under certain conditions; type `dar -L | more'
  for details.


> What is strange is in what you describe is the fact dar virtual memory
> footprints increased dramatically while it produces nothing or quite
> nothing on disk.... Just a guess, could you try recompiling dar without
> optimization (export CXXFLAGS ; export LDFLAGS; ./configure ; ...) and
> see whether the behavior changes?

Let's see (I hope -O2 won't spoil the build).

(1) I reinstalled libthreadad 1.3.0 to /opt/libthreadar just to be sure 
nothing from my earlier tests intervenes. Also uninstalled/removed all 
traces from former installs under /usr/local/

(2) cd libthreadar-1.3.0; ./configure --prefix=/opt/libthreadar; make -j 
12; make install

(3) first failed try to build dar without opt:
tar xfz dar-2.7.0.pre4.tar.gz
cd dar-2.7.0.pre4
unset LD_LIBRARY_PATH; export 
PKG_CONFIG_PATH=/opt/libthreadar/lib/pkgconfig; export CXXFLAGS ; export 
LDFLAGS; ./configure --prefix=/opt/dar-no-opt

this failed as configure thought libthreadar was not operable

(4) New try:
export LD_LIBRARY_PATH=/opt/libthreadar/lib/; export 
PKG_CONFIG_PATH=/opt/libthreadar/lib/pkgconfig; export CXXFLAGS ; export 
LDFLAGS; ./configure --prefix=/opt/dar-no-opt
make -j 12
make install
export LD_LIBRARY_PATH=/opt/libthreadar/lib/:/opt/dar-no-opt/lib
/opt/dar-no-opt/bin/dar -V

  dar version 2.7.0.pre4, Copyright (C) 2002-2021 Denis Corbin
    Long options support         : YES

  Using libdar 6.3.0 built with compilation time options:
    gzip compression (libz)      : YES
    bzip2 compression (libbzip2) : YES
    lzo compression (liblzo2)    : YES
    xz compression (liblzma)     : YES
    zstd compression (libzstd)   : YES
    lz4 compression (liblz4)     : YES
    Strong encryption (libgcrypt): YES
    Public key ciphers (gpgme)   : YES
    Extended Attributes support  : YES
    Large files support (> 2GB)  : YES
    ext2fs NODUMP flag support   : YES
    Integer size used            : 64 bits
    Thread safe support          : YES
    Furtive read mode support    : YES
    Linux ext2/3/4 FSA support   : YES
    Mac OS X HFS+ FSA support    : NO
    Linux statx() support        : YES
    Detected system/CPU endian   : little
    Posix fadvise support        : YES
    Large dir. speed optimi.     : YES
    Timestamp read accuracy      : 1 nanosecond
    Timestamp write accuracy     : 1 nanosecond
    Restores dates of symlinks   : YES
    Multiple threads (libthreads): YES (1.3.0)
    Delta compression (librsync) : YES
    Remote repository (libcurl)  : NO
    argon2 hashing (libargon2)   : NO

  compiled the Apr  1 2021 with GNUC version 8.3.0
  dar is part of the Disk ARchive suite (Release 2.7.0.pre4)
  dar comes with ABSOLUTELY NO WARRANTY; for details
  type `dar -W'. This is free software, and you are welcome
  to redistribute it under certain conditions; type `dar -L | more'
  for details.

Unfortunately, no change whatsoever a lock up after about 6400 lines of 
output, i.e. about 3200 files added and saved. I made a small screen 
cast of htop while running:

https://wolke7.aei.mpg.de/s/jFPxg6kkHe45FGi

(apologies to archive readers in the future, the 'movie' will expire at 
the end of July 2021)

> What type of data is it:
> - a lot of small files? (what is their size then more or less?)
> - a mix of small and larger files?
> - only files large files?
> 

It's a user's project directory of one of his many projects. It starts 
of with a largish python venv and contains otherwise many hdf5 files of 
various files. If necessary, I can try to generate a file size spectrum.

> another question, is the problem systematic or frequent but not
> systematic, if so then when it fails, do you always end with the same
> amount of data written do disk (same resulting backup size)?
> 

It's totally reproducible with this dataset:

stat dartest1.0001.dar
   File: dartest1.0001.dar
   Size: 88090117        Blocks: 172393     IO Block: 131072 regular file
Device: 36h/54d Inode: 8           Links: 1
Access: (0600/-rw-------)  Uid: ( 5001/ carsten)   Gid: ( 5100/     lvc)
Access: 2021-04-01 16:57:22.079594066 +0000
Modify: 2021-04-01 16:57:44.791764531 +0000
Change: 2021-04-01 16:57:44.791764531 +0000
  Birth: -

rm dartest1.0001.dar*

/opt/dar-no-opt/bin/dar --create dartest1 --verbose [...] | pv -labert > 
/dev/null

Arguments read from /opt/dar-no-opt/etc/darrc :

^C6.43k 0:00:29 [0.00 /s] [ 221 /s]
Received signal: Interrupt
Archive delayed termination engaged
Disabling signal handler, the next time this signal is received the 
program will abort immediately
^C

stat dartest1.0001.dar
   File: dartest1.0001.dar
   Size: 88090117        Blocks: 172357     IO Block: 131072 regular file
Device: 36h/54d Inode: 772         Links: 1
Access: (0600/-rw-------)  Uid: ( 5001/ carsten)   Gid: ( 5100/     lvc)
Access: 2021-04-01 17:04:51.510967270 +0000
Modify: 2021-04-01 17:05:14.163137290 +0000
Change: 2021-04-01 17:05:14.163137290 +0000
  Birth: -

> Thanks for the possibility to access your environment, we will come to
> that point only if necessary, but that's good to have this last resort
> solution, thank you for that.

Yeah, it will be a kind of last resort as it take quite a bit of effort 
- plus I would need to get the ok from the user to make his dataset 
available for testing by external people. Until then, please remote 
control me ;-)

Cheers

Carsten

_______________________________________________
Dar-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-support