Re: Successful first run of 6TB archiving project

On Mon, Jul 17 2023, Denis Corbin wrote:

>
> According to the numbers you provided, dar consumed an average of 998 bytes of
> RAM per inode... Ignore my questions if you don't have the info: it would be
> interesting to know whether EA and FSA were saved and also whether
> parallel/block compression and/or encryption has been used and with how much
> threads for each (Multi-thread implementation relies on memory chunks used to
> distribute/gather compression work to/from worker threads)

Hi Denis,

I did note that the RAM usage was higher than documented on the
limitations page, but it wasn't so much higher than I went searching for
causes.

But I'm happy to answer :-)

I used no specific options relating to EA or FSA.  I would not have
expected much of either.  The files are all stored on ZFS, but it does
seem like there is some degree of FSA stored, somewhat to my surprise.
As this isn't an ext* filesystem, I have no idea what it's storing.  I
saved the summary at the end of archive creation, but it is on a
different drive and I'm running a dar -t test right now, so I can grab
that a bit later.  I do see "Data + FSA(13)" for every file on the dar
-v -t output.

This was created on dar 2.7.8, from a build before I enabled libthreadar
linking, so no parallel features are used.  zstd compression is enabled,
but the exclude rules mean that the majority of the data by bytes (and
probably also by inode) is not being compressed since it is a lot of
image and video files.  There is no encryption in use, and no rsync sig
options in use.

dar -q -l on the isolated catalog shows:

total number of inode : 1295439
fully saved           : 1295439
binay delta patch     : 0
inode metadata only   : 0
distribution of inode(s)
 - directories        : 67694
 - plain files        : 1224095
 - symbolic links     : 3647
 - named pipes        : 3
 - unix sockets       : 0
 - character devices  : 0
 - block devices      : 0
 - Door entries       : 0
hard links information
 - number of inode with hard link           : 116267
 - number of reference to hard linked inodes: 246019
destroyed entries information
   0 file(s) have been record as destroyed since backup of reference

/usr/bin/time report on dar -q -l for this catalog:

12.83user 1.02system 0:13.87elapsed 99%CPU (0avgtext+0avgdata 1114820maxresident)k
159715inputs+0outputs (0major+283922minor)pagefaults 0swaps

which is a bit less than it was using during -c, but then I'm probably
not asking it to do as much with it.

Incidentally, in a bit of an echo of a conversation from a little while
back, I observe that when I use -q -l on the last slice of an archive,
as opposed to an isolated catalog, it will easily report the full size
of the archive (including all slices, I believe), while when run on an
isolated catalog, it only reports the size of the catalog.

By the time this project is done, I may have around 3 million inodes
represented.

- John