Re: Successful first run of 6TB archiving project

On 17/07/2023 16:59, John Goerzen wrote:
> On Mon, Jul 17 2023, Denis Corbin wrote:
> 
>>
>> According to the numbers you provided, dar consumed an average of 998 bytes of
>> RAM per inode... Ignore my questions if you don't have the info: it would be
>> interesting to know whether EA and FSA were saved and also whether
>> parallel/block compression and/or encryption has been used and with how much
>> threads for each (Multi-thread implementation relies on memory chunks used to
>> distribute/gather compression work to/from worker threads)
> 
> Hi Denis,
> 
> I did note that the RAM usage was higher than documented on the
> limitations page, but it wasn't so much higher than I went searching for
> causes.

Thanks for your answer, this amount is very dependent on the nature of 
the metadata each file has... so having more RAM utilization than 
"expected"/documented is harmless, this is rather due to new features 
added over time, which one of them consume a little more RAM... I will 
better update the information in the documentation.

> 
> But I'm happy to answer :-)
> 
> I used no specific options relating to EA or FSA.  I would not have
> expected much of either.  The files are all stored on ZFS, but it does
> seem like there is some degree of FSA stored, somewhat to my surprise.
> As this isn't an ext* filesystem, I have no idea what it's storing.  I
> saved the summary at the end of archive creation, but it is on a
> different drive and I'm running a dar -t test right now, so I can grab
> that a bit later.  I do see "Data + FSA(13)" for every file on the dar
> -v -t output.

Here the FSA libdar takes care about today are the following:

    enum fsa_nature { fsan_unset, //< except this one, which means no FSA
                       fsan_creation_date,
                       fsan_append_only,
                       fsan_compressed,
                       fsan_no_dump,
                       fsan_immutable,
                       fsan_data_journaling,
                       fsan_secure_deletion,
                       fsan_no_tail_merging,
                       fsan_undeletable,
                       fsan_noatime_update,
                       fsan_synchronous_directory,
                       fsan_synchronous_update,
                       fsan_top_of_dir_hierarchy };

If the operating system returns a value for a file using this system call:
	int ioctl(int fd, EXT2_IOC_GETFLAGS, int &val)
dar store the corresponding flags as FSA (12 boolean flags/FSA)

the FSA fsan_creation_date is fetched by another method, too long to 
expose here (see filesystem_specific_attribute_list::fill_HFS_FSA_with()
in src/libdar/filesystem_specific_attribute.cpp for more details)

Dar does not know much about the underlying filesystem, it justs asks 
file properties to the OS, depending on the OS response (or failure), as 
the first call gives information on 12 FSAs at once, you usually end 
with either 0, 1, 12 or 13 FSAs attached to a given file.

> 
> This was created on dar 2.7.8, from a build before I enabled libthreadar
> linking, so no parallel features are used.  zstd compression is enabled,
> but the exclude rules mean that the majority of the data by bytes (and
> probably also by inode) is not being compressed since it is a lot of
> image and video files.  There is no encryption in use, and no rsync sig
> options in use.
> 
> dar -q -l on the isolated catalog shows:
> 
> total number of inode : 1295439
> fully saved           : 1295439
> binay delta patch     : 0
> inode metadata only   : 0
> distribution of inode(s)
>   - directories        : 67694
>   - plain files        : 1224095
>   - symbolic links     : 3647
>   - named pipes        : 3
>   - unix sockets       : 0
>   - character devices  : 0
>   - block devices      : 0
>   - Door entries       : 0
> hard links information
>   - number of inode with hard link           : 116267
>   - number of reference to hard linked inodes: 246019
> destroyed entries information
>     0 file(s) have been record as destroyed since backup of reference
> 
> /usr/bin/time report on dar -q -l for this catalog:
> 
> 12.83user 1.02system 0:13.87elapsed 99%CPU (0avgtext+0avgdata 1114820maxresident)k
> 159715inputs+0outputs (0major+283922minor)pagefaults 0swaps
> 
> which is a bit less than it was using during -c, but then I'm probably
> not asking it to do as much with it.

yep

> 
> Incidentally, in a bit of an echo of a conversation from a little while
> back, I observe that when I use -q -l on the last slice of an archive,
> as opposed to an isolated catalog, it will easily report the full size
> of the archive (including all slices, I believe), while when run on an
> isolated catalog, it only reports the size of the catalog.

Release 2.7.10 only shows in addition, the slicing information of 
archive of reference, something that is stored in the isolated catalog.

The size of the whole backup is computed from the slicing information, 
and the fact for a backup (not an isolated catalog) at that time we know 
how much slice it has been split into and we also know the size of the 
last slice.

But the size of the whole backup is not stored in a backup nor in an 
isolated catalog, so it cannot be computed (and shown) when reading a 
isolated catalog (it lack the number of slice and the size of the last 
slice). Adding this information is possible but requires an archive 
format change, so it will be for next major release (version 2.8.0)

> 
> By the time this project is done, I may have around 3 million inodes
> represented.
> 
> - John
>