Re: Multi-threaded archive test mode

Denis Corbin <[email protected]> Sun, 28 Jan 2024 21:33:08 +0100
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>
On 28/01/2024 20:55, Mihai Moldovan wrote:
> * On 1/28/24 20:23, Denis Corbin wrote:
>> Though there is already such multi-threaded
>> creation/testing/extracting/comparing/... mode!
> 
> There's the -G/--multi-thread option, but that only applies to the compression
> and encryption steps.
> 
> I was suggesting a multi-thread option for the test operation when used with
> bare, i.e., uncompressed and unencrypted, dar archives.
> 

OK, I missed understanding that you were neither ciphering no 
compressing data in your backup, sorry for that.

> 
[...]

> 
> This is all true, but only if you actually use compression or encryption. :)

yep :)

> 
> For bare archives, dar's operations are mostly I/O-bound, minus a small penalty
> for CRC calculations, escape marks etc., which are all reasonably fast (compared
> to reading data from disks, anyway).

Yes, I agree.

> 
> 
>> Not exaclty, when a file is sparse, it has been stored as such to
>> replace holes by an in-lined structure telling the size of the hole. At
>> reading/testing time, all the data is read including these in-lined
>> placeholder structures and of course the CRC is calculated accordingly,
>> but there is not less disk pressure due to a sparse file, there is just
>> less time required to read a sparse file than an equivalent file in size
>> without holes.
> 
> Oh but in testing mode, there is a *lot* less I/O pressure with sparse files!

this is surprising to me! I must review the code, but probably that 
after reading the in-lined hole placeholder, the sparse-file layer 
emulate the hole with zeros for the CRC algorithm above to work the same 
on plain and sparse file... this would explain the CPU cycles without 
I/O pressure you observed.

At restoration time, hopefully, there is no such zeroed bytes emulation, 
but the file is seek()ed over the holes and data written after it to 
re-create the hole on destination filesystem, if that one support sparse 
files.

> 
> I'm currently testing an archive with a lot of sparse files (mostly VM disk
> images). When doing so, the dar process only reads a few KB/s off the disk and
> is hogging a CPU core intensely, computing CRC values. Totally CPU-bound.

yep

> 
> In such a case, it could test the archive so much faster if it was just
> processing other files in alternate threads. With my machine, I could easily see
> an 4x speedup for the testing operation with all the sparse files in the archive.

Yes, now I better understand your need.

> 
> 
[...]
> 
> 
>>> The -G/--multi-thread option is also not a good for it, it probably should be
>>> a new option such as --test-threads.
>>
>> Not sure to understand what you mean about -G option?
> 
> Sorry, I've had a typo in there. I meant that the existing -G option is not a
> good *fit* for this new feature I had in mind, because it's already used for
> compression and encryption threading. A new test mode threading option would
> make more sense than leveraging -G for this, especially (or only?) for bare
> archives.

OK.

There is two possible things to do to speedup the testing process in 
your case:
- make the CRC code able to support holes and avoid emulating zeroed 
bytes at testing time
- add a per-thread per file testing option you mention.

The second is more complicated than the first... but worth 
consideration...  I just wonder whether this would be feasible... but
  I have to look at the details.


> 
> 
> 
> Mihai
> 
>
OpenPGP_signature.asc (application/pgp-signature, 840 B)