Re: Multi-threaded archive test mode

* On 1/28/24 20:23, Denis Corbin wrote:
> Though there is already such multi-threaded
> creation/testing/extracting/comparing/... mode!

There's the -G/--multi-thread option, but that only applies to the compression
and encryption steps.

I was suggesting a multi-thread option for the test operation when used with
bare, i.e., uncompressed and unencrypted, dar archives.


> IMHO, the performance limitation is usually not due to disk I/O but to
> compression/decompression process [...]
>
> To be able to parallelize these [...]
>
> Encryption was already done per block so if you have libthreadar
> available with dar/libdar, multi-threading should work out of the box.

This is all true, but only if you actually use compression or encryption. :)

For bare archives, dar's operations are mostly I/O-bound, minus a small penalty
for CRC calculations, escape marks etc., which are all reasonably fast (compared
to reading data from disks, anyway).


> Not exaclty, when a file is sparse, it has been stored as such to
> replace holes by an in-lined structure telling the size of the hole. At
> reading/testing time, all the data is read including these in-lined
> placeholder structures and of course the CRC is calculated accordingly,
> but there is not less disk pressure due to a sparse file, there is just
> less time required to read a sparse file than an equivalent file in size
> without holes.

Oh but in testing mode, there is a *lot* less I/O pressure with sparse files!

I'm currently testing an archive with a lot of sparse files (mostly VM disk
images). When doing so, the dar process only reads a few KB/s off the disk and
is hogging a CPU core intensely, computing CRC values. Totally CPU-bound.

In such a case, it could test the archive so much faster if it was just
processing other files in alternate threads. With my machine, I could easily see
an 4x speedup for the testing operation with all the sparse files in the archive.


> Even for testing operation, where there is only a reading side (no
> writing side), this could lead I/O disk pressure and penalty for
> mechanical ones [...]

Yep, I realized that and addressed in the original mail, but for some workloads,
that's a reasonable trade-off if the speedup is so much higher than the penalty.


> But, that the idea is interesting, though it is only testing operation
> that could leverage it...

Also yup, that's only interesting for test mode.


> Yes, correct. The de/compression per block can still work for sequential
> read mode, let me know about the performance gain you get compared to
> the single thread using stream encryption method (default method).

I can't, because I don't use compression or encryption. :)


>> The -G/--multi-thread option is also not a good for it, it probably should be
>> a new option such as --test-threads.
>
> Not sure to understand what you mean about -G option?

Sorry, I've had a typo in there. I meant that the existing -G option is not a
good *fit* for this new feature I had in mind, because it's already used for
compression and encryption threading. A new test mode threading option would
make more sense than leveraging -G for this, especially (or only?) for bare
archives.



Mihai