Re: Multi-threaded archive test mode

Denis Corbin <[email protected]> Sun, 28 Jan 2024 20:24:08 +0100
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>
On 28/01/2024 00:31, Mihai Moldovan wrote:
 > Hi Denis >

Hi Mihai,

 >
 > If you ever find yourself bored, here's a potentially helpful new 
feature idea:
 > multi-threaded archive test mode support.

🙂 thanks for the idea


Though there is already such multi-threaded 
creation/testing/extracting/comparing/... mode!

 >
 > Why?
 >
 > dar's test mode can be really, really, really slow. All other modes are
 > I/O-bound, so wouldn't benefit from something like that, but test 
mode is different.

IMHO, the performance limitation is usually not due to disk I/O but to 
compression/decompression process and in second to the 
encryption/decryption one. OK it depends on the data under process, but 
for a common Linux computer I usually see CPU at high rate without 
having the kernel suspending the applications (uninterruptible state 'D' 
for example).

To be able to parallelize these (and use multiple thread), I had to 
change the compression algorithm to compress per block at archive 
creation time. This way, blocks can be distributed among different 
thread and gathered after treatment to go to disk. See -G and -z options 
for details. Thus you must activate encryption per block at creation 
time to leverage this multi-threading feature for decompression at 
testing time.

Encryption was already done per block so if you have libthreadar 
available with dar/libdar, multi-threading should work out of the box.

 >
 > Usually, the test mode is also more or less I/O-bound, either due to 
compression
 > being used (and decompression being reasonably fast) or CRC 
calculations going
 > on, which are also fast.

CRC calculation comes in third place long after encryption/decryption.

And, in forth place, there is also escape mark that cost some CPU cycles:

Escape/tape marks are what make sequential read possible: they are 
inserted all along the backup at each new file, data, ea, CRC, ... and 
to avoid confusing with data from file looking as a mark, all file's 
data and EA is looked for what could be considered as a mark and 
eventually escaped for it cannot be seen as a mark.

You can disable this by mean of -at option (but lose the ability to read 
a backup sequentially) and save some CPU cycles. And use "-1 0" to 
disable sparse file consideration.

 >
 > Unfortunately, there's an edge case to all of this: large sparse files.
 >
 > When handling sparse files, dar naturally knows that holes are 
all-zero, so it
 > doesn't need to actually read any data from the archive. Instead, it 
will just
 > happily calculate CRC values in a tight CPU-bound loop.

Not exaclty, when a file is sparse, it has been stored as such to 
replace holes by an in-lined structure telling the size of the hole. At 
reading/testing time, all the data is read including these in-lined 
placeholder structures and of course the CRC is calculated accordingly, 
but there is not less disk pressure due to a sparse file, there is just 
less time required to read a sparse file than an equivalent file in size 
without holes.

 >
 > With just one thread checking the data, a lot of wall clock time is 
wasted,
 > while the machine almost idle I/O-wise and only hogging up one 
logical CPU core.

correct, that's what I also observe in single-thread mode.

 >
 > To counter this, dar could check archive files in multiple threads, 
one file per
 > thread. This would make sure that either CPU or I/O resources are 
always fully used.

Even for testing operation, where there is only a reading side (no 
writing side), this could lead I/O disk pressure and penalty for 
mechanical ones, having the heads seeking forth and back to address 
several threads on different files. And second it could have a more 
important memory impact than the current de/compression per block/thread 
(-G option)

But, that the idea is interesting, though it is only testing operation 
that could leverage it...

 >
 >
 > I'm also aware of drawbacks to this approach. I was able to come up 
with these:
 >    - Doesn't work in sequential read mode, unless all files to be 
processed are
 >      fully buffered in memory first, which is a TERRIBLE idea for obvious
 >      reasons.

Yes, correct. The de/compression per block can still work for sequential 
read mode, let me know about the performance gain you get compared to 
the single thread using stream encryption method (default method).

 >    - Can reduce read performance drastically with conventional hard 
disk drives
 >      due to constant seeks within the archive.

yes, exactly

 >
 > Due to that, I don't think that this feature should be enabled by 
default (and
 > it shouldn't be possible to enable it for sequential read mode at all).
 >
 > The -G/--multi-thread option is also not a good for it, it probably 
should be a
 > new option such as --test-threads.

Not sure to understand what you mean about -G option? By the way, have 
you changed the compression to use per block compression to see the gain 
of multi-treading actually implemented in lidar?

  -G 4

note that if the archive has been compressed without block it cannot be 
uncompressed per block (and rely on multi-threading), you need to use 
block compression at creation time to also be able to rely on 
mutli-thread (-G option) for any other operation (including backup testing).

 >
 >
 > I'm looking forward to your analysis.
 >

Cheers,

 >
 >
 > Mihai
 >
 >

Denis
OpenPGP_signature.asc (application/pgp-signature, 840 B)