Re: Need help determining sliced archive corruption

>> Other than a bug in dar itself (which I prefer to not consider thanks
>> to your exceptional job)
>
> exceptional or not, all software can have bugs and I never excluded it,
> the problem is the lack of reproducibility in your context...

That's true, until a new backup comes up with the same issue. 
Full-system backups take a while to finish and they don't run every week 
so let's stay tuned.

Meanwhile, you suggested we rule out disk errors, transfer errors and 
wrong order of slices. But, I'm afraid, faulty RAM won't get us anywhere 
either because, as I have mentioned in the first message, I have two 
backups like that, created 1 month apart, from two different systems 
(different hardware, locations, age, usage patterns, etc. - only backup 
mechanism and Ubuntu/dar versions are the same). Both backups report

   Sorry, file size is unknown at this step of the program.

to -l -0 -q; strace shows reading of the last slice, then some preceding 
slice; file counts in -affs and -0 don't (fully) match:

dar -t -A -affs:

- 5689 x "compressed data CRC error"
- 1448 x "CRC error: data corruption"
- I haven't read through the entire list but it seems there are no 
directories and corruptions affect files of various sizes from tiny to 
huge and in different slices

   --------------------------------------------
   274471 item(s) treated
   7137 item(s) with error
   0 item(s) ignored (excluded by filters)
   --------------------------------------------
   Total number of items considered: 281608
   --------------------------------------------


dar -t -0:

- 8 x "No escape mark found for that file"

   --------------------------------------------
   724494 item(s) treated
   8 item(s) with error
   0 item(s) ignored (excluded by filters)
   --------------------------------------------
   Total number of items considered: 724502
   --------------------------------------------


dar -l <catalogue> -q:

   Archive version format               : 11.3
   Compression algorithm used           : bzip2
   Compression block size used          : 0
   Symmetric key encryption used        : none
   Asymmetric key encryption used       : none
   Archive is signed                    : no
   Sequential reading marks             : present
   User comment                         : redacted
   Catalogue size in archive            : 24538535 bytes

   Archive is composed of 1 file(s)
   File size: 24538747 bytes
   The global data compression ratio is:   25%

   WARNING! This archive only contains the catalogue of another archive, 
it can only be used as reference for differential backup or as rescue in 
case of corruption of the original archive's content. You cannot restore 
any data from this archive alone

   Archive of reference slicing:
           Other slices: 2147483648 byte(s)

   in-place path: redacted

   CATALOGUE CONTENTS :

   total number of inode : 724464
   fully saved           : 724464
   binay delta patch     : 0
   inode metadata only   : 0
   distribution of inode(s)
    - directories        : 28009
    - plain files        : 687898
    - symbolic links     : 8403
    - named pipes        : 12
    - unix sockets       : 138
    - character devices  : 3
    - block devices      : 1
    - Door entries       : 0
   hard links information
    - number of inode with hard link           : 21
    - number of reference to hard linked inodes: 51
   destroyed entries information
      0 file(s) have been record as destroyed since backup of reference


This backup consists of 95 slices totaling 190 GiB.

Certainly it'd be too much of a coincidence if two servers exhibited 
abnormal (RAM) behavior only during the creation of dar backups?

>> I still think it has something to do with wrong order/number of slices.
>
> to be honest, I don't see how mixing or even renumbering slices could
> cause the behavior you hit...

I only suggested that because dar doesn't detect out-of-sequence slices, 
emitting symptomatic errors instead. If there were such a combination of 
factors that could theoretically result in what I'm observing, I'd opt 
for the mixed slices cause. But I surely trust you on this one... then 
what else remains? What could potentially result in such archives?

>>> When testing the backup in sequential-read mode (-0/--sequential-read
>>> option) could you add the following option:
>>>      -E "echo reading slice %N"
>>> and report the last slice being reported as being read by this added -E
>>> option, before the failure takes place?
>>
>> As per the above output, it says "reading slice 21".
>>
>>> Also could you tell me the number of the last slice available?
>>
>> There are arc.1.dar through arc.29.dar, the total of 29 slices.
>
> OK so you don't miss slices, but have those corrupted at different
> places.

But why does it report "Unexpected value while reading archive version" 
after reading slice 21 after 29? It also happens with the other backup 
except it reads slice 95 then 76.

> In sequential read mode, obviously, the archive is read in sequence,
> after having read the data of a files, dar reads right after the
> expected CRC and compares it with the one calculated on the retreived
> data and eventually reports a difference. Then it skips forward looking
> for the next tape mark and once again and the loop repeats...
>
> Here, corruption occurred in four different files (and slices), not only
> the very last files (in the last slice) as I had understood, which is
> not the pattern of a truncated slice.

It makes sense. It happens to the other backup too.

> Can you double check with for example memtest86+ ?

The system needs to be powered down for that, it's not something we can 
do at this time...

>> 1. Archive UUID, as it's currently implemented, is perfect: (1) it
>> practically guarantees that one cannot mix up slices from different
>> archives; (2) dar itself checks that when testing; (3) can be
>> displayed with `file`:
>>
>> arc.10.dar: dar archive, label "82af71c2 00000000 fffff752"
>>
>> I could only suggest storing UUID of the reference archive in the -@
>> catalogue too (since -@ gets its own different UUID) and for dar
>> itself to provide some way to view this UUID. In any case, UUID helps
>> one work out the consistency among archives - but there's nothing
>> similar for working out slices for the archive itself, and it's a
>> problem for me.
>
> if you have a look at dar's documentation about the archive format, you
> will see that this is done with more flexibility:
> - internal name : links slices of an archive together
> - data name : links a catalog (or isolated catalog) to the data set to
> use. This prevents one trying to restore an archive with the help of an
> unrelated isolated catalog.
>
> http://dar.linux.free.fr/doc/Notes.html#archive_structure

I have glanced over that page but I have missed this. It's great and 
eliminates any confusion in mismatched slices/catalogues. Now, if only 
dar could provide some way to view this info, preferably in -0 mode for 
potentially corrupted slices. For now, my only diagnostic tool is 
`file`, and it's only reporting internal name it seems.

>> 2. Byte sizes of non/last slices would allow basic integrity check but
>> (1) `file` doesn't show them; (2) dar doesn't show them in -l -0 mode
>> (so unable to extract from invalid file); (3) dar -l -q works on
>> isolated catalogue but only reports non-last slice size - given our
>> problem here is likely last slice being truncated, it doesn't really
>> help.
>
> dar cannot know in advance how much slices will be necessary nor what
> will be the size of the last slice.

Of course it's streaming so I'm not suggesting storing total number of 
slices in each slice. My suggestion here is to store each slice's size - 
which is of course known beforehand. (I've written "non/last" but I 
shall recede from the "last" part indeed.)

> The sar layer (which manages slices)
> processes a stream of bytes from upper layers (ciphering, compression,
> filtering, and so forth) which is generated on the fly from the
> filesystem to backup, the sar takes this flow and creates slices from it
> up to the time the flow dries up.
>
> What is stored at the beginning of *each slice* is the so called
> slice-layout:
> - first slice max size,
> - other slices max size,
> - slice header size
> - and slice format (2 formats so far).
>
> Only this could be shown in sequential-read mode, not what is displayed
> today in direct mode which adds to this the overall size of the archive
> and the size of the last slice (easy to get because in that mode this is
> the last slice that get read first).

Okay, the ability to view the details of slice-layout in -0 mode would 
be welcome. I'd say any header info available to dar should be made 
visible on equal terms.

>> 3. Slice number is what I truly miss. According to my tests, messing
>> with non-last slices (removing, duplicating, reordering) is usually
>> detected by dar as different types of data inconsistency so even if
>> you have all slices perfectly good and well, just in the wrong order,
>> you would think the backup is botched, and those errors would only be
>> misleading.
>
> the slice number is expected to be in the filename and it is not
> expected that users would change/renumber it...

Let me explain our storage strategy briefly. Backup files are held off 
site for obvious reasons; the storage space is a set of buckets (could 
be one or more per server but they're static) storing opaque blobs whose 
only characteristics are acceptance date and data itself. Date is stored 
for FIFO-like purging. Blobs can be moved or duplicated elsewhere behind 
the scenes.

I understand it may not be the most elegant or efficient solution but 
that's what we have now. Allowing free form metadata like file names 
would complicate blob management. For backup purposes, this is not a 
problem because each backup location is tied to a particular bucket 
where archives and slices go in chronological order so by inspecting 
date and data size one can clearly divide entries into individual 
archives even if there are no file names. We store additional data in 
dar's user comment field, but it's obviously archive-wise.

So essentially our backup directory looks like this:

   402 | Sun Feb 22 11:43:58 AM UTC 2026 | 167511621 bytes
   403 | Tue Mar  3 06:56:46 PM UTC 2026 | 10613869728 bytes
   404 | Tue Mar  3 07:13:29 PM UTC 2026 | 10613869728 bytes
   405 | Tue Mar  3 07:36:56 PM UTC 2026 | 635571924 bytes

> This is something unclear to me: you have this slice number information
> in the slice filename... isn't it the easiest way to know what slice
> number is a slice? And how much slice do one have? And also what is the
> latest slice?

My point is that in some scenarios file names are unavailable. Even if 
not for our storage system, file names may be missing as a result of 
data recovery from a faulty drive, for example. Sure, that should not 
normally happen, and if you use par2 then it would fix the names for you 
as well, but we're talking about last-chance recovery here, and that's 
why dar has the -0 mode, so why not take it a step further with one or 
two simple new fields in the header?

> What I don't understand is how you could
> accidentally miss the first slice, I mean how do you let dar accessing
> the slices as dar, even in sequential-read mode open a slice after a
> given filename and if it is not present it complains and waits.

To deploy or even inspect a backup, its blobs should be pulled from 
storage somewhere dar can run, then renamed or symlinked to form 
arc.N.dar. Human factor makes such mistakes possible. If you forgot one 
or more leading slices yet they're named arc.1.dar, arc.2.dar, etc. - 
dar won't "complain and wait", it'd proceed straight to throwing errors 
at you.

>> Also, out of curiosity, I tried copying one slice twice and got a ton
>> of CRC errors.
>  > > With the help of 4 extra bytes (= slice number) in each slice's
> header,
>> we would be able to identify the problem precisely.
>
> why only 4 bytes? This would limit the number of slices... even if today
> you think it is large enough, this is the same way of thinking that led
> to the year 2000 bug, the problem of year 2038, the upper and high
> memory in DOS some decades ago, the 2 GB files boundary (and need of
> large file support)... and so forth.

I understand your point. But even if we take 1 KiB per slice, 2^32 
slices would add up to 4.4 TiB of data... and more "realistic" 1 MiB per 
slice would be 4.5 PiB... but if that is a concern, you could use 64 
bits, that would be (let me look that up on Wikipedia) around 2 exabytes 
of data if each slice only held 1 byte of payload... I'm afraid we'd 
need more than quantum computers to cross that boundary...

> A better approach could be to add a random fixed size field at end of
> slice and have this same random value at the beginning of the next. But
> I don't see the use case as mentioned earlier about number of slices in
> filenames.

If we're in the territory where 2^32 slices may not be enough, our 
random numbers may not be random enough too, especially within millions 
of slices... If we were choosing, I'd just use a fixed length numeric 
field for the sake of simplicity. VLQ if you must, although I'd grade 
that as an overkill...

>> Not only will it tell us slice order, it will also tell us the total
>> number of slices (since identifying last slice is trivial).
>
> not to my standpoint, as all slices may not be visible at a given time
> in particular when using removal media or remote storage.

Again, I'm not advocating storing *total* number of slices anywhere. My 
point is that, with a simple "current slice index" field embedded in 
each slice-layout, it's possible to eliminate all slice order-related 
errors and usually even derive the total number of slices by taking last 
slice's counter (given that last slice is smaller than others, detecting 
it empirically is trivial). Sparing several bytes in each slice is a 
small price to pay for this benefit, IMHO.

>>  Storing total number of slices in the catalogue would take it even
>> further,
>
> this would break the ability to re-slice (dar_xform) an archive, once
> the isolated catalogue would have been created.

Okay, no worries, I take this suggestion back.

Thank you for still being on board with me on this!