Re: unchanged files backed up multiple times

Bill,

Thank you very much for the information. I'll play with this and see if I
can figure something out. I wouldn't think the mtime should have changed,
but if it did, bacula was just doing its job.

Regards,
Robert Gerber
402-237-8692
[email protected]


On Thu, Dec 12, 2024 at 5:20 PM Bill Arlofski via Bacula-users <
[email protected]> wrote:

> On 12/12/24 3:28 PM, Rob Gerber wrote:
> > Hello,
> >
> > A few very large files don't appear to have changed since the last full
> backup, yet are being backed up repeatedly by
> > incremental / differential backups. Why?
> >
> > Details:
> > I am using bacula to backup a fileserver for one of my customers. For
> the primary share, I am using a full/diff/inc backup
> > cycle. I have a few very large files that won't change (system images
> for user PCs). I didn't want to use up my storage
> > backing up those system images multiple times, so I set up a simpler
> backup cycle that would take a full once, then a diff
> > and inc periodically thereafter. The expectation was that the files
> wouldn't change and we'd rarely add files, so most
> > backups would be 0 files / 0 bytes. The user's PC files are routinely
> backed up by a separate service.
> >
> > The problem is that the system images have been backed up 3 times during
> the last 6 months. I picked one of the system image
> > files, and ran an SQL query for it against the database. The results
> indicated that the target file was backed up 3 times.
> > The file hashes were the same each time, but the lstat column had some
> differences. Unfortunately, the metadata is encoded
> > somehow and I dont understand what changed. I ran sha512sum against the
> file in question, but its output was different than
> > that found in the bacula database. I used python to base64 decode the
> catalog hash, and when the output is displayed in hex
> > it matches the hash provided by sha512sum. So the file appears to be
> unchanged from its first time being backed up, yet has
> > been backed up 3 times.
> [...snip...]
> >
> > The file was backed up 3 times.
> > Jobid 70: full backup.
> > Jobid 642: incremental backup
> > Jobid 667: differential backup.
> >
> > It makes sense that a differential backup after an incremental backup
> would contain the contents of the incremental. However,
> > the bacula database has the same hash for each of the 3 backups. Doesn't
> that imply that the file hasn't changed? I don't
> > know why it was backed up again in the incremental backup. Please note
> that there were many incremental or differential
> > backups ran daily that didn't see any changes or needs to back up any
> files in this dataset - except for these few times.
> >
> > My query is in the pastebin link (I don't know if the length of the
> query will mess up someone's email client).
> > https://pastebin.com/6PCbZ21s <https://pastebin.com/6PCbZ21s>
> >
> > Here is the metadata section from the query:
> > |                                 lstat             |
> > +-----------------------------------------------------------------------
> > | P0j TaAhK IHw B Pu Pw A MQ2A6aB BAA BiGwPA BmB6ge BmB6gdBmB6ge A A C |
> > | P0j TaAhK IHw B Pu Pw A MQ2A6aB BAA BiGwPA BmNcn/ BmB6gdBnQLI1 A A C |
> > | P0j TaAhK IHw B Pu Pw A MQ2A6aB BAA BiGwPA BnQXpa BmB6gdBnQLI1 A A C |
> [...snip...]
> >
> > Here is the relevant fileset:
> > FileSet {
> >    Name = "Stallone_macrium_backups"
> >    Include {
> >      Options {
> >        signature = SHA512
> >   Verify = s3
> >      }
> > File = /mnt/data/shares/cncshare/Craeon/Backup/macrium
> >    }
> >
> >    Exclude {
> >    }
> > }
> >
> > If there is something else I can provide, please let me know.
> >
> > Regards,
> > Robert Gerber
>
> Hello Rob,
>
> For the first two, we can see that the lstat decoded tells us that the
> mtime and atime have changed:
> ----8<----
> *.bvfs_decode_lstat lstat="P0j TaAhK IHw B Pu Pw A MQ2A6aB BAA BiGwPA
> BmB6ge BmB6gdBmB6ge A A C"
> st_nlink=1
> st_mode=33264
> perm=-rwxrw----
> st_uid=1006
> st_gid=1008
> st_size=842719798913
> st_blocks=1645937600
> st_ino=325584970
> st_ctime=0
> st_mtime=6952011706864740382         <----
> st_atime=1711777822                  <----
> st_dev=64803
> LinkFI=0
> *.bvfs_decode_lstat lstat="P0j TaAhK IHw B Pu Pw A MQ2A6aB BAA BiGwPA
> BmNcn/ BmB6gdBnQLI1 A A C"
> st_nlink=1
> st_mode=33264
> perm=-rwxrw----
> st_uid=1006
> st_gid=1008
> st_size=842719798913
> st_blocks=1645937600
> st_ino=325584970
> st_ctime=0
> st_mtime=6952011706885255733        <---- More recent time
> st_atime=1714801151                 <---- More recent time
> st_dev=64803
> LinkFI=0
> ----8<----
>
> So the second one was backed up because it was indeed modified.
>
>
> But for the third one the only thing I see that changed was the atime:
> ----8<----
> *.bvfs_decode_lstat lstat="P0j TaAhK IHw B Pu Pw A MQ2A6aB BAA BiGwPA
> BnQXpa BmB6gdBnQLI1 A A C"
> st_nlink=1
> st_mode=33264
> perm=-rwxrw----
> st_uid=1006
> st_gid=1008
> st_size=842719798913
> st_blocks=1645937600
> st_ino=325584970
> st_ctime=0
> st_mtime=6952011706885255733
> st_atime=1732344410                 <---- More recent atime than the
> previous one, should not matter, not normally tested :)
> st_dev=64803
> LinkFI=0
> ----8<----
>
>
> So, this looks correct.   It gets backed by the full (jobid 70), then
> there is a change and it gets backed by the Inc (jobid
> 642), then of course it will get backed bu the subsequent differential
> (jobid 667)
>
> If you run another incremental, this file should not get backed up (unless
> of course it gets modified since the Differential :)
>
> I would double check that the jobids were actually run at the levels we
> think, and in the order we think.  So, it seems Full,
> Inc, Diff to me here, and that would explain the three backups with the
> levels run in that order with a modification
> occurring before the Inc.
>
> It's been a long day, so I may have this wrong. :)
>
>
> Hope this helps,
> Bill
>
> --
> Bill Arlofski
> [email protected]
> _______________________________________________
> Bacula-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>

_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users