Re: Backups not working after running out of space

Thanks Eric,

Idea #1 didn't help (none of the .gz files where corrupt), but the 
diagnostic messages I got running idea #2 helped me realize that I 
needed to remove a(nother) current_mirror, and after I did that the 
regress succeed, and a subsequent backup also succeeded.  After that, 
however, I took your initial advice, renamed that repo (I'll keep it 
around until my daily remove increments --older-than 1Y has removed all 
it's increments) and created a new repo from scratch last night.

Thanks for your help.

On 2023-11-16 04:06, Eric Lavarde Eric-at-Lavar.de |rdiff-backup-users| 
wrote:
> Hi,
>
> it gets difficult, once a shortage in storage happens, you need to be 
> lucky for rdiff-backup to break at a place where it can recover from. 
> My recommendation would be to create a new repo from scratch, and 
> forget about the broken one. That's I would anyway because even if you 
> manage somehow to recover, you'll never know if it's absolutely clean.
>
> Few ideas though how you might be able to recover:
>
> 1. write a small script to verify all compressed files (.gz) and 
> remove them if they're corrupt
> 2. call rdiff-backup regress with --allow-duplicate-timestamps
>
> but it's hoping for the best...
>
> KR, Eric
>
> PS: that's why I recommend all my customers to monitor at least 
> storage, from everything else you can generally recover more or less 
> gracefully. Data you can't recover.
>
> On 15.11.2023 20:52, [email protected] wrote:
>> Anyone have any suggestions?
>>
>> If nothing else, is there some way I could remove appropriate files
>> from the rdiff-backup-data directory to get my backups working again?
>>
>> thanks,
>> Peter
>>
>> On 2023-11-13 13:01, Peter Canning wrote:
>>> Here's the command I've been using, with -v 6 added, followed by the 
>>> output it produced:
>>>> C:\Users\pcanning\rdiff-backup-2.2.2-64\rdiff-backup.exe -v 6 
>>>> backup --print-statistics `
>>>>             --exclude "H:/Lightroom/Processing Catalog/Backups" `
>>>>             --exclude "H:/Lightroom/Processing Catalog/Processing 
>>>> Catalog Previews.lrdata" `
>>>>             --exclude "H:/Lightroom/Processing Catalog/Processing 
>>>> Catalog-v12 Previews.lrdata" `
>>>>             "H:/Lightroom/Processing Catalog" `
>>>> \\nas4free.local\lightroom-catalogs-backups\Lightroom-Processing-Catalog.rdiff-backup 
>>>>
>>>
>>>> C:\Users\pcanning\rdiff-backup-2.2.2-64\rdiff-backup.exe : WARNING: 
>>>> Previous backup seems to have failed, regressing destination now
>>>> At line:1 char:1
>>>> + C:\Users\pcanning\rdiff-backup-2.2.2-64\rdiff-backup.exe -v 6 
>>>> backup  ...
>>>> + 
>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>     + CategoryInfo          : NotSpecified: (WARNING: 
>>>> Previo...destination now:String) [], RemoteException
>>>>     + FullyQualifiedErrorId : NativeCommandError
>>>>
>>>> WARNING: Could not find mirror metadata file. Metadata will be read 
>>>> from filesystem instead
>>>> Fatal Error: No metadata for time Thu Nov  9 02:35:42 2023 
>>>> (1699526142) found, cannot regress
>>>> * Using repository 
>>>> '//nas4free.local/lightroom-catalogs-backups/Lightroom-Processing-Catalog.rdiff-backup'
>>>> * Hardlinks disabled by default on Windows
>>>> * Unable to import module (py)xattr. Extended attributes not 
>>>> supported on filesystem at path H:/Lightroom/Processing Catalog
>>>> * Unable to import module posix1e from pylibacl package. POSIX ACLs 
>>>> not supported on filesystem at path H:/Lightroom/Processing Catalog
>>>> * -----------------------------------------------------------------
>>>> Detected abilities for source (read only) file system:
>>>>   Access control lists                         Off
>>>>   Extended attributes                          Off
>>>>   Windows access control lists                 On
>>>>   Case sensitivity                             Off
>>>>   Escape DOS devices                           On
>>>>   Escape trailing spaces                       On
>>>>   Mac OS X style resource forks                Off
>>>>   Mac OS X Finder information                  Off
>>>> -----------------------------------------------------------------
>>>> * Directories on file system at path 
>>>> //nas4free.local/lightroom-catalogs-backups/Lightroom-Processing-Catalog.rdiff-backup/rdiff-backup-data/rdiff-backup.tmp.0 
>>>> are not fsyncable. Assuming it's unnecessary.
>>>> * Unable to import module (py)xattr. Extended attributes not 
>>>> supported on filesystem at path 
>>>> //nas4free.local/lightroom-catalogs-backups/Lightroom-Processing-Catalog.rdiff-backup/rdiff-backup-data/rdiff-ba
>>>> ckup.tmp.0
>>>> * Unable to import module posix1e from pylibacl package. POSIX ACLs 
>>>> not supported on filesystem at path 
>>>> //nas4free.local/lightroom-catalogs-backups/Lightroom-Processing-Catalog.rdiff-backup/rdiff-backup-da
>>>> ta/rdiff-backup.tmp.0
>>>> * -----------------------------------------------------------------
>>>> Detected abilities for destination (read/write) file system:
>>>>   Ownership changing                           Off
>>>>   Hard linking                                 On
>>>>   fsync() directories                          Off
>>>>   Directory inc permissions                    Off
>>>>   High-bit permissions                         On
>>>>   Symlink permissions                          Off
>>>>   Extended filenames                           On
>>>>   Windows reserved filenames                   On
>>>>   Access control lists                         Off
>>>>   Extended attributes                          Off
>>>>   Windows access control lists                 On
>>>>   Case sensitivity                             Off
>>>>   Escape DOS devices                           On
>>>>   Escape trailing spaces                       On
>>>>   Mac OS X style resource forks                Off
>>>>   Mac OS X Finder information                  Off
>>>> -----------------------------------------------------------------
>>>> * Backup: escape_dos_devices = False
>>>> * Backup: escape_trailing_spaces = False
>>>> * Enabled use_compatible_timestamps
>>>> NOTE: Symbolic links excluded by default on Windows
>>>> NOTE: Regressing to date/time Thu Nov  9 02:35:42 2023
>>>> * Cleaning up
>>>
>>>
>>> On 2023-11-13 04:57, Patrik Dufresne patrik-at-ikus-soft.com 
>>> |rdiff-backup-users| wrote:
>>>> Hello Peter,
>>>>
>>>> I apologize for the inconvenience you've experienced in recovering 
>>>> your
>>>> backup after encountering space issues. Typically, rdiff-backup can 
>>>> handle
>>>> recovery seamlessly following an increase in disk space. However, 
>>>> it seems
>>>> that the manual changes you made might have caused interference.
>>>>
>>>> To assist you more effectively, could you please share the backup 
>>>> command
>>>> line you are currently using? Additionally, I recommend executing the
>>>> command with higher verbosity by adding "-v 6." This will provide more
>>>> detailed information that can help us pinpoint the issue.
>>>>
>>>> In the expected scenario, rdiff-backup should detect the 
>>>> interruption in
>>>> the previous backup and initiate a regression of the repository to
>>>> eliminate any intermediate changes. This process is designed to 
>>>> restore the
>>>> health of your repository. Subsequently, the backup will resume its 
>>>> normal
>>>> operation.
>>>>
>>>> Your cooperation in providing the requested information will 
>>>> greatly aid us
>>>> in resolving this issue promptly. If you have any questions or 
>>>> concerns,
>>>> please don't hesitate to reach out.
>>>>
>>>> Thank you for your understanding and cooperation.
>>>>
>>>>
>>>> Le lun. 13 nov. 2023, à 00 h 12,<[email protected]>  a écrit :
>>>>
>>>>> Using rdiff-backup 2.2.2 on Windows 10 Pro 22H2, I have a 
>>>>> scheduled task
>>>>> that runs my system backup PowerShell script at 2 AM every 
>>>>> morning.  The
>>>>> script runs rdiff-backup several times to backup several different
>>>>> directories.  A few days ago one of the backups failed because the 
>>>>> NAS
>>>>> volume containing targetdir filled up.  I took some remedial steps 
>>>>> that
>>>>> didn't help, but at this point I don't remember exactly what I 
>>>>> did.  I do
>>>>> know that this first thing I tried, not thinking very careful, and
>>>>> realizing that I had recently created some large files in 
>>>>> sourcedir that I
>>>>> didn't need any more, I deleted those files and tried doing a backup
>>>>> running rdiff-backup manually (incorrectly thinking rdiff-backup 
>>>>> would
>>>>> remove some files from targetdir).  Of course since rdiff-backup is
>>>>> creating incremental backups, and targetdir was still full, that 
>>>>> backup
>>>>> failed too.  At some point I increased the size of the NAS volume, 
>>>>> and here
>>>>> is where my memory is failing me. I probably tried running 
>>>>> rdiff-backup
>>>>> manually again.   Either then, or shortly thereafter, I started 
>>>>> getting
>>>>> "regressing destination" and then rdiff-backup failing whenever I 
>>>>> tried to
>>>>> backup or verify, so I attempted to follow the instructions in FAQ 
>>>>> item
>>>>> 15.  I did some other troubleshooting that I don't remember, but the
>>>>> backups are still failing.
>>>>>
>>>>> Now, running "rdiff-backup.exe backup --print-statistics ..." doesn't
>>>>> print the expected statistics (so I assume it didn't backup 
>>>>> anything) and
>>>>> produces the following output:
>>>>>
>>>>> C:\Users\pcanning\rdiff-backup-2.2.2-64\rdiff-backup.exe : WARNING:
>>>>> Previous backup seems to have failed, regressing destination now
>>>>> At line:1 char:1
>>>>> + C:\Users\pcanning\rdiff-backup-2.2.2-64\rdiff-backup.exe backup 
>>>>> --pri ...
>>>>> + 
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>      + CategoryInfo          : NotSpecified: (WARNING: 
>>>>> Previo...destination
>>>>> now:String) [], RemoteException
>>>>>      + FullyQualifiedErrorId : NativeCommandError
>>>>>
>>>>> WARNING: Could not find mirror metadata file. Metadata will be 
>>>>> read from
>>>>> filesystem instead
>>>>> Fatal Error: No metadata for time Thu Nov  9 02:35:42 2023 
>>>>> (1699526142)
>>>>> found, cannot regress
>>>>> NOTE: Symbolic links excluded by default on Windows
>>>>> NOTE: Regressing to date/time Thu Nov  9 02:35:42 2023
>>>>>
>>>>> Running "rdiff-backup.exe verify ..." produces the following output:
>>>>>
>>>>> C:\Users\pcanning\rdiff-backup-2.2.2-64\rdiff-backup.exe : 
>>>>> WARNING: Two
>>>>> different times for current mirror found
>>>>> At line:1 char:1
>>>>> + C:\Users\pcanning\rdiff-backup-2.2.2-64\rdiff-backup.exe verify 
>>>>> \\nas ...
>>>>> + 
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>      + CategoryInfo          : NotSpecified: (WARNING: Two di...nt 
>>>>> mirror
>>>>> found:String) [], RemoteException
>>>>>      + FullyQualifiedErrorId : NativeCommandError
>>>>>
>>>>> WARNING: Could not find mirror metadata file. Metadata will be 
>>>>> read from
>>>>> filesystem instead
>>>>> WARNING: Mirror metadata not found, reading from directory
>>>>>
>>>>> followed by many lines saying
>>>>>
>>>>> WARNING: Cannot find SHA1 digest for file ..., perhaps because this
>>>>> feature was added in v1.1.1
>>>>>
>>>>> and ending with the line:
>>>>> NOTE: Verification found 1428 files without hash, all others could be
>>>>> verified successfully
>>>>>
>>>>> Any suggestions for how to get my backups (and verifies) working 
>>>>> again?
>>>>>
>>>>> thanks,
>>>>> Peter
>>>>>
>>>>>
>>>>>
>>>
>