Re: Filesystem full errors when there is plenty of space? [SOLVED]

Hello Derek, I'm glad you figure it out

On 5/6/26 09:13, Derek Atkins wrote:
> Hi,
>
> Just to confirm -- I ran the backup manually yesterday with the --tempdir
> option pointing to a space adjacent to the target (so it had lots of free
> space) and it succeeded.  I've added that --tempdir option to this
> specific run to protect it in the future, and will add it more generally
> to my backup scripts if I see this error in the future.
>
> Thank you so much for pointing me in the right direction!
>
> -derek
>
> On Tue, May 5, 2026 10:48 am, Derek Atkins wrote:
>> Hi Patrik,
>>
>> Thank you for the inputs.
>> More inline below..
>>
>> On Tue, May 5, 2026 10:00 am, Patrik Dufresne via Any discussion of
>> rdiff-backup wrote:
>>> Hello Derek,
>>>
>>> Hard to tell with the information you provided, but this error might be
>>> caused by the following reasons:
>> Alas, the information I provided is all the information rdiff-backup gave
>> me!  I wish it provided more details!
>>
>>>   1.
>>>
>>>      *Not enough disk space at destination* (this doesn't seem to be your
>>>      case, as you have ~25TB free on /mnt/backup/)
>>>
>>>   2.
>>>
>>>      *Not enough inodes at destination*. You can check with:
>>>
>>>      |df -i /mnt/backup/ |
>>>
>>>      Note that encfs filesystems can sometimes have inode limitations
>>>      inherited from the underlying filesystem, so also check:
>>>
>>>      |df -i / |
>>>
>>>      (or whatever filesystem underlies your encfs mount)
>> I actually looked at the inodes after I sent the email and they all look
>> like there is plenty of room:
>>
>> [backup@backup ~]$ df -i /mnt/backup/
>> Filesystem          Inodes   IUsed       IFree IUse% Mounted on
>> encfs          51847503204 9129350 51838373854    1% /mnt/backup
>>
>> [backup@backup ~]$ df -i .
>> Filesystem                      Inodes  IUsed   IFree IUse% Mounted on
>> /dev/mapper/fedora_backup-root 7864320 102189 7762131    2% /
>>
>> NB: I did not check the underlying FS, but I am sure it's pretty much what
>> encfs shows.
>>
>>>   3.
>>>
>>>      *Not enough disk space or inodes in your TEMP folder*. rdiff-backup
>>>      may write temporary files to the system's temp directory (typically
>>>      |/tmp|) during regression/restore operations. You can check with:
>>>
>>>      |df /tmp df -i /tmp |
>> [backup@backup ~]$ df /tmp
>> Filesystem     1K-blocks  Used Available Use% Mounted on
>> tmpfs            3808324     4   3808320   1% /tmp
>> [backup@backup ~]$ df -i /tmp
>> Filesystem     Inodes IUsed  IFree IUse% Mounted on
>> tmpfs          952081     8 952073    1% /tmp
>>
>>>      With only ~8GB free on your |/| filesystem (which likely also hosts
>>>      |/tmp|), this could very well be the culprit, especially during a
>>>      regression pass which involves reconstructing files by applying
>>>      rdiff patches.
>>>
>>> Given your traceback, the error is happening during a *regression*
>>> (rolling back a previously failed backup), which involves reconstructing
>>> and writing out full file contents as temporary files. This can require
>>> significant temporary space depending on the size of the files being
>>> reconstructed.
>>>
>>> My best guess given your numbers is *option 3* — the 8GB free on your
>>> backup server's root filesystem may not be sufficient for the temporary
>>> files being written during regression.
>> Interesting observation.  My /tmp is on a tmpfs and only 3.7GB -- so even
>> less space free!  Right now none of TMP, TEMP, nor TMPDIR are set to
>> anything, so it's using /tmp -- and given your new information (to me)
>> about it reconstructing full files, I bet it's running out of space in
>> /tmp!
>>
>> As a test -- I'm re-running with "--tmpdir /mnt/backup/tmp" and we'll see
>> if that fixes the problem.
>>
>> Thanks!
>>
>> -derek
>>
>>>
>>> On 5/5/26 08:52, Derek Atkins wrote:
>>>> Hi
>>>>
>>>> I've suddenly, recently, starting getting "File System Full" errors from the backup of one of my servers. This particular 
>>>> server is still using 1.2.8. What I don't know is: what filesystem 
>>>> is full? There is plenty of space on the system being backed up: 
>>>> [root@ovirt-0 ~]# df . Filesystem 1K-blocks Used Available Use% 
>>>> Mounted on /dev/mapper/centos--ovirt--0-root 209608708 107709568 
>>>> 101899140 52% / And there is plenty of space in the filesystem 
>>>> where the backup is being saved: [backup@backup ~]$ df /mnt/backup/ 
>>>> Filesystem 1K-blocks Used Available Use% Mounted on encfs 
>>>> 27143488384 1224301312 25919187072 5% /mnt/backup Is the issue the 
>>>> space in the CWD of the backup server? I've only got 8G free in CWD 
>>>> there: [backup@backup ~]$ df . Filesystem 1K-blocks Used Available 
>>>> Use% Mounted on /dev/mapper/fedora_backup-root 15718400 6388680 
>>>> 9329720 41% / Unfortunately the backtrace does not tell me what 
>>>> file is being written when it runs out of space, so I don't know 
>>>> what to "fix".
>>>> Also,
>>>> due to
>>>> "reasons" I cannot upgrade -- but I am happy to hot-patch if there is a
>>>> simple way to add the target that is failing.
>>>>
>>>> I'd appreciate any guidance.
>>>>
>>>> Thanks,
>>>>
>>>> -derek
>>>>
>>>> Backup Command and output:
>>>>
>>>> *** Starting backup for ovirt-0 at Tue 05 May 2026 06:44:21 AM EDT
>>>> /usr/bin/rdiff-backup-1 --exclude /mnt/snap/dev --exclude
>>>> /mnt/snap/lost+found
>>>> --exclude /mnt/snap/media --exclude /mnt/snap/mnt --exclude
>>>> /mnt/snap/proc
>>>> --exclude
>>>> /mnt/snap/run --exclude /mnt/snap/sys --exclude /mnt/snap//ovirt
>>>> --exclude
>>>> /mnt/snap//rhev --include /mnt/snap//var/lib/ovirt-hosted-engine-setup/
>>>> --include
>>>> /mnt/snap//var/lib/ovirt-hosted-engine-ha/ --include
>>>> /mnt/snap//var/lib/mysql
>>>> --include /mnt/snap//var/lib/vdsm --include /mnt/snap/var/spool/cron
>>>> --include
>>>> /mnt/snap/usr/local --exclude /mnt/snap/bin --exclude /mnt/snap/lib
>>>> --exclude
>>>> /mnt/snap/lib64 --exclude /mnt/snap/sbin --exclude /mnt/snap/tmp
>>>> --exclude
>>>> /mnt/snap/usr --exclude /mnt/snap/var ovirt-0_backup::/mnt/snap/
>>>> /mnt/backup/backups/ovirt-0
>>>> File descriptor 3 (socket:[104543988]) leaked on lvcreate invocation.
>>>> Parent PID
>>>> 13337: /bin/bash
>>>>     Logical volume "snap" created.
>>>> Previous backup seems to have failed, regressing destination now.
>>>> Exception '[Errno 28] No space left on device' raised of class '<type
>>>> 'exceptions.IOError'>':
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/robust.py",
>>>> line
>>>> 32, in
>>>> check_common_error
>>>>       try: return function(*args)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>>>> line
>>>> 468, in
>>>> get_fp
>>>>       Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Rdiff.py",
>>>> line
>>>> 73, in
>>>> write_patched_fp
>>>>       rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp),
>>>> out_fp)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py",
>>>> line
>>>> 64, in
>>>> copyfileobj
>>>>       outputfp.write(inbuf)
>>>>
>>>> Exception '[Errno 28] No space left on device' raised of class '<type
>>>> 'exceptions.IOError'>':
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>>> 304, in
>>>> error_check_Main
>>>>       try: Main(arglist)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>>> 324, in Main
>>>>       take_action(rps)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>>> 280, in
>>>> take_action
>>>>       elif action == "backup": Backup(rps[0], rps[1])
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>>> 337, in Backup
>>>>       backup_final_init(rpout)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>>> 501, in
>>>> backup_final_init
>>>>       checkdest_if_necessary(rpout)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>>> 920, in
>>>> checkdest_if_necessary
>>>>       dest_rp.conn.regress.Regress(dest_rp)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>>>> line
>>>> 71, in
>>>> Regress
>>>>       for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index,
>>>> rf)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/rorpiter.py",
>>>> line
>>>> 281, in
>>>> __call__
>>>>       last_branch.fast_process(*args)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>>>> line
>>>> 268, in
>>>> fast_process
>>>>       if rf.metadata_rorp.isreg(): self.restore_orig_regfile(rf)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>>>> line
>>>> 290, in
>>>> restore_orig_regfile
>>>>       tf.write_from_fileobj(rf.get_restore_fp())
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>>>> line
>>>> 488, in
>>>> get_restore_fp
>>>>       return robust.check_common_error(error_handler, get_fp)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/robust.py",
>>>> line
>>>> 32, in
>>>> check_common_error
>>>>       try: return function(*args)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>>>> line
>>>> 468, in
>>>> get_fp
>>>>       Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Rdiff.py",
>>>> line
>>>> 73, in
>>>> write_patched_fp
>>>>       rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp),
>>>> out_fp)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py",
>>>> line
>>>> 64, in
>>>> copyfileobj
>>>>       outputfp.write(inbuf)
>>>>
>>>> Traceback (most recent call last):
>>>>     File "/usr/bin/rdiff-backup-1", line 30, in <module>
>>>>       rdiff_backup.Main.error_check_Main(sys.argv[1:])
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>>> 304, in
>>>> error_check_Main
>>>>       try: Main(arglist)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>>> 324, in Main
>>>>       take_action(rps)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>>> 280, in
>>>> take_action
>>>>       elif action == "backup": Backup(rps[0], rps[1])
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>>> 337, in Backup
>>>>       backup_final_init(rpout)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>>> 501, in
>>>> backup_final_init
>>>>       checkdest_if_necessary(rpout)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>>> 920, in
>>>> checkdest_if_necessary
>>>>       dest_rp.conn.regress.Regress(dest_rp)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>>>> line
>>>> 71, in
>>>> Regress
>>>>       for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index,
>>>> rf)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/rorpiter.py",
>>>> line
>>>> 281, in
>>>> __call__
>>>>       last_branch.fast_process(*args)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>>>> line
>>>> 268, in
>>>> fast_process
>>>>       if rf.metadata_rorp.isreg(): self.restore_orig_regfile(rf)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>>>> line
>>>> 290, in
>>>> restore_orig_regfile
>>>>       tf.write_from_fileobj(rf.get_restore_fp())
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>>>> line
>>>> 488, in
>>>> get_restore_fp
>>>>       return robust.check_common_error(error_handler, get_fp)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/robust.py",
>>>> line
>>>> 32, in
>>>> check_common_error
>>>>       try: return function(*args)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>>>> line
>>>> 468, in
>>>> get_fp
>>>>       Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/Rdiff.py",
>>>> line
>>>> 73, in
>>>> write_patched_fp
>>>>       rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp),
>>>> out_fp)
>>>>     File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py",
>>>> line
>>>> 64, in
>>>> copyfileobj
>>>>       outputfp.write(inbuf)
>>>> IOError: [Errno 28] No space left on device
>>>> Fatal Error: Lost connection to the remote system
>>>> === Backup for ovirt-0 exited with error code 1 at Tue 05 May 2026
>>>> 06:46:20 AM EDT
>>>>
>>>>
>>>>
>>>>
>>> --
>>> Alerte Vacance: du 23 mai au 2 Juin 2026
>>> Patrik Dufresne
>>> IKUS Software
>>> https://www.ikus-soft.com/
>>> 514-971-6442
>>> St-Colomban, QC J5K 1T9
>>>
>>
>> --
>>         Derek Atkins                 617-623-3745
>>         [email protected]             www.ihtfp.com
>>         Computer and Internet Security Consultant
>>
>>
>>
>
-- 
Alerte Vacance: du 23 mai au 2 Juin 2026
Patrik Dufresne
IKUS Software
https://www.ikus-soft.com/
514-971-6442
St-Colomban, QC J5K 1T9