Re: Filesystem full errors when there is plenty of space? [SOLVED]

Hi,

Just to confirm -- I ran the backup manually yesterday with the --tempdir
option pointing to a space adjacent to the target (so it had lots of free
space) and it succeeded.  I've added that --tempdir option to this
specific run to protect it in the future, and will add it more generally
to my backup scripts if I see this error in the future.

Thank you so much for pointing me in the right direction!

-derek

On Tue, May 5, 2026 10:48 am, Derek Atkins wrote:
> Hi Patrik,
>
> Thank you for the inputs.
> More inline below..
>
> On Tue, May 5, 2026 10:00 am, Patrik Dufresne via Any discussion of
> rdiff-backup wrote:
>> Hello Derek,
>>
>> Hard to tell with the information you provided, but this error might be
>> caused by the following reasons:
>
> Alas, the information I provided is all the information rdiff-backup gave
> me!  I wish it provided more details!
>
>>
>>  1.
>>
>>     *Not enough disk space at destination* (this doesn't seem to be your
>>     case, as you have ~25TB free on /mnt/backup/)
>>
>>  2.
>>
>>     *Not enough inodes at destination*. You can check with:
>>
>>     |df -i /mnt/backup/ |
>>
>>     Note that encfs filesystems can sometimes have inode limitations
>>     inherited from the underlying filesystem, so also check:
>>
>>     |df -i / |
>>
>>     (or whatever filesystem underlies your encfs mount)
>
> I actually looked at the inodes after I sent the email and they all look
> like there is plenty of room:
>
> [backup@backup ~]$ df -i /mnt/backup/
> Filesystem          Inodes   IUsed       IFree IUse% Mounted on
> encfs          51847503204 9129350 51838373854    1% /mnt/backup
>
> [backup@backup ~]$ df -i .
> Filesystem                      Inodes  IUsed   IFree IUse% Mounted on
> /dev/mapper/fedora_backup-root 7864320 102189 7762131    2% /
>
> NB: I did not check the underlying FS, but I am sure it's pretty much what
> encfs shows.
>
>>
>>  3.
>>
>>     *Not enough disk space or inodes in your TEMP folder*. rdiff-backup
>>     may write temporary files to the system's temp directory (typically
>>     |/tmp|) during regression/restore operations. You can check with:
>>
>>     |df /tmp df -i /tmp |
>
> [backup@backup ~]$ df /tmp
> Filesystem     1K-blocks  Used Available Use% Mounted on
> tmpfs            3808324     4   3808320   1% /tmp
> [backup@backup ~]$ df -i /tmp
> Filesystem     Inodes IUsed  IFree IUse% Mounted on
> tmpfs          952081     8 952073    1% /tmp
>
>>
>>     With only ~8GB free on your |/| filesystem (which likely also hosts
>>     |/tmp|), this could very well be the culprit, especially during a
>>     regression pass which involves reconstructing files by applying
>>     rdiff patches.
>>
>> Given your traceback, the error is happening during a *regression*
>> (rolling back a previously failed backup), which involves reconstructing
>> and writing out full file contents as temporary files. This can require
>> significant temporary space depending on the size of the files being
>> reconstructed.
>>
>> My best guess given your numbers is *option 3* — the 8GB free on your
>> backup server's root filesystem may not be sufficient for the temporary
>> files being written during regression.
>
> Interesting observation.  My /tmp is on a tmpfs and only 3.7GB -- so even
> less space free!  Right now none of TMP, TEMP, nor TMPDIR are set to
> anything, so it's using /tmp -- and given your new information (to me)
> about it reconstructing full files, I bet it's running out of space in
> /tmp!
>
> As a test -- I'm re-running with "--tmpdir /mnt/backup/tmp" and we'll see
> if that fixes the problem.
>
> Thanks!
>
> -derek
>
>>
>>
>> On 5/5/26 08:52, Derek Atkins wrote:
>>> Hi
>>>
>>> I've suddenly, recently, starting getting "File System Full" errors
>>> from
>>> the backup of one of my servers. This particular server
>>> is still using 1.2.8. What I don't know is: what filesystem is full?
>>> There is plenty of space on the system being backed up: [root@ovirt-0
>>> ~]# df . Filesystem 1K-blocks Used Available Use% Mounted on
>>> /dev/mapper/centos--ovirt--0-root 209608708 107709568 101899140 52% /
>>> And there is plenty of space in the filesystem where the backup is
>>> being saved: [backup@backup ~]$ df /mnt/backup/ Filesystem 1K-blocks
>>> Used Available Use% Mounted on encfs 27143488384 1224301312
>>> 25919187072 5% /mnt/backup Is the issue the space in the CWD of the
>>> backup server? I've only got 8G free in CWD there: [backup@backup ~]$
>>> df . Filesystem 1K-blocks Used Available Use% Mounted on
>>> /dev/mapper/fedora_backup-root 15718400 6388680 9329720 41% /
>>> Unfortunately the backtrace does not tell me what file is being
>>> written when it runs out of space, so I don't know what to "fix".
>>> Also,
>>> due to
>>> "reasons" I cannot upgrade -- but I am happy to hot-patch if there is a
>>> simple way to add the target that is failing.
>>>
>>> I'd appreciate any guidance.
>>>
>>> Thanks,
>>>
>>> -derek
>>>
>>> Backup Command and output:
>>>
>>> *** Starting backup for ovirt-0 at Tue 05 May 2026 06:44:21 AM EDT
>>> /usr/bin/rdiff-backup-1 --exclude /mnt/snap/dev --exclude
>>> /mnt/snap/lost+found
>>> --exclude /mnt/snap/media --exclude /mnt/snap/mnt --exclude
>>> /mnt/snap/proc
>>> --exclude
>>> /mnt/snap/run --exclude /mnt/snap/sys --exclude /mnt/snap//ovirt
>>> --exclude
>>> /mnt/snap//rhev --include /mnt/snap//var/lib/ovirt-hosted-engine-setup/
>>> --include
>>> /mnt/snap//var/lib/ovirt-hosted-engine-ha/ --include
>>> /mnt/snap//var/lib/mysql
>>> --include /mnt/snap//var/lib/vdsm --include /mnt/snap/var/spool/cron
>>> --include
>>> /mnt/snap/usr/local --exclude /mnt/snap/bin --exclude /mnt/snap/lib
>>> --exclude
>>> /mnt/snap/lib64 --exclude /mnt/snap/sbin --exclude /mnt/snap/tmp
>>> --exclude
>>> /mnt/snap/usr --exclude /mnt/snap/var ovirt-0_backup::/mnt/snap/
>>> /mnt/backup/backups/ovirt-0
>>> File descriptor 3 (socket:[104543988]) leaked on lvcreate invocation.
>>> Parent PID
>>> 13337: /bin/bash
>>>    Logical volume "snap" created.
>>> Previous backup seems to have failed, regressing destination now.
>>> Exception '[Errno 28] No space left on device' raised of class '<type
>>> 'exceptions.IOError'>':
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/robust.py",
>>> line
>>> 32, in
>>> check_common_error
>>>      try: return function(*args)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>>> line
>>> 468, in
>>> get_fp
>>>      Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Rdiff.py",
>>> line
>>> 73, in
>>> write_patched_fp
>>>      rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp),
>>> out_fp)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py",
>>> line
>>> 64, in
>>> copyfileobj
>>>      outputfp.write(inbuf)
>>>
>>> Exception '[Errno 28] No space left on device' raised of class '<type
>>> 'exceptions.IOError'>':
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>> 304, in
>>> error_check_Main
>>>      try: Main(arglist)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>> 324, in Main
>>>      take_action(rps)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>> 280, in
>>> take_action
>>>      elif action == "backup": Backup(rps[0], rps[1])
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>> 337, in Backup
>>>      backup_final_init(rpout)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>> 501, in
>>> backup_final_init
>>>      checkdest_if_necessary(rpout)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>> 920, in
>>> checkdest_if_necessary
>>>      dest_rp.conn.regress.Regress(dest_rp)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>>> line
>>> 71, in
>>> Regress
>>>      for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index,
>>> rf)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/rorpiter.py",
>>> line
>>> 281, in
>>> __call__
>>>      last_branch.fast_process(*args)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>>> line
>>> 268, in
>>> fast_process
>>>      if rf.metadata_rorp.isreg(): self.restore_orig_regfile(rf)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>>> line
>>> 290, in
>>> restore_orig_regfile
>>>      tf.write_from_fileobj(rf.get_restore_fp())
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>>> line
>>> 488, in
>>> get_restore_fp
>>>      return robust.check_common_error(error_handler, get_fp)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/robust.py",
>>> line
>>> 32, in
>>> check_common_error
>>>      try: return function(*args)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>>> line
>>> 468, in
>>> get_fp
>>>      Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Rdiff.py",
>>> line
>>> 73, in
>>> write_patched_fp
>>>      rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp),
>>> out_fp)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py",
>>> line
>>> 64, in
>>> copyfileobj
>>>      outputfp.write(inbuf)
>>>
>>> Traceback (most recent call last):
>>>    File "/usr/bin/rdiff-backup-1", line 30, in <module>
>>>      rdiff_backup.Main.error_check_Main(sys.argv[1:])
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>> 304, in
>>> error_check_Main
>>>      try: Main(arglist)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>> 324, in Main
>>>      take_action(rps)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>> 280, in
>>> take_action
>>>      elif action == "backup": Backup(rps[0], rps[1])
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>> 337, in Backup
>>>      backup_final_init(rpout)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>> 501, in
>>> backup_final_init
>>>      checkdest_if_necessary(rpout)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>>> 920, in
>>> checkdest_if_necessary
>>>      dest_rp.conn.regress.Regress(dest_rp)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>>> line
>>> 71, in
>>> Regress
>>>      for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index,
>>> rf)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/rorpiter.py",
>>> line
>>> 281, in
>>> __call__
>>>      last_branch.fast_process(*args)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>>> line
>>> 268, in
>>> fast_process
>>>      if rf.metadata_rorp.isreg(): self.restore_orig_regfile(rf)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>>> line
>>> 290, in
>>> restore_orig_regfile
>>>      tf.write_from_fileobj(rf.get_restore_fp())
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>>> line
>>> 488, in
>>> get_restore_fp
>>>      return robust.check_common_error(error_handler, get_fp)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/robust.py",
>>> line
>>> 32, in
>>> check_common_error
>>>      try: return function(*args)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>>> line
>>> 468, in
>>> get_fp
>>>      Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Rdiff.py",
>>> line
>>> 73, in
>>> write_patched_fp
>>>      rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp),
>>> out_fp)
>>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py",
>>> line
>>> 64, in
>>> copyfileobj
>>>      outputfp.write(inbuf)
>>> IOError: [Errno 28] No space left on device
>>> Fatal Error: Lost connection to the remote system
>>> === Backup for ovirt-0 exited with error code 1 at Tue 05 May 2026
>>> 06:46:20 AM EDT
>>>
>>>
>>>
>>>
>> --
>> Alerte Vacance: du 23 mai au 2 Juin 2026
>> Patrik Dufresne
>> IKUS Software
>> https://www.ikus-soft.com/
>> 514-971-6442
>> St-Colomban, QC J5K 1T9
>>
>
>
> --
>        Derek Atkins                 617-623-3745
>        [email protected]             www.ihtfp.com
>        Computer and Internet Security Consultant
>
>
>


-- 
       Derek Atkins                 617-623-3745
       [email protected]             www.ihtfp.com
       Computer and Internet Security Consultant