Re: Filesystem full errors when there is plenty of space?
"Derek Atkins" <[email protected]> Tue, 5 May 2026 10:48:23 -0400
| Newsgroups | gmane.comp.sysutils.backup.rdiff-backup.general |
|---|---|
| Message-ID | <[email protected]> |
Hi Patrik,
Thank you for the inputs.
More inline below..
On Tue, May 5, 2026 10:00 am, Patrik Dufresne via Any discussion of
rdiff-backup wrote:
> Hello Derek,
>
> Hard to tell with the information you provided, but this error might be
> caused by the following reasons:
Alas, the information I provided is all the information rdiff-backup gave
me! I wish it provided more details!
>
> 1.
>
> *Not enough disk space at destination* (this doesn't seem to be your
> case, as you have ~25TB free on /mnt/backup/)
>
> 2.
>
> *Not enough inodes at destination*. You can check with:
>
> |df -i /mnt/backup/ |
>
> Note that encfs filesystems can sometimes have inode limitations
> inherited from the underlying filesystem, so also check:
>
> |df -i / |
>
> (or whatever filesystem underlies your encfs mount)
I actually looked at the inodes after I sent the email and they all look
like there is plenty of room:
[backup@backup ~]$ df -i /mnt/backup/
Filesystem Inodes IUsed IFree IUse% Mounted on
encfs 51847503204 9129350 51838373854 1% /mnt/backup
[backup@backup ~]$ df -i .
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/fedora_backup-root 7864320 102189 7762131 2% /
NB: I did not check the underlying FS, but I am sure it's pretty much what
encfs shows.
>
> 3.
>
> *Not enough disk space or inodes in your TEMP folder*. rdiff-backup
> may write temporary files to the system's temp directory (typically
> |/tmp|) during regression/restore operations. You can check with:
>
> |df /tmp df -i /tmp |
[backup@backup ~]$ df /tmp
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 3808324 4 3808320 1% /tmp
[backup@backup ~]$ df -i /tmp
Filesystem Inodes IUsed IFree IUse% Mounted on
tmpfs 952081 8 952073 1% /tmp
>
> With only ~8GB free on your |/| filesystem (which likely also hosts
> |/tmp|), this could very well be the culprit, especially during a
> regression pass which involves reconstructing files by applying
> rdiff patches.
>
> Given your traceback, the error is happening during a *regression*
> (rolling back a previously failed backup), which involves reconstructing
> and writing out full file contents as temporary files. This can require
> significant temporary space depending on the size of the files being
> reconstructed.
>
> My best guess given your numbers is *option 3* — the 8GB free on your
> backup server's root filesystem may not be sufficient for the temporary
> files being written during regression.
Interesting observation. My /tmp is on a tmpfs and only 3.7GB -- so even
less space free! Right now none of TMP, TEMP, nor TMPDIR are set to
anything, so it's using /tmp -- and given your new information (to me)
about it reconstructing full files, I bet it's running out of space in
/tmp!
As a test -- I'm re-running with "--tmpdir /mnt/backup/tmp" and we'll see
if that fixes the problem.
Thanks!
-derek
>
>
> On 5/5/26 08:52, Derek Atkins wrote:
>> Hi
>>
>> I've suddenly, recently, starting getting "File System Full" errors from
>> the backup of one of my servers. This particular server
>> is still using 1.2.8. What I don't know is: what filesystem is full?
>> There is plenty of space on the system being backed up: [root@ovirt-0
>> ~]# df . Filesystem 1K-blocks Used Available Use% Mounted on
>> /dev/mapper/centos--ovirt--0-root 209608708 107709568 101899140 52% /
>> And there is plenty of space in the filesystem where the backup is
>> being saved: [backup@backup ~]$ df /mnt/backup/ Filesystem 1K-blocks
>> Used Available Use% Mounted on encfs 27143488384 1224301312
>> 25919187072 5% /mnt/backup Is the issue the space in the CWD of the
>> backup server? I've only got 8G free in CWD there: [backup@backup ~]$
>> df . Filesystem 1K-blocks Used Available Use% Mounted on
>> /dev/mapper/fedora_backup-root 15718400 6388680 9329720 41% /
>> Unfortunately the backtrace does not tell me what file is being
>> written when it runs out of space, so I don't know what to "fix". Also,
>> due to
>> "reasons" I cannot upgrade -- but I am happy to hot-patch if there is a
>> simple way to add the target that is failing.
>>
>> I'd appreciate any guidance.
>>
>> Thanks,
>>
>> -derek
>>
>> Backup Command and output:
>>
>> *** Starting backup for ovirt-0 at Tue 05 May 2026 06:44:21 AM EDT
>> /usr/bin/rdiff-backup-1 --exclude /mnt/snap/dev --exclude
>> /mnt/snap/lost+found
>> --exclude /mnt/snap/media --exclude /mnt/snap/mnt --exclude
>> /mnt/snap/proc
>> --exclude
>> /mnt/snap/run --exclude /mnt/snap/sys --exclude /mnt/snap//ovirt
>> --exclude
>> /mnt/snap//rhev --include /mnt/snap//var/lib/ovirt-hosted-engine-setup/
>> --include
>> /mnt/snap//var/lib/ovirt-hosted-engine-ha/ --include
>> /mnt/snap//var/lib/mysql
>> --include /mnt/snap//var/lib/vdsm --include /mnt/snap/var/spool/cron
>> --include
>> /mnt/snap/usr/local --exclude /mnt/snap/bin --exclude /mnt/snap/lib
>> --exclude
>> /mnt/snap/lib64 --exclude /mnt/snap/sbin --exclude /mnt/snap/tmp
>> --exclude
>> /mnt/snap/usr --exclude /mnt/snap/var ovirt-0_backup::/mnt/snap/
>> /mnt/backup/backups/ovirt-0
>> File descriptor 3 (socket:[104543988]) leaked on lvcreate invocation.
>> Parent PID
>> 13337: /bin/bash
>> Logical volume "snap" created.
>> Previous backup seems to have failed, regressing destination now.
>> Exception '[Errno 28] No space left on device' raised of class '<type
>> 'exceptions.IOError'>':
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/robust.py",
>> line
>> 32, in
>> check_common_error
>> try: return function(*args)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>> line
>> 468, in
>> get_fp
>> Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Rdiff.py", line
>> 73, in
>> write_patched_fp
>> rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp), out_fp)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py", line
>> 64, in
>> copyfileobj
>> outputfp.write(inbuf)
>>
>> Exception '[Errno 28] No space left on device' raised of class '<type
>> 'exceptions.IOError'>':
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>> 304, in
>> error_check_Main
>> try: Main(arglist)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>> 324, in Main
>> take_action(rps)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>> 280, in
>> take_action
>> elif action == "backup": Backup(rps[0], rps[1])
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>> 337, in Backup
>> backup_final_init(rpout)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>> 501, in
>> backup_final_init
>> checkdest_if_necessary(rpout)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>> 920, in
>> checkdest_if_necessary
>> dest_rp.conn.regress.Regress(dest_rp)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>> line
>> 71, in
>> Regress
>> for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/rorpiter.py",
>> line
>> 281, in
>> __call__
>> last_branch.fast_process(*args)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>> line
>> 268, in
>> fast_process
>> if rf.metadata_rorp.isreg(): self.restore_orig_regfile(rf)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>> line
>> 290, in
>> restore_orig_regfile
>> tf.write_from_fileobj(rf.get_restore_fp())
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>> line
>> 488, in
>> get_restore_fp
>> return robust.check_common_error(error_handler, get_fp)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/robust.py",
>> line
>> 32, in
>> check_common_error
>> try: return function(*args)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>> line
>> 468, in
>> get_fp
>> Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Rdiff.py", line
>> 73, in
>> write_patched_fp
>> rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp), out_fp)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py", line
>> 64, in
>> copyfileobj
>> outputfp.write(inbuf)
>>
>> Traceback (most recent call last):
>> File "/usr/bin/rdiff-backup-1", line 30, in <module>
>> rdiff_backup.Main.error_check_Main(sys.argv[1:])
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>> 304, in
>> error_check_Main
>> try: Main(arglist)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>> 324, in Main
>> take_action(rps)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>> 280, in
>> take_action
>> elif action == "backup": Backup(rps[0], rps[1])
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>> 337, in Backup
>> backup_final_init(rpout)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>> 501, in
>> backup_final_init
>> checkdest_if_necessary(rpout)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
>> 920, in
>> checkdest_if_necessary
>> dest_rp.conn.regress.Regress(dest_rp)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>> line
>> 71, in
>> Regress
>> for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/rorpiter.py",
>> line
>> 281, in
>> __call__
>> last_branch.fast_process(*args)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>> line
>> 268, in
>> fast_process
>> if rf.metadata_rorp.isreg(): self.restore_orig_regfile(rf)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.py",
>> line
>> 290, in
>> restore_orig_regfile
>> tf.write_from_fileobj(rf.get_restore_fp())
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>> line
>> 488, in
>> get_restore_fp
>> return robust.check_common_error(error_handler, get_fp)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/robust.py",
>> line
>> 32, in
>> check_common_error
>> try: return function(*args)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.py",
>> line
>> 468, in
>> get_fp
>> Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/Rdiff.py", line
>> 73, in
>> write_patched_fp
>> rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp), out_fp)
>> File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py", line
>> 64, in
>> copyfileobj
>> outputfp.write(inbuf)
>> IOError: [Errno 28] No space left on device
>> Fatal Error: Lost connection to the remote system
>> === Backup for ovirt-0 exited with error code 1 at Tue 05 May 2026
>> 06:46:20 AM EDT
>>
>>
>>
>>
> --
> Alerte Vacance: du 23 mai au 2 Juin 2026
> Patrik Dufresne
> IKUS Software
> https://www.ikus-soft.com/
> 514-971-6442
> St-Colomban, QC J5K 1T9
>
--
Derek Atkins 617-623-3745
[email protected] www.ihtfp.com
Computer and Internet Security Consultant