Re: Filesystem full errors when there is plenty of space? [SOLVED]

It's great the you figured it out, but isn't the real issue that rdiff-b=
ackup should produce more information for these kinds of filesystem erro=
rs to let the user know what filesystem or filesystem path the error occ=
urred on?

     Peter

On Wed, May 6, 2026, at 06:13, Derek Atkins wrote:
> Hi,
>=20
> Just to confirm -- I ran the backup manually yesterday with the --temp=
dir
> option pointing to a space adjacent to the target (so it had lots of f=
ree
> space) and it succeeded.  I've added that --tempdir option to this
> specific run to protect it in the future, and will add it more general=
ly
> to my backup scripts if I see this error in the future.
>=20
> Thank you so much for pointing me in the right direction!
>=20
> -derek
>=20
> On Tue, May 5, 2026 10:48 am, Derek Atkins wrote:
> > Hi Patrik,
> >
> > Thank you for the inputs.
> > More inline below..
> >
> > On Tue, May 5, 2026 10:00 am, Patrik Dufresne via Any discussion of
> > rdiff-backup wrote:
> >> Hello Derek,
> >>
> >> Hard to tell with the information you provided, but this error migh=
t be
> >> caused by the following reasons:
> >
> > Alas, the information I provided is all the information rdiff-backup=
 gave
> > me!  I wish it provided more details!
> >
> >>
> >>  1.
> >>
> >>     *Not enough disk space at destination* (this doesn't seem to be=
 your
> >>     case, as you have ~25TB free on /mnt/backup/)
> >>
> >>  2.
> >>
> >>     *Not enough inodes at destination*. You can check with:
> >>
> >>     |df -i /mnt/backup/ |
> >>
> >>     Note that encfs filesystems can sometimes have inode limitations
> >>     inherited from the underlying filesystem, so also check:
> >>
> >>     |df -i / |
> >>
> >>     (or whatever filesystem underlies your encfs mount)
> >
> > I actually looked at the inodes after I sent the email and they all =
look
> > like there is plenty of room:
> >
> > [backup@backup ~]$ df -i /mnt/backup/
> > Filesystem          Inodes   IUsed       IFree IUse% Mounted on
> > encfs          51847503204 9129350 51838373854    1% /mnt/backup
> >
> > [backup@backup ~]$ df -i .
> > Filesystem                      Inodes  IUsed   IFree IUse% Mounted =
on
> > /dev/mapper/fedora_backup-root 7864320 102189 7762131    2% /
> >
> > NB: I did not check the underlying FS, but I am sure it's pretty muc=
h what
> > encfs shows.
> >
> >>
> >>  3.
> >>
> >>     *Not enough disk space or inodes in your TEMP folder*. rdiff-ba=
ckup
> >>     may write temporary files to the system's temp directory (typic=
ally
> >>     |/tmp|) during regression/restore operations. You can check wit=
h:
> >>
> >>     |df /tmp df -i /tmp |
> >
> > [backup@backup ~]$ df /tmp
> > Filesystem     1K-blocks  Used Available Use% Mounted on
> > tmpfs            3808324     4   3808320   1% /tmp
> > [backup@backup ~]$ df -i /tmp
> > Filesystem     Inodes IUsed  IFree IUse% Mounted on
> > tmpfs          952081     8 952073    1% /tmp
> >
> >>
> >>     With only ~8GB free on your |/| filesystem (which likely also h=
osts
> >>     |/tmp|), this could very well be the culprit, especially during=
 a
> >>     regression pass which involves reconstructing files by applying
> >>     rdiff patches.
> >>
> >> Given your traceback, the error is happening during a *regression*
> >> (rolling back a previously failed backup), which involves reconstru=
cting
> >> and writing out full file contents as temporary files. This can req=
uire
> >> significant temporary space depending on the size of the files being
> >> reconstructed.
> >>
> >> My best guess given your numbers is *option 3* =E2=80=94 the 8GB fr=
ee on your
> >> backup server's root filesystem may not be sufficient for the tempo=
rary
> >> files being written during regression.
> >
> > Interesting observation.  My /tmp is on a tmpfs and only 3.7GB -- so=
 even
> > less space free!  Right now none of TMP, TEMP, nor TMPDIR are set to
> > anything, so it's using /tmp -- and given your new information (to m=
e)
> > about it reconstructing full files, I bet it's running out of space =
in
> > /tmp!
> >
> > As a test -- I'm re-running with "--tmpdir /mnt/backup/tmp" and we'l=
l see
> > if that fixes the problem.
> >
> > Thanks!
> >
> > -derek
> >
> >>
> >>
> >> On 5/5/26 08:52, Derek Atkins wrote:
> >>> Hi
> >>>
> >>> I've suddenly, recently, starting getting "File System Full" errors
> >>> from
> >>> the backup of one of my servers. This particular server
> >>> is still using 1.2.8. What I don't know is: what filesystem is ful=
l?
> >>> There is plenty of space on the system being backed up: [root@ovir=
t-0
> >>> ~]# df . Filesystem 1K-blocks Used Available Use% Mounted on
> >>> /dev/mapper/centos--ovirt--0-root 209608708 107709568 101899140 52=
% /
> >>> And there is plenty of space in the filesystem where the backup is
> >>> being saved: [backup@backup ~]$ df /mnt/backup/ Filesystem 1K-bloc=
ks
> >>> Used Available Use% Mounted on encfs 27143488384 1224301312
> >>> 25919187072 5% /mnt/backup Is the issue the space in the CWD of the
> >>> backup server? I've only got 8G free in CWD there: [backup@backup =
~]$
> >>> df . Filesystem 1K-blocks Used Available Use% Mounted on
> >>> /dev/mapper/fedora_backup-root 15718400 6388680 9329720 41% /
> >>> Unfortunately the backtrace does not tell me what file is being
> >>> written when it runs out of space, so I don't know what to "fix".
> >>> Also,
> >>> due to
> >>> "reasons" I cannot upgrade -- but I am happy to hot-patch if there=
 is a
> >>> simple way to add the target that is failing.
> >>>
> >>> I'd appreciate any guidance.
> >>>
> >>> Thanks,
> >>>
> >>> -derek
> >>>
> >>> Backup Command and output:
> >>>
> >>> *** Starting backup for ovirt-0 at Tue 05 May 2026 06:44:21 AM EDT
> >>> /usr/bin/rdiff-backup-1 --exclude /mnt/snap/dev --exclude
> >>> /mnt/snap/lost+found
> >>> --exclude /mnt/snap/media --exclude /mnt/snap/mnt --exclude
> >>> /mnt/snap/proc
> >>> --exclude
> >>> /mnt/snap/run --exclude /mnt/snap/sys --exclude /mnt/snap//ovirt
> >>> --exclude
> >>> /mnt/snap//rhev --include /mnt/snap//var/lib/ovirt-hosted-engine-s=
etup/
> >>> --include
> >>> /mnt/snap//var/lib/ovirt-hosted-engine-ha/ --include
> >>> /mnt/snap//var/lib/mysql
> >>> --include /mnt/snap//var/lib/vdsm --include /mnt/snap/var/spool/cr=
on
> >>> --include
> >>> /mnt/snap/usr/local --exclude /mnt/snap/bin --exclude /mnt/snap/lib
> >>> --exclude
> >>> /mnt/snap/lib64 --exclude /mnt/snap/sbin --exclude /mnt/snap/tmp
> >>> --exclude
> >>> /mnt/snap/usr --exclude /mnt/snap/var ovirt-0_backup::/mnt/snap/
> >>> /mnt/backup/backups/ovirt-0
> >>> File descriptor 3 (socket:[104543988]) leaked on lvcreate invocati=
on.
> >>> Parent PID
> >>> 13337: /bin/bash
> >>>    Logical volume "snap" created.
> >>> Previous backup seems to have failed, regressing destination now.
> >>> Exception '[Errno 28] No space left on device' raised of class '<t=
ype
> >>> 'exceptions.IOError'>':
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/robust.py=
",
> >>> line
> >>> 32, in
> >>> check_common_error
> >>>      try: return function(*args)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.p=
y",
> >>> line
> >>> 468, in
> >>> get_fp
> >>>      Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Rdiff.py",
> >>> line
> >>> 73, in
> >>> write_patched_fp
> >>>      rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp),
> >>> out_fp)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py",
> >>> line
> >>> 64, in
> >>> copyfileobj
> >>>      outputfp.write(inbuf)
> >>>
> >>> Exception '[Errno 28] No space left on device' raised of class '<t=
ype
> >>> 'exceptions.IOError'>':
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py",=
 line
> >>> 304, in
> >>> error_check_Main
> >>>      try: Main(arglist)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py",=
 line
> >>> 324, in Main
> >>>      take_action(rps)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py",=
 line
> >>> 280, in
> >>> take_action
> >>>      elif action =3D=3D "backup": Backup(rps[0], rps[1])
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py",=
 line
> >>> 337, in Backup
> >>>      backup_final_init(rpout)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py",=
 line
> >>> 501, in
> >>> backup_final_init
> >>>      checkdest_if_necessary(rpout)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py",=
 line
> >>> 920, in
> >>> checkdest_if_necessary
> >>>      dest_rp.conn.regress.Regress(dest_rp)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.p=
y",
> >>> line
> >>> 71, in
> >>> Regress
> >>>      for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.inde=
x,
> >>> rf)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/rorpiter.=
py",
> >>> line
> >>> 281, in
> >>> __call__
> >>>      last_branch.fast_process(*args)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.p=
y",
> >>> line
> >>> 268, in
> >>> fast_process
> >>>      if rf.metadata_rorp.isreg(): self.restore_orig_regfile(rf)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.p=
y",
> >>> line
> >>> 290, in
> >>> restore_orig_regfile
> >>>      tf.write_from_fileobj(rf.get_restore_fp())
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.p=
y",
> >>> line
> >>> 488, in
> >>> get_restore_fp
> >>>      return robust.check_common_error(error_handler, get_fp)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/robust.py=
",
> >>> line
> >>> 32, in
> >>> check_common_error
> >>>      try: return function(*args)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.p=
y",
> >>> line
> >>> 468, in
> >>> get_fp
> >>>      Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Rdiff.py",
> >>> line
> >>> 73, in
> >>> write_patched_fp
> >>>      rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp),
> >>> out_fp)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py",
> >>> line
> >>> 64, in
> >>> copyfileobj
> >>>      outputfp.write(inbuf)
> >>>
> >>> Traceback (most recent call last):
> >>>    File "/usr/bin/rdiff-backup-1", line 30, in <module>
> >>>      rdiff_backup.Main.error_check_Main(sys.argv[1:])
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py",=
 line
> >>> 304, in
> >>> error_check_Main
> >>>      try: Main(arglist)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py",=
 line
> >>> 324, in Main
> >>>      take_action(rps)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py",=
 line
> >>> 280, in
> >>> take_action
> >>>      elif action =3D=3D "backup": Backup(rps[0], rps[1])
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py",=
 line
> >>> 337, in Backup
> >>>      backup_final_init(rpout)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py",=
 line
> >>> 501, in
> >>> backup_final_init
> >>>      checkdest_if_necessary(rpout)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py",=
 line
> >>> 920, in
> >>> checkdest_if_necessary
> >>>      dest_rp.conn.regress.Regress(dest_rp)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.p=
y",
> >>> line
> >>> 71, in
> >>> Regress
> >>>      for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.inde=
x,
> >>> rf)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/rorpiter.=
py",
> >>> line
> >>> 281, in
> >>> __call__
> >>>      last_branch.fast_process(*args)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.p=
y",
> >>> line
> >>> 268, in
> >>> fast_process
> >>>      if rf.metadata_rorp.isreg(): self.restore_orig_regfile(rf)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/regress.p=
y",
> >>> line
> >>> 290, in
> >>> restore_orig_regfile
> >>>      tf.write_from_fileobj(rf.get_restore_fp())
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.p=
y",
> >>> line
> >>> 488, in
> >>> get_restore_fp
> >>>      return robust.check_common_error(error_handler, get_fp)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/robust.py=
",
> >>> line
> >>> 32, in
> >>> check_common_error
> >>>      try: return function(*args)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/restore.p=
y",
> >>> line
> >>> 468, in
> >>> get_fp
> >>>      Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/Rdiff.py",
> >>> line
> >>> 73, in
> >>> write_patched_fp
> >>>      rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp),
> >>> out_fp)
> >>>    File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py",
> >>> line
> >>> 64, in
> >>> copyfileobj
> >>>      outputfp.write(inbuf)
> >>> IOError: [Errno 28] No space left on device
> >>> Fatal Error: Lost connection to the remote system
> >>> =3D=3D=3D Backup for ovirt-0 exited with error code 1 at Tue 05 Ma=
y 2026
> >>> 06:46:20 AM EDT
> >>>
> >>>
> >>>
> >>>
> >> --
> >> Alerte Vacance: du 23 mai au 2 Juin 2026
> >> Patrik Dufresne
> >> IKUS Software
> >> https://www.ikus-soft.com/
> >> 514-971-6442
> >> St-Colomban, QC J5K 1T9
> >>
> >
> >
> > --
> >        Derek Atkins                 617-623-3745
> >        [email protected]             www.ihtfp.com
> >        Computer and Internet Security Consultant
> >
> >
> >
>=20
>=20
> --=20
>        Derek Atkins                 617-623-3745
>        [email protected]             www.ihtfp.com
>        Computer and Internet Security Consultant
>=20
>=20
>=20