Re: Could this be SVN shard directory logic bug?
William Muriithi <[email protected]> Mon, 25 May 2026 13:13:03 -0400
| Newsgroups | gmane.comp.version-control.subversion.user |
|---|---|
| Message-ID | <CAE9rU+61GL-9q+tts4FiP2Qk+3+kOq9eCRDWTv4=JsLiTZU9Ww@mail.gmail.com> |
--0000000000006afa660652a77e8a Content-Type: text/plain; charset="UTF-8" Hi guys and girls Something I missed to mention. Originally, the file contents looked like this:- [root@repos ~]# cat /var/repos/svn/projects/db/current 269999 [root@repos ~]# cat /var/repos/svn/projects/db/txn-current 5sbz These files seem to be important on how SVN makes the decision of which directory to dump/look for a file. So I looked around and google result suggested that the last file may be incorrect. So I changed it to:- [root@repos ~]# cat /var/repos/svn/projects/db/txn-current 5r3j MOVE forward: I then created the files strace point as SVN access points. All below commands were executed as user apache mkdir /var/repos/svn/projects/db/revs/270 cd /var/repos/svn/projects/db/revs/270 touch 270000 270016 270080 270336 svnadmin recover /var/repos/svn/projects/ When I tried committing a new, I got a new error:- svnadmin: E160004: Revision 270336 has a revs file but no revprops file MOVE backward: [root@repos ~]# cat /var/repos/svn/projects/db/current 269999 [root@repos ~]# cat /var/repos/svn/projects/db/txn-current 5r3j svnadmin recover /var/repos/svn/projects/ (As apache user) - It run successfully. Then we went back to SVN client and got the old error Basically, we can't seem to move it back or forward. Regards, William On Mon, 25 May 2026 at 12:24, William Muriithi <[email protected]> wrote: > Hi guys, > > [root@repos ~]# rpm -qa | grep subversion > subversion-libs-1.14.5-3.el10.x86_64_v2 > subversion-1.14.5-3.el10.x86_64_v2 > subversion-tools-1.14.5-3.el10.x86_64_v2 > [root@repos ~]# > [root@repos ~]# > [root@repos ~]# cat /etc/redhat-release > AlmaLinux release 10.1 (Heliotrope Lion) > [root@repos ~]# > [root@repos ~]# rpm -qa | grep svn > mod_dav_svn-1.14.5-3.el10.x86_64_v2 > > > I think I have hit a bug. We have a relatively large SVN repository - > About 2TB uncompressed. 2 weeks ago, someone noticed that we can't commit > any more. When we looked at it, it looked like a permission issue. > > This is the error we are getting from the SVN client. > > svn: E000001: Commit failed (details follow): > svn: E000001: Can't set permissions on > '/var/repos/svn/application/db/revs/270' > william@william-ryzen:~/Documents/application$ [email protected] > > So we first attempted to "FIX" permissions. Didn't work. Then, when > reading, I noticed opening up the permission is not a good idea as I can > see from strace that svn does try to narrow down the file permissions. So > we decided to restore a fresh to remove all the chmod we had done to the > files and directory. > > This time, when restoring, we created the repo using an apache account. > We also loaded the dump file using the svn account. This removed all > possibility this is a file permission issue. The repo was restored > successfully, BUT the problem remained. > > We looked at the file limit too and to make sure that isn't causing any > issues, we changed it to unlimited for the apache user. The problem > remained. > > It's at this point we attempted to use strace to see what svn is doing. > Oddly, it seemed to be looking for files from directory 270, which didn't > exist at the time. We came to the conclusion that was what was happening > because these came up when one attempts to commit. > > openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270336", > O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270336", > O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270080", > O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270080", > O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270016", > O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270016", > O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270000", > O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270000", > O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) > > Note, the directory 270 didn't exist. According to google, this can > happen due to either a hook or CI committing too often creating > corruption. We can't seem to clear the corruption though. We are also a > bit skeptical of its corruption as it should have cleared on a freshly > restored repository. > > > So over the weekend, we attempted to separate the projects on the repo to > individual repos. This is to reduce the size of the repo as it could have > been causing the issue. We however kept the revision numbers as they are > in documentations and tags. However, on the now small repo, with only 187 > GB in size, but with the last successful commit # 269999, we still see > the problem. > > This now looks more like a software bug than a setup issue. We have "yum > updated" the system in an attempt to make sure we have all the bug fixes > out there but nothing works. > > Would anyone know if there is a transaction limit in the current SVN > code? Anyone have a repository with more than 270000 revisions out there > please? Any pointers to something we may have overlooked so far? > Something else, the svndump is cleanly at 2 TB, any chance there may be a > limit of how big a repo glow?? This was our first thought but since we > have created smaller repos and the issue remains, it now looks far more > like a sharding logic error to us. > > > Regards, > William > > > > > > > > > > --0000000000006afa660652a77e8a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Hi guys and girls</div><div><br></div><div><br></div>= <div>Something I missed to mention. Originally, the file contents looked li= ke this:-</div><div><br></div><div>[root@repos ~]# cat /var/repos/svn/proje= cts/db/current<br>269999<br>[root@repos ~]# cat /var/repos/svn/projects/db/= txn-current<br>5sbz</div><div><br></div><div>These files seem to be importa= nt on how SVN makes the decision of which directory to dump/look for a file= .=C2=A0 =C2=A0So I looked around=C2=A0 and google result suggested that the= last file may be incorrect.</div><div><br></div><div>So I changed it to:-<= /div><div><br></div><div>[root@repos ~]# cat /var/repos/svn/projects/db/txn= -current<br>5r3j</div><div><br></div><div>MOVE forward:</div><div>I then cr= eated the files strace point as SVN access points.=C2=A0 All below commands= were executed as user apache</div><div><br></div><div>mkdir /var/repos/svn= /projects/db/revs/270</div><div>cd /var/repos/svn/projects/db/revs/270</div= ><div>touch 270000 270016 270080 270336</div><div>svnadmin recover /var/rep= os/svn/projects/</div><div><br></div><div>When I tried committing a new, I = got a new error:-</div><div><br></div><div>svnadmin: E160004: Revision 2703= 36 has a revs file but no revprops file</div><div><br></div><div>MOVE backw= ard:</div><div>[root@repos ~]# cat /var/repos/svn/projects/db/current<br>26= 9999</div><div>[root@repos ~]# cat /var/repos/svn/projects/db/txn-current<b= r>5r3j</div><div><br></div><div>svnadmin recover /var/repos/svn/projects/ (= As apache user) - It run successfully.=C2=A0</div><div><br></div><div>Then = we went back to SVN client and got the old error</div><div><br></div><div>B= asically, we can't seem to move it back or forward.</div><div><br></div= ><div>Regards,</div><div>William=C2=A0</div><div><br></div><div><br></div><= /div><br><div class=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr" = class=3D"gmail_attr">On Mon, 25 May 2026 at 12:24, William Muriithi <<a = href=3D"mailto:[email protected]">[email protected]</a>&g= t; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0p= x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div d= ir=3D"ltr"><div>Hi guys,</div><div><br></div><div>[root@repos ~]# rpm -qa |= grep subversion<br>subversion-libs-1.14.5-3.el10.x86_64_v2<br>subversion-1= .14.5-3.el10.x86_64_v2<br>subversion-tools-1.14.5-3.el10.x86_64_v2<br>[root= @repos ~]# <br>[root@repos ~]# <br>[root@repos ~]# cat /etc/redhat-release = <br>AlmaLinux release 10.1 (Heliotrope Lion)<br>[root@repos ~]#</div><div>[= root@repos ~]# rpm -qa | grep svn<br>mod_dav_svn-1.14.5-3.el10.x86_64_v2</d= iv><div><br></div><div><br></div><div>I think I have hit a bug.=C2=A0 We ha= ve a relatively large SVN repository - About 2TB uncompressed.=C2=A0 2 week= s ago, someone noticed that we can't commit any more.=C2=A0 When we loo= ked at it, it looked like a permission issue.</div><div><br></div><div>This= is the error we are getting from the SVN client.</div><div><br>svn: E00000= 1: Commit failed (details follow):<br>svn: E000001: Can't set permissio= ns on '/var/repos/svn/application/db/revs/270'<br>william@william-r= yzen:~/Documents/application$ <a href=3D"mailto:[email protected]= " target=3D"_blank">[email protected]</a><br></div><div><br></div= ><div>So we first attempted to "FIX" permissions.=C2=A0 Didn'= t work.=C2=A0 Then, when reading, I noticed opening up the permission is no= t a good idea as I can see from strace that svn does try to narrow down the= file permissions.=C2=A0 So we decided to restore a fresh to remove all the= chmod we had done to the files and directory.</div><div><br></div><div>Thi= s time, when restoring, we created the repo using an apache account.=C2=A0 = We also loaded the dump file using the svn account.=C2=A0 This removed all = possibility=C2=A0this is a file permission issue.=C2=A0 The repo was restor= ed successfully, BUT the problem remained.</div><div><br></div><div>We look= ed at the file limit too and to make sure that isn't causing any issues= , we changed it to unlimited for the apache user.=C2=A0 The problem remaine= d.</div><div><br></div><div>It's at this point we attempted to use stra= ce to see what svn is doing.=C2=A0 Oddly, it seemed to be looking for files= from directory 270, which didn't exist at the time.=C2=A0 We came to t= he conclusion that was what was happening because these came up when one at= tempts to commit.</div><div><br></div><div><span><span><span>openat(AT_FDCW= D, "/var/repos/svn/projects/db/revs/270/270336", O_RDONLY|O_CLOEX= EC) =3D -1 ENOENT (No such file or directory)<br>openat(AT_FDCWD, "/va= r/repos/svn/projects/db/revs/270/270336", O_RDONLY|O_CLOEXEC) =3D -1 E= NOENT (No such file or directory)<br>openat(AT_FDCWD, "/var/repos/svn/= projects/db/revs/270/270080", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No su= ch file or directory)<br>openat(AT_FDCWD, "/var/repos/svn/projects/db/= revs/270/270080", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or d= irectory)<br>openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270= 016", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory)<br= >openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270016", O= _RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory)<br>openat(AT_F= DCWD, "/var/repos/svn/projects/db/revs/270/270000", O_RDONLY|O_CL= OEXEC) =3D -1 ENOENT (No such file or directory)<br>openat(AT_FDCWD, "= /var/repos/svn/projects/db/revs/270/270000", O_RDONLY|O_CLOEXEC) =3D -= 1 ENOENT (No such file or directory)</span></span></span></div><div><span><= span><span><br></span></span></span></div><div><span><span><span>Note, the = directory 270 didn't exist.=C2=A0 According to google, this can happen = due to either a hook or CI committing too often creating corruption.=C2=A0 = We can't seem to clear the corruption though.=C2=A0 We are also a bit= =C2=A0</span></span></span>skeptical of its corruption as it should have cl= eared on a freshly restored repository.</div><div><span><span><span><br></s= pan></span></span></div><div><span><span><span><br></span></span></span></d= iv><div><span><span><span>So over the weekend, we attempted to separate the= projects on the repo to individual repos.=C2=A0 This is to reduce the size= of the repo as it could have been causing the issue.=C2=A0 We however kept= the revision numbers as they are in documentations and tags.=C2=A0 However= , on the now small repo, with only 187 GB in size, but with the last succes= sful commit #=C2=A0</span></span></span>269999, we still see the problem.</= div><div><br></div><div>This now looks more like a software bug than a setu= p issue.=C2=A0 We have "yum updated" the system in an attempt to = make sure we have all the bug fixes out there but nothing works.</div><div>= <br></div><div>Would anyone know if there is a transaction limit in the cur= rent SVN code?=C2=A0 Anyone have a repository with more than 270000 revisio= ns out there please?=C2=A0 Any pointers to something we may have overlooked= so far?=C2=A0 Something else, the svndump=C2=A0is cleanly at 2 TB, any cha= nce there may be a limit of how big a repo glow??=C2=A0 This was our first = thought but since we=C2=A0 have created smaller repos and the issue remains= , it now looks far more like a sharding logic error to us.</div><div><br></= div><div><br></div><div>Regards,</div><div>William</div><div><br></div><div= ><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div= ><br></div><div><br></div><div><br></div></div> </blockquote></div> --0000000000006afa660652a77e8a--