Re: Could this be SVN shard directory logic bug?
Daniel Sahlberg <[email protected]> Mon, 25 May 2026 20:06:32 +0200
| Newsgroups | gmane.comp.version-control.subversion.user |
|---|---|
| Message-ID | <CAMHy98MPKQ_vLStvapXk=oURM58g-5nNM64DdEXci+x8u4Y6Gg@mail.gmail.com> |
--000000000000c6241a0652a83d20 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, If you look at one of your existing working copies (run for example svn info from the command line), what is the highest revision number you have there? I have a feeling that something has caused the 270 directory to disappear, thus your repository is now in an inconsistent state. I'm not sure if you can recover except restoring a backup. If you can run svnadmin dump and then svnadmin load to a new repository, you can probably get a working repository with the first 269999 revisions, but you of course loose everything later than that. To answer the question about maximum revision number: The ASF main repository (https://svn.apache.org/repos/asf) is currently at revision 1934607, so over 7 times your repository. I'm not sure about size but I believe it is quite huge. Kind regards, Daniel Den m=C3=A5n 25 maj 2026 kl 19:13 skrev William Muriithi < [email protected]>: > Hi guys and girls > > > Something I missed to mention. Originally, the file contents looked like > this:- > > [root@repos ~]# cat /var/repos/svn/projects/db/current > 269999 > [root@repos ~]# cat /var/repos/svn/projects/db/txn-current > 5sbz > > These files seem to be important on how SVN makes the decision of which > directory to dump/look for a file. So I looked around and google resul= t > suggested that the last file may be incorrect. > > So I changed it to:- > > [root@repos ~]# cat /var/repos/svn/projects/db/txn-current > 5r3j > > MOVE forward: > I then created the files strace point as SVN access points. All below > commands were executed as user apache > > mkdir /var/repos/svn/projects/db/revs/270 > cd /var/repos/svn/projects/db/revs/270 > touch 270000 270016 270080 270336 > svnadmin recover /var/repos/svn/projects/ > > When I tried committing a new, I got a new error:- > > svnadmin: E160004: Revision 270336 has a revs file but no revprops file > > MOVE backward: > [root@repos ~]# cat /var/repos/svn/projects/db/current > 269999 > [root@repos ~]# cat /var/repos/svn/projects/db/txn-current > 5r3j > > svnadmin recover /var/repos/svn/projects/ (As apache user) - It run > successfully. > > Then we went back to SVN client and got the old error > > Basically, we can't seem to move it back or forward. > > Regards, > William > > > > On Mon, 25 May 2026 at 12:24, William Muriithi <[email protected]= m> > wrote: > >> Hi guys, >> >> [root@repos ~]# rpm -qa | grep subversion >> subversion-libs-1.14.5-3.el10.x86_64_v2 >> subversion-1.14.5-3.el10.x86_64_v2 >> subversion-tools-1.14.5-3.el10.x86_64_v2 >> [root@repos ~]# >> [root@repos ~]# >> [root@repos ~]# cat /etc/redhat-release >> AlmaLinux release 10.1 (Heliotrope Lion) >> [root@repos ~]# >> [root@repos ~]# rpm -qa | grep svn >> mod_dav_svn-1.14.5-3.el10.x86_64_v2 >> >> >> I think I have hit a bug. We have a relatively large SVN repository - >> About 2TB uncompressed. 2 weeks ago, someone noticed that we can't comm= it >> any more. When we looked at it, it looked like a permission issue. >> >> This is the error we are getting from the SVN client. >> >> svn: E000001: Commit failed (details follow): >> svn: E000001: Can't set permissions on >> '/var/repos/svn/application/db/revs/270' >> william@william-ryzen:~/Documents/application$ >> [email protected] >> >> So we first attempted to "FIX" permissions. Didn't work. Then, when >> reading, I noticed opening up the permission is not a good idea as I can >> see from strace that svn does try to narrow down the file permissions. = So >> we decided to restore a fresh to remove all the chmod we had done to the >> files and directory. >> >> This time, when restoring, we created the repo using an apache account. >> We also loaded the dump file using the svn account. This removed all >> possibility this is a file permission issue. The repo was restored >> successfully, BUT the problem remained. >> >> We looked at the file limit too and to make sure that isn't causing any >> issues, we changed it to unlimited for the apache user. The problem >> remained. >> >> It's at this point we attempted to use strace to see what svn is doing. >> Oddly, it seemed to be looking for files from directory 270, which didn'= t >> exist at the time. We came to the conclusion that was what was happenin= g >> because these came up when one attempts to commit. >> >> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270336", >> O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory) >> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270336", >> O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory) >> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270080", >> O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory) >> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270080", >> O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory) >> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270016", >> O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory) >> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270016", >> O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory) >> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270000", >> O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory) >> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270000", >> O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory) >> >> Note, the directory 270 didn't exist. According to google, this can >> happen due to either a hook or CI committing too often creating >> corruption. We can't seem to clear the corruption though. We are also = a >> bit skeptical of its corruption as it should have cleared on a freshly >> restored repository. >> >> >> So over the weekend, we attempted to separate the projects on the repo t= o >> individual repos. This is to reduce the size of the repo as it could ha= ve >> been causing the issue. We however kept the revision numbers as they ar= e >> in documentations and tags. However, on the now small repo, with only 1= 87 >> GB in size, but with the last successful commit # 269999, we still see >> the problem. >> >> This now looks more like a software bug than a setup issue. We have "yu= m >> updated" the system in an attempt to make sure we have all the bug fixes >> out there but nothing works. >> >> Would anyone know if there is a transaction limit in the current SVN >> code? Anyone have a repository with more than 270000 revisions out ther= e >> please? Any pointers to something we may have overlooked so far? >> Something else, the svndump is cleanly at 2 TB, any chance there may be = a >> limit of how big a repo glow?? This was our first thought but since we >> have created smaller repos and the issue remains, it now looks far more >> like a sharding logic error to us. >> >> >> Regards, >> William >> >> >> >> >> >> >> >> >> >> --000000000000c6241a0652a83d20 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Hi,</div><div><br></div><div>If you look at one of yo= ur existing working copies (run for example svn info from the command line)= , what is the highest revision number you have there?</div><div><br></div><= div>I have a feeling that something has caused the 270 directory to disappe= ar, thus your repository is now in an inconsistent state. I'm not sure = if you can recover except restoring a backup. If you can run svnadmin dump = and then svnadmin load to a new repository, you can probably get a working = repository with the first 269999 revisions, but you of course loose everyth= ing later than that.</div><div><br></div><div>To answer the question about = maximum revision number: The ASF main repository (<a href=3D"https://svn.ap= ache.org/repos/asf">https://svn.apache.org/repos/asf</a>) is currently at r= evision 1934607, so over 7 times your repository. I'm not sure about si= ze but I believe it is quite huge.</div><div><br></div><div>Kind regards,</= div><div>Daniel</div><div><br></div><div><br></div><br><div class=3D"gmail_= quote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">Den m=C3= =A5n 25 maj 2026 kl 19:13 skrev William Muriithi <<a href=3D"mailto:will= [email protected]">[email protected]</a>>:<br></div><block= quote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1= px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Hi guys a= nd girls</div><div><br></div><div><br></div><div>Something I missed to ment= ion. Originally, the file contents looked like this:-</div><div><br></div><= div>[root@repos ~]# cat /var/repos/svn/projects/db/current<br>269999<br>[ro= ot@repos ~]# cat /var/repos/svn/projects/db/txn-current<br>5sbz</div><div><= br></div><div>These files seem to be important on how SVN makes the decisio= n of which directory to dump/look for a file.=C2=A0 =C2=A0So I looked aroun= d=C2=A0 and google result suggested that the last file may be incorrect.</d= iv><div><br></div><div>So I changed it to:-</div><div><br></div><div>[root@= repos ~]# cat /var/repos/svn/projects/db/txn-current<br>5r3j</div><div><br>= </div><div>MOVE forward:</div><div>I then created the files strace point as= SVN access points.=C2=A0 All below commands were executed as user apache</= div><div><br></div><div>mkdir /var/repos/svn/projects/db/revs/270</div><div= >cd /var/repos/svn/projects/db/revs/270</div><div>touch 270000 270016 27008= 0 270336</div><div>svnadmin recover /var/repos/svn/projects/</div><div><br>= </div><div>When I tried committing a new, I got a new error:-</div><div><br= ></div><div>svnadmin: E160004: Revision 270336 has a revs file but no revpr= ops file</div><div><br></div><div>MOVE backward:</div><div>[root@repos ~]# = cat /var/repos/svn/projects/db/current<br>269999</div><div>[root@repos ~]# = cat /var/repos/svn/projects/db/txn-current<br>5r3j</div><div><br></div><div= >svnadmin recover /var/repos/svn/projects/ (As apache user) - It run succes= sfully.=C2=A0</div><div><br></div><div>Then we went back to SVN client and = got the old error</div><div><br></div><div>Basically, we can't seem to = move it back or forward.</div><div><br></div><div>Regards,</div><div>Willia= m=C2=A0</div><div><br></div><div><br></div></div><br><div class=3D"gmail_qu= ote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, 25 May 2026 at 12:24, Wi= lliam Muriithi <<a href=3D"mailto:[email protected]" target=3D"= _blank">[email protected]</a>> wrote:<br></div><blockquote clas= s=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid r= gb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Hi guys,</div><div>= <br></div><div>[root@repos ~]# rpm -qa | grep subversion<br>subversion-libs= -1.14.5-3.el10.x86_64_v2<br>subversion-1.14.5-3.el10.x86_64_v2<br>subversio= n-tools-1.14.5-3.el10.x86_64_v2<br>[root@repos ~]# <br>[root@repos ~]# <br>= [root@repos ~]# cat /etc/redhat-release <br>AlmaLinux release 10.1 (Heliotr= ope Lion)<br>[root@repos ~]#</div><div>[root@repos ~]# rpm -qa | grep svn<b= r>mod_dav_svn-1.14.5-3.el10.x86_64_v2</div><div><br></div><div><br></div><d= iv>I think I have hit a bug.=C2=A0 We have a relatively large SVN repositor= y - About 2TB uncompressed.=C2=A0 2 weeks ago, someone noticed that we can&= #39;t commit any more.=C2=A0 When we looked at it, it looked like a permiss= ion issue.</div><div><br></div><div>This is the error we are getting from t= he SVN client.</div><div><br>svn: E000001: Commit failed (details follow):<= br>svn: E000001: Can't set permissions on '/var/repos/svn/applicati= on/db/revs/270'<br>william@william-ryzen:~/Documents/application$ <a hr= ef=3D"mailto:[email protected]" target=3D"_blank">users@subversio= n.apache.org</a><br></div><div><br></div><div>So we first attempted to &quo= t;FIX" permissions.=C2=A0 Didn't work.=C2=A0 Then, when reading, I= noticed opening up the permission is not a good idea as I can see from str= ace that svn does try to narrow down the file permissions.=C2=A0 So we deci= ded to restore a fresh to remove all the chmod we had done to the files and= directory.</div><div><br></div><div>This time, when restoring, we created = the repo using an apache account.=C2=A0 We also loaded the dump file using = the svn account.=C2=A0 This removed all possibility=C2=A0this is a file per= mission issue.=C2=A0 The repo was restored successfully, BUT the problem re= mained.</div><div><br></div><div>We looked at the file limit too and to mak= e sure that isn't causing any issues, we changed it to unlimited for th= e apache user.=C2=A0 The problem remained.</div><div><br></div><div>It'= s at this point we attempted to use strace to see what svn is doing.=C2=A0 = Oddly, it seemed to be looking for files from directory 270, which didn'= ;t exist at the time.=C2=A0 We came to the conclusion that was what was hap= pening because these came up when one attempts to commit.</div><div><br></d= iv><div><span><span><span>openat(AT_FDCWD, "/var/repos/svn/projects/db= /revs/270/270336", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or = directory)<br>openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/27= 0336", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory)<b= r>openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270080", = O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory)<br>openat(AT_= FDCWD, "/var/repos/svn/projects/db/revs/270/270080", O_RDONLY|O_C= LOEXEC) =3D -1 ENOENT (No such file or directory)<br>openat(AT_FDCWD, "= ;/var/repos/svn/projects/db/revs/270/270016", O_RDONLY|O_CLOEXEC) =3D = -1 ENOENT (No such file or directory)<br>openat(AT_FDCWD, "/var/repos/= svn/projects/db/revs/270/270016", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (N= o such file or directory)<br>openat(AT_FDCWD, "/var/repos/svn/projects= /db/revs/270/270000", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file = or directory)<br>openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270= /270000", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory= )</span></span></span></div><div><span><span><span><br></span></span></span= ></div><div><span><span><span>Note, the directory 270 didn't exist.=C2= =A0 According to google, this can happen due to either a hook or CI committ= ing too often creating corruption.=C2=A0 We can't seem to clear the cor= ruption though.=C2=A0 We are also a bit=C2=A0</span></span></span>skeptical= of its corruption as it should have cleared on a freshly restored reposito= ry.</div><div><span><span><span><br></span></span></span></div><div><span><= span><span><br></span></span></span></div><div><span><span><span>So over th= e weekend, we attempted to separate the projects on the repo to individual = repos.=C2=A0 This is to reduce the size of the repo as it could have been c= ausing the issue.=C2=A0 We however kept the revision numbers as they are in= documentations and tags.=C2=A0 However, on the now small repo, with only 1= 87 GB in size, but with the last successful commit #=C2=A0</span></span></s= pan>269999, we still see the problem.</div><div><br></div><div>This now loo= ks more like a software bug than a setup issue.=C2=A0 We have "yum upd= ated" the system in an attempt to make sure we have all the bug fixes = out there but nothing works.</div><div><br></div><div>Would anyone know if = there is a transaction limit in the current SVN code?=C2=A0 Anyone have a r= epository with more than 270000 revisions out there please?=C2=A0 Any point= ers to something we may have overlooked so far?=C2=A0 Something else, the s= vndump=C2=A0is cleanly at 2 TB, any chance there may be a limit of how big = a repo glow??=C2=A0 This was our first thought but since we=C2=A0 have crea= ted smaller repos and the issue remains, it now looks far more like a shard= ing logic error to us.</div><div><br></div><div><br></div><div>Regards,</di= v><div>William</div><div><br></div><div><br></div><div><br></div><div><br><= /div><div><br></div><div><br></div><div><br></div><div><br></div><div><br><= /div></div> </blockquote></div> </blockquote></div></div> --000000000000c6241a0652a83d20--