Re: Could this be SVN shard directory logic bug?
Barry <[email protected]> Tue, 21 Jul 2026 14:36:18 +0100
| Newsgroups | gmane.comp.version-control.subversion.user |
|---|---|
| Message-ID | <[email protected]> |
--Apple-Mail-928165E7-86BE-4D59-9FEB-F28FD1009D38 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html class=3D"apple-mail-supports-explicit-dark-mode"><head><meta http-equi= v=3D"content-type" content=3D"text/html; charset=3Dutf-8"></head><body dir=3D= "auto"><div dir=3D"ltr"></div><div dir=3D"ltr"><br></div><div dir=3D"ltr"><b= r><blockquote type=3D"cite">On 25 May 2026, at 19:45, William Muriithi <w= [email protected]> wrote:<br><br></blockquote></div><blockquote t= ype=3D"cite"><div dir=3D"ltr">=EF=BB=BF<div dir=3D"ltr"><div dir=3D"ltr"><di= v>Hi Daniel</div><div><br></div></div><div class=3D"gmail_quote gmail_quote_= container"><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8= ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"= ><div>If you look at one of your existing working copies (run for example sv= n info from the command line), what is the highest revision number you have t= here?</div></div></blockquote><div><div>Have the same revision # on the clie= nt too</div><div><br></div><div><br></div><div>william@william-ryzen:~/Docum= ents/application$ svn info<br>Path: .<br>Working Copy Root Path: /home/willi= am/Documents/application<br>URL: <a href=3D"https://carbon.example.com/svn/a= pplication">https://carbon.example.com/svn/application</a><br>Relative URL: ^= /<br>Repository Root: <a href=3D"https://carbon.example.com/svn/application"= >https://carbon.example.com/svn/application</a><br>Repository UUID: 02a4eb3e= -e1fa-4ead-9f6a-c97b0409a8a7<br>Revision: 269999<br>Node Kind: directory<br>= Schedule: normal<br>Last Changed Author: nathan<br>Last Changed Rev: 269999<= br></div></div></div></div></div></blockquote><div><br></div>I hit a bug lik= e this with a repo that jammed on 1999.<div>I used something to force in r20= 00 and then everything started working again.</div><div><br></div><div>I sus= pect I use root and committed a trivial change use file:// url to the repo.<= /div><div>But it was a while ago and I did not keep notes.</div><div><br></d= iv><div>Barty</div><div><br></div><div><blockquote type=3D"cite"><div dir=3D= "ltr"><div dir=3D"ltr"><div class=3D"gmail_quote gmail_quote_container"><div= ><div>Last Changed Date: 2026-05-11 15:54:29 -0400 (Mon, 11 May 2026)<br><br= >william@william-ryzen:~/Documents/application$ </div> </div><bloc= kquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1= px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div><br></div>= <div>I have a feeling that something has caused the 270 directory to disappe= ar, thus your repository is now in an inconsistent state. I'm not sure if yo= u can recover except restoring a backup. If you can run svnadmin dump and th= en svnadmin load to a new repository, you can probably get a working reposit= ory with the first 269999 revisions, but you of course loose everything late= r than that.</div></div></blockquote><div><br></div><div>Possibly, I am open= to that being the case. Is there a way to be confident this is what h= appened?</div><div><br></div><div>- I have two systems - the backup which wa= s using svnsync is also in the same state. I suspect having a replica d= oesn't change much as cron was scheduled to run every 5 minutes, so if the d= irectory 270 was created and destroyed within 5 minutes, the replica wouldn'= t have picked the change</div><div>[root@boron ~]# cat /usr/local/bin/svnsyn= c.sh<br>/usr/bin/svnsync sync file:///var/repos/svn/projects</div><div= >- The last svndump restore leaves the repo in the same stat= e. Wouldn't the repo dump have flagged that something is off? Wo= uldn't the restore the restore fail when it try to replay the entry <sp= an><span><span>270080 that SVN seems to assume exit?</span></span></span></d= iv><div>- The svnadmin verify pass all the commits up to 269999</= div><div><br></div><div>What else can we do to figure out where the corrupti= on is?</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.= 8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr= "><div><br></div><div>To answer the question about maximum revision number: T= he ASF main repository (<a href=3D"https://svn.apache.org/repos/asf" target=3D= "_blank">https://svn.apache.org/repos/asf</a>) is currently at revision 1934= 607, so over 7 times your repository. I'm not sure about size but I believe i= t is quite huge.</div><div><br></div></div></blockquote><div><br></div><div>= Ok, so there exit repo bigger than this then, thanks.</div><div><br></div><d= iv><br></div><div>Regards,</div><div>William </div><blockquote class=3D= "gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(20= 4,204,204);padding-left:1ex"><div dir=3D"ltr"><div></div><div>Kind regards,<= /div><div>Daniel</div><div><br></div><div><br></div><br><div class=3D"gmail_= quote"><div dir=3D"ltr" class=3D"gmail_attr">Den m=C3=A5n 25 maj 2026 kl 19:= 13 skrev William Muriithi <<a href=3D"mailto:[email protected]" t= arget=3D"_blank">[email protected]</a>>:<br></div><blockquote cl= ass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid r= gb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Hi guys and girls</d= iv><div><br></div><div><br></div><div>Something I missed to mention. Origina= lly, the file contents looked like this:-</div><div><br></div><div>[root@rep= os ~]# cat /var/repos/svn/projects/db/current<br>269999<br>[root@repos ~]# c= at /var/repos/svn/projects/db/txn-current<br>5sbz</div><div><br></div><div>T= hese files seem to be important on how SVN makes the decision of which direc= tory to dump/look for a file. So I looked around and googl= e result suggested that the last file may be incorrect.</div><div><br></div>= <div>So I changed it to:-</div><div><br></div><div>[root@repos ~]# cat /var/= repos/svn/projects/db/txn-current<br>5r3j</div><div><br></div><div>MOVE forw= ard:</div><div>I then created the files strace point as SVN access points.&n= bsp; All below commands were executed as user apache</div><div><br></div><di= v>mkdir /var/repos/svn/projects/db/revs/270</div><div>cd /var/repos/svn/proj= ects/db/revs/270</div><div>touch 270000 270016 270080 270336</div><div>svnad= min recover /var/repos/svn/projects/</div><div><br></div><div>When I tried c= ommitting a new, I got a new error:-</div><div><br></div><div>svnadmin: E160= 004: Revision 270336 has a revs file but no revprops file</div><div><br></di= v><div>MOVE backward:</div><div>[root@repos ~]# cat /var/repos/svn/projects/= db/current<br>269999</div><div>[root@repos ~]# cat /var/repos/svn/projects/d= b/txn-current<br>5r3j</div><div><br></div><div>svnadmin recover /var/repos/s= vn/projects/ (As apache user) - It run successfully. </div><div><br></d= iv><div>Then we went back to SVN client and got the old error</div><div><br>= </div><div>Basically, we can't seem to move it back or forward.</div><div><b= r></div><div>Regards,</div><div>William </div><div><br></div><div><br><= /div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_at= tr">On Mon, 25 May 2026 at 12:24, William Muriithi <<a href=3D"mailto:wil= [email protected]" target=3D"_blank">[email protected]</a>>= ; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0= px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D= "ltr"><div>Hi guys,</div><div><br></div><div>[root@repos ~]# rpm -qa | grep s= ubversion<br>subversion-libs-1.14.5-3.el10.x86_64_v2<br>subversion-1.14.5-3.= el10.x86_64_v2<br>subversion-tools-1.14.5-3.el10.x86_64_v2<br>[root@repos ~]= # <br>[root@repos ~]# <br>[root@repos ~]# cat /etc/redhat-release <br>AlmaLi= nux release 10.1 (Heliotrope Lion)<br>[root@repos ~]#</div><div>[root@repos ~= ]# rpm -qa | grep svn<br>mod_dav_svn-1.14.5-3.el10.x86_64_v2</div><div><br><= /div><div><br></div><div>I think I have hit a bug. We have a relativel= y large SVN repository - About 2TB uncompressed. 2 weeks ago, someone n= oticed that we can't commit any more. When we looked at it, it looked l= ike a permission issue.</div><div><br></div><div>This is the error we are ge= tting from the SVN client.</div><div><br>svn: E000001: Commit failed (detail= s follow):<br>svn: E000001: Can't set permissions on '/var/repos/svn/applica= tion/db/revs/270'<br>william@william-ryzen:~/Documents/application$ <a href=3D= "mailto:[email protected]" target=3D"_blank">[email protected]= he.org</a><br></div><div><br></div><div>So we first attempted to "FIX" permi= ssions. Didn't work. Then, when reading, I noticed opening up th= e permission is not a good idea as I can see from strace that svn does try t= o narrow down the file permissions. So we decided to restore a fresh t= o remove all the chmod we had done to the files and directory.</div><div><br= ></div><div>This time, when restoring, we created the repo using an apache a= ccount. We also loaded the dump file using the svn account. This= removed all possibility this is a file permission issue. The rep= o was restored successfully, BUT the problem remained.</div><div><br></div><= div>We looked at the file limit too and to make sure that isn't causing any i= ssues, we changed it to unlimited for the apache user. The problem rem= ained.</div><div><br></div><div>It's at this point we attempted to use strac= e to see what svn is doing. Oddly, it seemed to be looking for files f= rom directory 270, which didn't exist at the time. We came to the conc= lusion that was what was happening because these came up when one attempts t= o commit.</div><div><br></div><div><span><span><span>openat(AT_FDCWD, "/var/= repos/svn/projects/db/revs/270/270336", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (N= o such file or directory)<br>openat(AT_FDCWD, "/var/repos/svn/projects/db/re= vs/270/270336", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory= )<br>openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270080", O_RDONLY= |O_CLOEXEC) =3D -1 ENOENT (No such file or directory)<br>openat(AT_FDCWD, "/= var/repos/svn/projects/db/revs/270/270080", O_RDONLY|O_CLOEXEC) =3D -1 ENOEN= T (No such file or directory)<br>openat(AT_FDCWD, "/var/repos/svn/projects/d= b/revs/270/270016", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or direc= tory)<br>openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270016", O_RD= ONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory)<br>openat(AT_FDCWD= , "/var/repos/svn/projects/db/revs/270/270000", O_RDONLY|O_CLOEXEC) =3D -1 E= NOENT (No such file or directory)<br>openat(AT_FDCWD, "/var/repos/svn/projec= ts/db/revs/270/270000", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or d= irectory)</span></span></span></div><div><span><span><span><br></span></span= ></span></div><div><span><span><span>Note, the directory 270 didn't exist.&n= bsp; According to google, this can happen due to either a hook or CI committ= ing too often creating corruption. We can't seem to clear the corrupti= on though. We are also a bit </span></span></span>skeptical of it= s corruption as it should have cleared on a freshly restored repository.</di= v><div><span><span><span><br></span></span></span></div><div><span><span><sp= an><br></span></span></span></div><div><span><span><span>So over the weekend= , we attempted to separate the projects on the repo to individual repos.&nbs= p; This is to reduce the size of the repo as it could have been causing the i= ssue. We however kept the revision numbers as they are in documentatio= ns and tags. However, on the now small repo, with only 187 GB in size,= but with the last successful commit # </span></span></span>269999, we s= till see the problem.</div><div><br></div><div>This now looks more like a so= ftware bug than a setup issue. We have "yum updated" the system in an a= ttempt to make sure we have all the bug fixes out there but nothing works.</= div><div><br></div><div>Would anyone know if there is a transaction limit in= the current SVN code? Anyone have a repository with more than 270000 r= evisions out there please? Any pointers to something we may have overl= ooked so far? Something else, the svndump is cleanly at 2 TB, any= chance there may be a limit of how big a repo glow?? This was our fir= st thought but since we have created smaller repos and the issue remai= ns, it now looks far more like a sharding logic error to us.</div><div><br><= /div><div><br></div><div>Regards,</div><div>William</div><div><br></div><div= ><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>= <br></div><div><br></div><div><br></div></div> </blockquote></div> </blockquote></div></div> </blockquote></div></div> </div></blockquote></div></body></html>= --Apple-Mail-928165E7-86BE-4D59-9FEB-F28FD1009D38--