Re: Many orphaned inodes after resize2fs
Patrik Horník <[email protected]> Sat, 19 Apr 2014 17:42:12 +0200
| Newsgroups | gmane.comp.file-systems.ext3.user |
|---|---|
| Message-ID | <CAAOsTSn8v3XmG-ri7Om+=em8zZVQjecdf5A0ng8jhZsntAWfCg@mail.gmail.com> |
--047d7bacbaaa3629c804f7671c1e Content-Type: multipart/alternative; boundary=047d7bacbaaa3629c404f7671c1c --047d7bacbaaa3629c404f7671c1c Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable OK, so I patched it myself and confirmed that all errors were caused by this, because patched version does not warn about any inode and about anything. Thus it seems resize2fs did not harm the filesystem at all and it was all because of the false positive in e2fsck. I patched it by considering inode part of suspected corrupted orphan list only if i_dtime is lower than specific constant around 1.1 bilion. This is some time before creation of my filesystem. You can find patch against 1.42.9 attached. Please confirm that this is fully correct solution (for my purpose, not elegant clean way for official fix) and it has no negative consequences. It seems that way but I did not analyze all code paths the fixed code is in. BTW were there any other negative consequences of this bug in e2fsck except changing i_dtime of inodes to current time? Thanks. Patrik 2014-04-19 1:20 GMT+02:00 Patrik Horn=EDk <[email protected]>: > Hi, > > it seems you got it right! I don't know if you read email I sent you > before posting to the mailing list, but I accidentally diagnosed the > cause... :) I've noticed that inodes fsck warned me about, at least ones > that I checked, all have all four timestamps latest in 2010... > > The filesystem has maximum 1281998848 inodes, which is timestamp in augus= t > 2010. I don't know how it got that big, I think I did not specified big > value initially. But I've resized it couple of times. BTW what is default > of group size / inode count ratio? Mine ratio is not at the maximum you > mentioned, but it is not that far. > > So almost sure it is false positive by the code / bug in e2fsck/pass1.c > around line 1070 in current version. I want to be sure that all these > errors were caused by this, so can you please send me promptly patched > version? I can easily patch it myself by some fixed condition, but I don'= t > want miss something important... BTW maybe you can compare i_dtime with > filesystem creation timestamp, so you dont have to put fixed number there= . > > BTW I dont know specifics of ext3, I just looked at sources of kernel > driver and e2fsprogs now. But what indicates that inode is / was created > and valid ? (I did not need it to find problematic test you mentioned, di= d > not see it in part of code I look at and it is not apparent to me from > definition of struct ext3_inode). > > Thanks. > > Patrik > > > 2014-04-18 22:20 GMT+02:00 <[email protected]>: > > On Fri, Apr 18, 2014 at 06:56:57PM +0200, Patrik Horn=EDk wrote: >> > >> > yesterday I experienced following problem with my ext3 filesystem: >> > >> > - I had ext3 filesystem of the size of a few TB with journal. I >> correctly >> > unmounted it and it was marked clean. >> > >> > - I then ran fsck.etx3 -f on it and it did not find any problem. >> > >> > - After increasing size of its LVM volume by 1.5 TB I resized the >> > filesystem by resize2fs lvm_volume and it finished without problem. >> > >> > - But fsck.ext3 -f immediately after that showed "Inodes that were par= t >> of >> > a corrupted orphan linked list found." and many thousands of "Inode XX= X >> was >> > part of the orphaned inode list." I did not accepted fix. According to >> > debugfs all the inodes I check from these reported orphaned inodes (I >> > checked only some from beginning of list of errors) have size 0. >> >> Can you send the output of dumpe2fs -h? I'm curious how many inodes >> you had after the resize, and what file system features might have >> been enabled on your file system. >> >> If the only file system corruption errors that you saw were from about >> the corrupted orphan inode list, then things are probably OK. >> >> What this error message means is that there are d_time values which >> look like they belong to inode numbers (as opposed to number of >> seconds since January 1, 1970). So if you ran the system where the >> clock was set incorrectly, so that the time was January 1, 1970, and >> you delete a lot of files, you can run into this error --- it's >> basically a sanity check that we put in a long time ago to catch >> potential file system bugs caused by a corrupted orphan inode list. >> >> I'm thinking that we should turn off this check if the e2fsck.conf >> "broken_system_lock" is enabled, since if the system has a busted >> system clock, this can end up triggering a bunch of scary warnings. >> >> In any case, when you grew the size of the file system, this also >> increased the number of inodes, which means it would increase the >> sensitivity of hitting this bug. It's also possible that if you >> created your file system with the number of inodes per block group >> close to the maximum (assuming an average file size 4k, which would be >> highly wasteful of space, so it' s not the default), that you ended up >> with the maximum number of inodes exceeding 1.2 or 1.3 billion inodes, >> at which point it would trigger a false positive. (And indeed, I >> should probably put in a fix to e2fsprogs so that if a file system >> does have more than 1.2 billion inodes, to disable this check.) >> >> Cheers, >> >> - Ted > > --047d7bacbaaa3629c404f7671c1c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">OK,<div><br></div><div>so I patched it myself and confirme= d that all errors were caused by this, because patched version does not war= n about any inode and about anything. Thus it seems resize2fs did not harm = the filesystem at all and it was all because of the false positive in e2fsc= k.</div> <div><br></div><div>I patched it by considering inode part of suspected cor= rupted orphan list only if i_dtime is lower than specific constant around 1= .1 bilion. This is some time before creation of my filesystem. You can find= patch against 1.42.9 attached.</div> <div><br></div><div>Please confirm that this is fully correct solution (for= my purpose, not elegant clean way for official fix) and it has no negative= consequences. It seems that way but I did not analyze all code paths the f= ixed code is in.</div> <div><br></div><div>BTW were there any other negative consequences of this = bug in e2fsck except changing i_dtime of inodes to current time?</div><div>= <br></div><div>Thanks.</div><div class=3D"gmail_extra"><br clear=3D"all"><d= iv> Patrik</div> <br><br><div class=3D"gmail_quote">2014-04-19 1:20 GMT+02:00 Patrik Horn=ED= k <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D"_blan= k">[email protected]</a>></span>:<br><blockquote class=3D"gmail_quote" st= yle=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb= (204,204,204);border-left-style:solid;padding-left:1ex"> <div dir=3D"ltr"><div>Hi,</div><div><br></div><div>it seems you got it righ= t! I don't know if you read email I sent you before posting to the mail= ing list, but I accidentally diagnosed the cause... :) I've noticed tha= t inodes fsck warned me about, at least ones that I checked, all have all f= our timestamps latest in 2010...</div> <div><br></div><div>The filesystem has maximum 1281998848 inodes, which is = timestamp in august 2010. I don't know how it got that big, I think I d= id not specified big value initially. But I've resized it couple of tim= es. BTW what is default of group size / inode count ratio? Mine ratio is no= t at the maximum you mentioned, but it is not that far.</div> <div><br></div><div>So almost sure it is false positive by the code / bug i= n e2fsck/pass1.c around line 1070 in current version. I want to be sure tha= t all these errors were caused by this, so can you please send me promptly = patched version? I can easily patch it myself by some fixed condition, but = I don't want miss something important... BTW maybe you can compare i_dt= ime with filesystem creation timestamp, so you dont have to put fixed numbe= r there.</div> <div><br></div><div>BTW I dont know specifics of ext3, I just looked at sou= rces of kernel driver and e2fsprogs now. But what indicates that inode is /= was created and valid ? (I did not need it to find problematic test you me= ntioned, did not see it in part of code I look at and it is not apparent to= me from definition of=A0<span style=3D"font-size:13px;color:rgb(131,0,0)">= struct</span><span style=3D"font-size:13px"> ext3_inode</span>).</div> <div><br></div><div>Thanks.</div><div><br></div><div class=3D"gmail_extra">= <div>Patrik</div> <br><br><div class=3D"gmail_quote">2014-04-18 22:20 GMT+02:00 <span dir=3D= "ltr"><<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]<= /a>></span>:<div><div class=3D"h5"><br><blockquote class=3D"gmail_quote"= style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:= rgb(204,204,204);border-left-style:solid;padding-left:1ex"> <div>On Fri, Apr 18, 2014 at 06:56:57PM +0200, Patrik Horn=EDk wrote:<br> ><br> > yesterday I experienced following problem with my ext3 filesystem:<br> ><br> > - I had ext3 filesystem of the size of a few TB with journal. I correc= tly<br> > unmounted it and it was marked clean.<br> ><br> > - I then ran fsck.etx3 -f on it and it did not find any problem.<br> ><br> > - After increasing size of its LVM volume by 1.5 TB I resized the<br> > filesystem by resize2fs lvm_volume and it finished without problem.<br= > ><br> > - But fsck.ext3 -f immediately after that showed "Inodes that wer= e part of<br> > a corrupted orphan linked list found." and many thousands of &quo= t;Inode XXX was<br> > part of the orphaned inode list." I did not accepted fix. Accordi= ng to<br> > debugfs all the inodes I check from these reported orphaned inodes (I<= br> > checked only some from beginning of list of errors) have size 0.<br> <br> </div>Can you send the output of dumpe2fs -h? =A0I'm curious how many i= nodes<br> you had after the resize, and what file system features might have<br> been enabled on your file system.<br> <br> If the only file system corruption errors that you saw were from about<br> the corrupted orphan inode list, then things are probably OK.<br> <br> What this error message means is that there are d_time values which<br> look like they belong to inode numbers (as opposed to number of<br> seconds since January 1, 1970). =A0So if you ran the system where the<br> clock was set incorrectly, so that the time was January 1, 1970, and<br> you delete a lot of files, you can run into this error --- it's<br> basically a sanity check that we put in a long time ago to catch<br> potential file system bugs caused by a corrupted orphan inode list.<br> <br> I'm thinking that we should turn off this check if the e2fsck.conf<br> "broken_system_lock" is enabled, since if the system has a busted= <br> system clock, this can end up triggering a bunch of scary warnings.<br> <br> In any case, when you grew the size of the file system, this also<br> increased the number of inodes, which means it would increase the<br> sensitivity of hitting this bug. =A0It's also possible that if you<br> created your file system with the number of inodes per block group<br> close to the maximum (assuming an average file size 4k, which would be<br> highly wasteful of space, so it' s not the default), that you ended up<= br> with the maximum number of inodes exceeding 1.2 or 1.3 billion inodes,<br> at which point it would trigger a false positive. =A0(And indeed, I<br> should probably put in a fix to e2fsprogs so that if a file system<br> does have more than 1.2 billion inodes, to disable this check.)<br> <br> Cheers,<br> <br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 - Ted</blockquote></div></div></div></div></div> </blockquote></div><br></div></div> --047d7bacbaaa3629c404f7671c1c-- --047d7bacbaaa3629c804f7671c1e Content-Type: application/octet-stream; name="big-fs-fsck-fix.patch" Content-Disposition: attachment; filename="big-fs-fsck-fix.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hu723df10 ZGlmZiAtdXJwTiBlMmZzcHJvZ3MtMS40Mi45L2UyZnNjay9wYXNzMS5jIGUyZnNwcm9ncy0xLjQy LjktZGV2L2UyZnNjay9wYXNzMS5jCi0tLSBlMmZzcHJvZ3MtMS40Mi45L2UyZnNjay9wYXNzMS5j CTIwMTMtMTItMjkgMDU6MTg6MDIuMDAwMDAwMDAwICswMTAwCisrKyBlMmZzcHJvZ3MtMS40Mi45 LWRldi9lMmZzY2svcGFzczEuYwkyMDE0LTA0LTE5IDE3OjIxOjEwLjI5NzcwNDA4MyArMDIwMApA QCAtOTkzLDYgKzk5Myw3IEBAIHZvaWQgZTJmc2NrX3Bhc3MxKGUyZnNja190IGN0eCkKIAkJICog c2hvdWxkbid0IGJlIGFueSBidWdzIGluIHRoZSBvcnBoYW4gbGlzdCBoYW5kbGluZy4gIDotKQog CQkgKi8KIAkJaWYgKGlub2RlLT5pX2R0aW1lICYmICFidXN0ZWRfZnNfdGltZSAmJgorCQkgICAg aW5vZGUtPmlfZHRpbWUgPCAxMTgzMjQwODAwICYmCiAJCSAgICBpbm9kZS0+aV9kdGltZSA8IGN0 eC0+ZnMtPnN1cGVyLT5zX2lub2Rlc19jb3VudCkgewogCQkJaWYgKGZpeF9wcm9ibGVtKGN0eCwg UFJfMV9MT1dfRFRJTUUsICZwY3R4KSkgewogCQkJCWlub2RlLT5pX2R0aW1lID0gaW5vZGUtPmlf bGlua3NfY291bnQgPwo= --047d7bacbaaa3629c804f7671c1e Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ext3-users mailing list [email protected] https://www.redhat.com/mailman/listinfo/ext3-users --047d7bacbaaa3629c804f7671c1e--