RE: recovering corrupt file system
"Boylan, Ross" <[email protected]> Thu, 19 Nov 2015 21:13:09 +0000
| Newsgroups | gmane.comp.file-systems.ext3.user |
|---|---|
| Message-ID | <[email protected]> |
--===============8528072869855309582==
Content-Language: en-US
Content-Type: multipart/alternative;
boundary="_000_F1F13E14A610474196571953929C02090BDB7DFDex08netucsfedu_"
--_000_F1F13E14A610474196571953929C02090BDB7DFDex08netucsfedu_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Thanks for the pointer. Turning to my other bad file system, I could use s=
ome help interpreting e2fsck. I have the source and have been looking at v=
arious web resources, so I suppose I could figure this out eventually.
Actually, maybe I should ask a simpler question: should I just run e2fck, a=
ccepting its recommendations, and live with the results? No matter what I =
do I don't think I can recover any more information.
Here's a little diagram:
media01/root # LVM logical volume on which the ext4 filesystem resides
VM's sda, sdb, various partitions # physical volumes making up the media0=
1VG
------ virtual machine above here ---
--- physical machine/ host below here -------------------
media01b.vdi # host file backing virtual disk sdb
# note I have made a spare copy of media01b.vdi.
# The file backing virtual sda had no hardware problems.
## various more layers here
physical disk
The physical disk at the bottom is failing. I used (g)ddrescue to copy as m=
uch of the media01b.vdi file as I could; the file is about 700G, and there =
were 2 chunks of 0x1000 bytes that could not be recovered and are now 0 fil=
led.
The basic structure of the virtual disks appears intact: the partition tabl=
es are still there and the logical volumes can still be assembled.
If it's worth getting into the details, here's what e2fsck, run inside anot=
her VM that has the problems disks temporarily inserted says. What do the =
individual block bitmap differences mean? I'm guessing + and minus indicat=
e whether the block was found in the scan only or in the file system tables=
on disk one, but I don't know which. And what do the numbers mean? Offse=
ts in bytes? sectors? relative to ??
root@wheezy02:~# e2fsck -vn /dev/media01-vg/root
e2fsck 1.42.12 (29-Aug-2014)
One or more block group descriptor checksums are invalid. Fix? no
Group descriptor 465 checksum is 0x5e7a, should be 0xa22b. IGNORED.
Group descriptor 482 checksum is 0x69eb, should be 0x73a5. IGNORED.
Group descriptor 485 checksum is 0xbd9b, should be 0x21c9. IGNORED.
Group descriptor 496 checksum is 0xe550, should be 0x9a62. IGNORED.
Group descriptor 508 checksum is 0xf4d0, should be 0x2466. IGNORED.
/dev/media01-vg/root contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: +15243264 +(15511418--15511423) +(15511488--1551=
1551) +(15523264--15523327) +(15812608--15813174) -(15813349--15813503) -(1=
5813632--158\
14054) +(15814656--15815176) +(15815680--15816191) -(15816505--15816703) -(=
15823872--15824895) -(15850505--15851519) -(15852544--15853567) -(15896583-=
-15898623) +\
(16029735--16029759) +(16029786--16029823) +(16029852--16029855) +(16029884=
--16031743) -(16261152--16261954) -(16263740--16263743) -16459791 +(1645979=
5--16459799)\
-(16459808--16459814) -(16459825--16459839) -(16460000--16460026) -(164602=
88--16460799) +(16668689--16670719) +(17210175--17211391) +17498112 -(17498=
624--1749913\
5) -(17499262--17500159) +17534976 -(17536000--17537023) -(17953505--179548=
15) +(18026714--18028543) +(18031327--18032639) -(18032653--18034687) +(180=
62252--18063\
359) +(18655232--18657173) -(19314688--19316735) -(19331072--19333119) -194=
06880 -25174048 -(41954376--41955015) -(42999849--42999850) -(42999852--429=
99859) -(429\
99861--42999862) -(43524128--43546654) -(45621280--45625870) -(45637632--45=
641520) -46669856 -(48242720--48246756) -(67436544--67446783)
Fix? no
Free blocks count wrong for group #473 (0, counted=3D134).
Fix? no
## quite a few more Free blocks wrong messages
Free blocks count wrong (48434540, counted=3D48384585).
Fix? no
Inode bitmap differences: -4849670
Fix? no
Free inodes count wrong for group #592 (8187, counted=3D8186).
Fix? no
Free inodes count wrong (16811016, counted=3D16811015).
Fix? no
Padding at end of block bitmap is not set. Fix? no
/dev/media01-vg/root: ********** WARNING: Filesystem still has errors *****=
*****
56312 inodes used (0.33%, out of 16867328)
91 non-contiguous files (0.2%)
53 non-contiguous directories (0.1%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 51385/43
19012244 blocks used (28.19%, out of 67446784)
0 bad blocks
14 large files
46167 regular files
5101 directories
12 character device files
25 block device files
0 fifos
________________________________
From: [email protected] [[email protected]] on behalf of Stephen Samuel [sa=
[email protected]]
Sent: Thursday, November 19, 2015 8:00 AM
To: Boylan, Ross
Cc: [email protected]
Subject: Re: recovering corrupt file system
well, the next place to go, if fsck isn't enough would be to to try debugfs=
(1)
man debugfs.
On Wed, Nov 18, 2015 at 8:39 PM, Boylan, Ross <[email protected]<mailto:=
[email protected]>> wrote:
I guess some of the trouble was that the virtual disk was mounted read-only=
at the VM level. When I mounted read/write I was able to do fsck, which g=
ave messages about replaying the logs and a couple messages about changing =
the inode counts (sorry, don't have the exact words). Then I ran fsck -f, =
which didn't report any problems. Then I mounted it, and everything seems =
OK.
I'm still interested in the general question about how to diagnose and reco=
ver from file system errors, since I have another virtual machine that was =
backed by a failing real disk.
________________________________________
From: Boylan, Ross
Sent: Wednesday, November 18, 2015 4:35 PM
To: [email protected]<mailto:[email protected]>
Subject: recovering corrupt file system
Any recommendations for tools to diagnose and recover problems on an ext4 f=
ile system?
In particular:
root@jessie01:~# mount -o ro /dev/markov02/root /mnt/markov02
mount: wrong fs type, bad option, bad superblock on /dev/mapper/markov02-ro=
ot,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
and e2fsck says
root@jessie01:~# e2fsck /dev/markov02/root
e2fsck 1.42.12 (29-Aug-2014)
/dev/markov02/root: recovering journal
Superblock needs_recovery flag is clear, but journal has data.
markov02/root is an LVM volume, built on partitions from 2 disks in a virtu=
al machine. The initial symptom was that the VM the disks were in original=
ly would only get as far as busybox when it started. However, I think the =
filesystem was OK even after that, since it was visible in busybox and in a=
nother VM. I think virt-manager might have overwritten on of the disks bec=
ause I left "allocate entire disk now" checked when I moved one of the disk=
s between machines.
I'm making copies of the virtual disks now.
Ross Boylan
_______________________________________________
Ext3-users mailing list
[email protected]<mailto:[email protected]>
https://www.redhat.com/mailman/listinfo/ext3-users
--
Stephen Samuel http://www.bcgreen.com Software, like love,
778-861-7641 grows when you give it away
--_000_F1F13E14A610474196571953929C02090BDB7DFDex08netucsfedu_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html dir=3D"ltr">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style id=3D"owaParaStyle" type=3D"text/css">P {margin-top:0;margin-bottom:=
0;}</style>
</head>
<body ocsi=3D"0" fpstyle=3D"1">
<div style=3D"direction: ltr;font-family: Tahoma;color: #000000;font-size: =
10pt;">Thanks for the pointer. Turning to my other bad file system, I=
could use some help interpreting e2fsck. I have the source and have =
been looking at various web resources, so I
suppose I could figure this out eventually.<br>
<br>
Actually, maybe I should ask a simpler question: should I just run e2fck, a=
ccepting its recommendations, and live with the results? No matter wh=
at I do I don't think I can recover any more information.
<br>
<br>
Here's a little diagram:<br>
media01/root # LVM logical volume on which the ext4 filesystem =
resides<br>
VM's sda, sdb, various partitions # physical volumes making up =
the media01VG<br>
------ virtual machine above here ---<br>
--- physical machine/ host below here -------------------<br>
media01b.vdi &nb=
sp; # host file backing vir=
tual disk sdb<br>
# note I have made a spare copy of media01b.vdi.<br>
# The file backing virtual sda had no hardware problems.<br>
## various more layers here<br>
physical disk<br>
<br>
The physical disk at the bottom is failing. I used (g)ddrescue to copy as m=
uch of the media01b.vdi file as I could; the file is about 700G, and there =
were 2 chunks of 0x1000 bytes that could not be recovered and are now 0 fil=
led.<br>
<br>
The basic structure of the virtual disks appears intact: the partition tabl=
es are still there and the logical volumes can still be assembled.<br>
<br>
If it's worth getting into the details, here's what e2fsck, run inside anot=
her VM that has the problems disks temporarily inserted says. What do=
the individual block bitmap differences mean? I'm guessing + and=
minus indicate whether the block was found in
the scan only or in the file system tables on disk one, but I don't know w=
hich. And what do the numbers mean? Offsets in bytes? sectors? =
relative to ??<br>
<br>
root@wheezy02:~# e2fsck -vn /dev/media01-vg/root<br>
e2fsck 1.42.12 (29-Aug-2014)<br>
One or more block group descriptor checksums are invalid. Fix? no<br>
<br>
Group descriptor 465 checksum is 0x5e7a, should be 0xa22b. IGNORED.<b=
r>
Group descriptor 482 checksum is 0x69eb, should be 0x73a5. IGNORED.<b=
r>
Group descriptor 485 checksum is 0xbd9b, should be 0x21c9. IGNORED.<b=
r>
Group descriptor 496 checksum is 0xe550, should be 0x9a62. IGNORED.<b=
r>
Group descriptor 508 checksum is 0xf4d0, should be 0x2466. IGNORED.<b=
r>
/dev/media01-vg/root contains a file system with errors, check forced.<br>
Pass 1: Checking inodes, blocks, and sizes<br>
Pass 2: Checking directory structure<br>
Pass 3: Checking directory connectivity<br>
Pass 4: Checking reference counts<br>
Pass 5: Checking group summary information<br>
Block bitmap differences: +15243264 +(15511418--15511423) =
3;(15511488--15511551) +(15523264--15523327) +(15812608--15813174) =
-(15813349--15813503) -(15813632--158\<br>
14054) +(15814656--15815176) +(15815680--15816191) -(15816505--1581=
6703) -(15823872--15824895) -(15850505--15851519) -(15852544--15853567) -(1=
5896583--15898623) +\<br>
(16029735--16029759) +(16029786--16029823) +(16029852--16029855) &#=
43;(16029884--16031743) -(16261152--16261954) -(16263740--16263743) -164597=
91 +(16459795--16459799)\<br>
-(16459808--16459814) -(16459825--16459839) -(16460000--16460026) -(1=
6460288--16460799) +(16668689--16670719) +(17210175--17211391) +=
;17498112 -(17498624--1749913\<br>
5) -(17499262--17500159) +17534976 -(17536000--17537023) -(17953505--17=
954815) +(18026714--18028543) +(18031327--18032639) -(18032653--180=
34687) +(18062252--18063\<br>
359) +(18655232--18657173) -(19314688--19316735) -(19331072--19333119) =
-19406880 -25174048 -(41954376--41955015) -(42999849--42999850) -(42999852-=
-42999859) -(429\<br>
99861--42999862) -(43524128--43546654) -(45621280--45625870) -(45637632--45=
641520) -46669856 -(48242720--48246756) -(67436544--67446783)<br>
Fix? no<br>
<br>
Free blocks count wrong for group #473 (0, counted=3D134).<br>
Fix? no<br>
<br>
## quite a few more Free blocks wrong messages<br>
<br>
Free blocks count wrong (48434540, counted=3D48384585).<br>
Fix? no<br>
<br>
Inode bitmap differences: -4849670<br>
Fix? no<br>
<br>
Free inodes count wrong for group #592 (8187, counted=3D8186).<br>
Fix? no<br>
<br>
Free inodes count wrong (16811016, counted=3D16811015).<br>
Fix? no<br>
<br>
Padding at end of block bitmap is not set. Fix? no<br>
<br>
<br>
/dev/media01-vg/root: ********** WARNING: Filesystem still has errors *****=
*****<br>
<br>
<br>
56312 inodes used (0.33%, out of 16867=
328)<br>
91 non-contiguous fi=
les (0.2%)<br>
53 non-contiguous di=
rectories (0.1%)<br>
# =
of inodes with ind/dind/tind blocks: 0/0/0<br>
Ex=
tent depth histogram: 51385/43<br>
19012244 blocks used (28.19%, out of 67446784)<br>
0 bad blocks<b=
r>
14 large files<br>
<br>
46167 regular files<br>
5101 directories<br>
12 character device =
files<br>
25 block device file=
s<br>
0 fifos<br>
<br>
<div style=3D"font-family: Times New Roman; color: #000000; font-size: 16px=
">
<hr tabindex=3D"-1">
<div style=3D"direction: ltr;" id=3D"divRpF798926"><font color=3D"#000000" =
face=3D"Tahoma" size=3D"2"><b>From:</b> [email protected] [[email protected]=
m] on behalf of Stephen Samuel [[email protected]]<br>
<b>Sent:</b> Thursday, November 19, 2015 8:00 AM<br>
<b>To:</b> Boylan, Ross<br>
<b>Cc:</b> [email protected]<br>
<b>Subject:</b> Re: recovering corrupt file system<br>
</font><br>
</div>
<div></div>
<div>
<div dir=3D"ltr">well, the next place to go, if fsck isn't enough would be =
to to try debugfs(1)
<div>man debugfs.</div>
</div>
<div class=3D"gmail_extra"><br>
<div class=3D"gmail_quote">On Wed, Nov 18, 2015 at 8:39 PM, Boylan, Ross <s=
pan dir=3D"ltr">
<<a href=3D"mailto:[email protected]" target=3D"_blank">Ross.Boylan@u=
csf.edu</a>></span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex; border-left:1=
px #ccc solid; padding-left:1ex">
I guess some of the trouble was that the virtual disk was mounted read-only=
at the VM level. When I mounted read/write I was able to do fsck, wh=
ich gave messages about replaying the logs and a couple messages about chan=
ging the inode counts (sorry, don't have
the exact words). Then I ran fsck -f, which didn't report any proble=
ms. Then I mounted it, and everything seems OK.<br>
<br>
I'm still interested in the general question about how to diagnose and reco=
ver from file system errors, since I have another virtual machine that was =
backed by a failing real disk.<br>
________________________________________<br>
From: Boylan, Ross<br>
Sent: Wednesday, November 18, 2015 4:35 PM<br>
To: <a href=3D"mailto:[email protected]" target=3D"_blank">Ext3-users@r=
edhat.com</a><br>
Subject: recovering corrupt file system<br>
<div class=3D"HOEnZb">
<div class=3D"h5"><br>
Any recommendations for tools to diagnose and recover problems on an ext4 f=
ile system?<br>
<br>
In particular:<br>
root@jessie01:~# mount -o ro /dev/markov02/root /mnt/markov02<br>
mount: wrong fs type, bad option, bad superblock on /dev/mapper/markov02-ro=
ot,<br>
missing codepage or helper program, or other err=
or<br>
<br>
In some cases useful info is found in syslog - t=
ry<br>
dmesg | tail or so.<br>
and e2fsck says<br>
root@jessie01:~# e2fsck /dev/markov02/root<br>
e2fsck 1.42.12 (29-Aug-2014)<br>
/dev/markov02/root: recovering journal<br>
Superblock needs_recovery flag is clear, but journal has data.<br>
<br>
markov02/root is an LVM volume, built on partitions from 2 disks in a virtu=
al machine. The initial symptom was that the VM the disks were in ori=
ginally would only get as far as busybox when it started. However, I =
think the filesystem was OK even after that,
since it was visible in busybox and in another VM. I think virt-mana=
ger might have overwritten on of the disks because I left "allocate en=
tire disk now" checked when I moved one of the disks between machines.=
<br>
<br>
I'm making copies of the virtual disks now.<br>
Ross Boylan<br>
<br>
_______________________________________________<br>
Ext3-users mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">Ext3-users@redha=
t.com</a><br>
<a href=3D"https://www.redhat.com/mailman/listinfo/ext3-users" rel=3D"noref=
errer" target=3D"_blank">https://www.redhat.com/mailman/listinfo/ext3-users=
</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear=3D"all">
<div><br>
</div>
-- <br>
<div class=3D"gmail_signature">Stephen Samuel <a href=3D"http://www.bcgreen=
.com" target=3D"_blank">
http://www.bcgreen.com</a> Software, like love, <br>
778-861-7641 =
grows when you give it away</div>
</div>
</div>
</div>
</div>
</body>
</html>
--_000_F1F13E14A610474196571953929C02090BDB7DFDex08netucsfedu_--
--===============8528072869855309582==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Ext3-users mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/ext3-users
--===============8528072869855309582==--