Re: recovering corrupt file system
Stephen Samuel <[email protected]> Fri, 20 Nov 2015 13:08:40 -0800
| Newsgroups | gmane.comp.file-systems.ext3.user |
|---|---|
| Message-ID | <CALp1NBhiyxPLuObm6Y3D9zDAXeCWee72UWx20mOQ+4v12QQARA@mail.gmail.com> |
--===============3186047637098440175==
Content-Type: multipart/alternative; boundary=001a11348b0499e7830524ff47c6
--001a11348b0499e7830524ff47c6
Content-Type: text/plain; charset=UTF-8
You can try using the secondary superblock:
fsck -b 32768 /dev/whatever
This presumes that you're using 4K blocks in the filesystem. you can get
a (more accurate) list of available
secondary superblocks with
mkfs -n -{other options used to make the filesystem} /dev/whatever
On Thu, Nov 19, 2015 at 7:06 PM, Boylan, Ross <[email protected]> wrote:
> I tried the "just run e2fsck", but it reduced the filesystem to almost
> nothing. Before there were nearly 700G of files; after there were 70G.*
> Also, the overall filesystem size shrunk to under 300G. I ran resize2fs
> to get the space back, but of course that didn't get the files back.
>
> This seems like an awful lot of damage from losing a total of 8,192 bytes
> out of ~700G. Maybe the first block of zeros caused the recovery to decide
> it had reached the end? The logical volume got about 64GB from the first,
> presumably OK, virtual disk. The holes in the file occur around 174GB into
> the 2nd virtual hard disk.
>
> I've still got copies from before e2fsck, and I'm still interested in
> recovering them (lots of recorded shows on them).
>
> Sorry about the top-posting; my mail client doesn't provide good way to do
> otherwise.
> Ross
>
> *I had expected only to lose the newer files, but a lot of the program
> files seem to be gone too. startx doesn't exist, for example.
>
>
> From: Boylan, Ross
>
> Sent: Thursday, November 19, 2015 1:13 PM
>
> To: Stephen Samuel
>
> Cc: [email protected]
>
> Subject: RE: recovering corrupt file system
>
>
>
>
>
>
> Thanks for the pointer. Turning to my other bad file system, I could use
> some help interpreting e2fsck. I have the source and have been looking at
> various web resources, so I suppose
> I could figure this out eventually.
>
>
>
> Actually, maybe I should ask a simpler question: should I just run e2fck,
> accepting its recommendations, and live with the results? No matter what I
> do I don't think I can recover any more information.
>
>
>
>
> Here's a little diagram:
>
> media01/root # LVM logical volume on which the ext4 filesystem resides
>
> VM's sda, sdb, various partitions # physical volumes making up the
> media01VG
>
> ------ virtual machine above here ---
>
> --- physical machine/ host below here -------------------
>
> media01b.vdi # host file backing virtual disk sdb
>
> # note I have made a spare copy of media01b.vdi.
>
> # The file backing virtual sda had no hardware problems.
>
> ## various more layers here
>
> physical disk
>
>
>
> The physical disk at the bottom is failing. I used (g)ddrescue to copy as
> much of the media01b.vdi file as I could; the file is about 700G, and there
> were 2 chunks of 0x1000 bytes that could not be recovered and are now 0
> filled.
>
>
>
> The basic structure of the virtual disks appears intact: the partition
> tables are still there and the logical volumes can still be assembled.
>
>
>
> If it's worth getting into the details, here's what e2fsck, run inside
> another VM that has the problems disks temporarily inserted says. What do
> the individual block bitmap differences mean? I'm guessing + and minus
> indicate whether the block was found in
> the scan only or in the file system tables on disk one, but I don't know
> which. And what do the numbers mean? Offsets in bytes? sectors? relative
> to ??
>
>
>
> root@wheezy02:~# e2fsck -vn /dev/media01-vg/root
>
> e2fsck 1.42.12 (29-Aug-2014)
>
> One or more block group descriptor checksums are invalid. Fix? no
>
>
>
> Group descriptor 465 checksum is 0x5e7a, should be 0xa22b. IGNORED.
>
> Group descriptor 482 checksum is 0x69eb, should be 0x73a5. IGNORED.
>
> Group descriptor 485 checksum is 0xbd9b, should be 0x21c9. IGNORED.
>
> Group descriptor 496 checksum is 0xe550, should be 0x9a62. IGNORED.
>
> Group descriptor 508 checksum is 0xf4d0, should be 0x2466. IGNORED.
>
> /dev/media01-vg/root contains a file system with errors, check forced.
>
> Pass 1: Checking inodes, blocks, and sizes
>
> Pass 2: Checking directory structure
>
> Pass 3: Checking directory connectivity
>
> Pass 4: Checking reference counts
>
> Pass 5: Checking group summary information
>
> Block bitmap differences: +15243264 +(15511418--15511423)
> +(15511488--15511551) +(15523264--15523327) +(15812608--15813174)
> -(15813349--15813503) -(15813632--158\
>
> 14054) +(15814656--15815176) +(15815680--15816191) -(15816505--15816703)
> -(15823872--15824895) -(15850505--15851519) -(15852544--15853567)
> -(15896583--15898623) +\
>
> (16029735--16029759) +(16029786--16029823) +(16029852--16029855)
> +(16029884--16031743) -(16261152--16261954) -(16263740--16263743) -16459791
> +(16459795--16459799)\
>
> -(16459808--16459814) -(16459825--16459839) -(16460000--16460026)
> -(16460288--16460799) +(16668689--16670719) +(17210175--17211391) +17498112
> -(17498624--1749913\
>
> 5) -(17499262--17500159) +17534976 -(17536000--17537023)
> -(17953505--17954815) +(18026714--18028543) +(18031327--18032639)
> -(18032653--18034687) +(18062252--18063\
>
> 359) +(18655232--18657173) -(19314688--19316735) -(19331072--19333119)
> -19406880 -25174048 -(41954376--41955015) -(42999849--42999850)
> -(42999852--42999859) -(429\
>
> 99861--42999862) -(43524128--43546654) -(45621280--45625870)
> -(45637632--45641520) -46669856 -(48242720--48246756) -(67436544--67446783)
>
> Fix? no
>
>
>
> Free blocks count wrong for group #473 (0, counted=134).
>
> Fix? no
>
>
>
> ## quite a few more Free blocks wrong messages
>
>
>
> Free blocks count wrong (48434540, counted=48384585).
>
> Fix? no
>
>
>
> Inode bitmap differences: -4849670
>
> Fix? no
>
>
>
> Free inodes count wrong for group #592 (8187, counted=8186).
>
> Fix? no
>
>
>
> Free inodes count wrong (16811016, counted=16811015).
>
> Fix? no
>
>
>
> Padding at end of block bitmap is not set. Fix? no
>
>
>
>
>
> /dev/media01-vg/root: ********** WARNING: Filesystem still has errors
> **********
>
>
>
>
>
> 56312 inodes used (0.33%, out of 16867328)
>
> 91 non-contiguous files (0.2%)
>
> 53 non-contiguous directories (0.1%)
>
> # of inodes with ind/dind/tind blocks: 0/0/0
>
> Extent depth histogram: 51385/43
>
> 19012244 blocks used (28.19%, out of 67446784)
>
> 0 bad blocks
>
> 14 large files
>
>
>
> 46167 regular files
>
> 5101 directories
>
> 12 character device files
>
> 25 block device files
>
> 0 fifos
>
>
>
>
>
> From: [email protected] [[email protected]] on behalf of Stephen Samuel [
> [email protected]]
>
> Sent: Thursday, November 19, 2015 8:00 AM
>
> To: Boylan, Ross
>
> Cc: [email protected]
>
> Subject: Re: recovering corrupt file system
>
>
>
>
>
>
> well, the next place to go, if fsck isn't enough would be to to try
> debugfs(1)
> man debugfs.
>
>
>
> On Wed, Nov 18, 2015 at 8:39 PM, Boylan, Ross
> <[email protected]> wrote:
>
>
> I guess some of the trouble was that the virtual disk was mounted
> read-only at the VM level. When I mounted read/write I was able to do
> fsck, which gave messages about replaying the logs and a couple messages
> about changing the inode counts (sorry, don't have
> the exact words). Then I ran fsck -f, which didn't report any problems.
> Then I mounted it, and everything seems OK.
>
>
>
> I'm still interested in the general question about how to diagnose and
> recover from file system errors, since I have another virtual machine that
> was backed by a failing real disk.
>
> ________________________________________
>
> From: Boylan, Ross
>
> Sent: Wednesday, November 18, 2015 4:35 PM
>
> To:
> [email protected]
>
> Subject: recovering corrupt file system
>
>
>
>
> Any recommendations for tools to diagnose and recover problems on an ext4
> file system?
>
>
>
> In particular:
>
> root@jessie01:~# mount -o ro /dev/markov02/root /mnt/markov02
>
> mount: wrong fs type, bad option, bad superblock on
> /dev/mapper/markov02-root,
>
> missing codepage or helper program, or other error
>
>
>
> In some cases useful info is found in syslog - try
>
> dmesg | tail or so.
>
> and e2fsck says
>
> root@jessie01:~# e2fsck /dev/markov02/root
>
> e2fsck 1.42.12 (29-Aug-2014)
>
> /dev/markov02/root: recovering journal
>
> Superblock needs_recovery flag is clear, but journal has data.
>
>
>
> markov02/root is an LVM volume, built on partitions from 2 disks in a
> virtual machine. The initial symptom was that the VM the disks were in
> originally would only get as far as busybox when it started. However, I
> think the filesystem was OK even after that,
> since it was visible in busybox and in another VM. I think virt-manager
> might have overwritten on of the disks because I left "allocate entire disk
> now" checked when I moved one of the disks between machines.
>
>
>
> I'm making copies of the virtual disks now.
>
> Ross Boylan
>
>
>
> _______________________________________________
>
> Ext3-users mailing list
>
> [email protected]
>
> https://www.redhat.com/mailman/listinfo/ext3-users
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Stephen Samuel
> http://www.bcgreen.com Software, like love,
>
> 778-861-7641 grows when you give it away
>
>
>
>
>
>
>
>
--
Stephen Samuel http://www.bcgreen.com Software, like love,
778-861-7641 grows when you give it away
--001a11348b0499e7830524ff47c6
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">=C2=A0You can try using the secondary superblock:=C2=A0<di=
v>fsck -b=C2=A032768 /dev/whatever=C2=A0</div><div><br></div><div>This pres=
umes that =C2=A0you're using 4K blocks in the filesystem. =C2=A0you can=
get a (more accurate)=C2=A0 list of available=C2=A0</div><div>secondary su=
perblocks with=C2=A0</div><div><br></div><div>mkfs -n -{other options used =
to make the filesystem} =C2=A0/dev/whatever</div><div><br></div><div><br></=
div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Thu,=
Nov 19, 2015 at 7:06 PM, Boylan, Ross <span dir=3D"ltr"><<a href=3D"mai=
lto:[email protected]" target=3D"_blank">[email protected]</a>></s=
pan> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex=
;border-left:1px #ccc solid;padding-left:1ex">I tried the "just run e2=
fsck", but it reduced the filesystem to almost nothing.=C2=A0 Before t=
here were nearly 700G of files; after there were 70G.*=C2=A0 Also, the over=
all filesystem size shrunk to under 300G.=C2=A0 I ran resize2fs to=C2=A0 ge=
t the=C2=A0 space back, but of course that didn't get the files back.<b=
r>
<br>
This seems like an awful lot of damage from losing a total of 8,192 bytes o=
ut of ~700G.=C2=A0 Maybe the first block of zeros caused the recovery to de=
cide it had reached the end?=C2=A0 The logical volume got about 64GB from t=
he first, presumably OK, virtual disk.=C2=A0 The holes in the file occur ar=
ound 174GB into the 2nd virtual hard disk.<br>
<br>
I've still got copies from before e2fsck, and I'm still interested =
in recovering them (lots of recorded shows on them).<br>
<br>
Sorry about the top-posting; my mail client doesn't provide good way to=
do otherwise.<br>
Ross<br>
<br>
*I had expected only to lose the newer files, but a lot of the program file=
s seem to be gone too. startx doesn't exist, for example.<br>
<br>
<br>
From: Boylan, Ross<br>
<br>
Sent: Thursday, November 19, 2015 1:13 PM<br>
<br>
To: Stephen Samuel<br>
<br>
Cc: <a href=3D"mailto:[email protected]">[email protected]</a><br>
<br>
Subject: RE: recovering corrupt file system<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br>
<br>
<br>
<br>
<br>
<br>
Thanks for the pointer.=C2=A0 Turning to my other bad file system, I could =
use some help interpreting e2fsck.=C2=A0 I have the source and have been lo=
oking at various web resources, so=C2=A0 I suppose<br>
=C2=A0I could figure this out eventually.<br>
<br>
<br>
<br>
Actually, maybe I should ask a simpler question: should I just run e2fck, a=
ccepting its recommendations, and live with the results?=C2=A0 No matter wh=
at I do I don't think I can recover any more information.<br>
<br>
<br>
<br>
<br>
Here's a little diagram:<br>
<br>
media01/root=C2=A0 =C2=A0# LVM logical volume on which the ext4 filesystem =
resides<br>
<br>
VM's sda, sdb, various partitions=C2=A0 =C2=A0# physical volumes making=
up the media01VG<br>
<br>
------ virtual machine above here ---<br>
<br>
--- physical machine/ host below here -------------------<br>
<br>
media01b.vdi=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 # host file backing virtual disk sdb<br>
<br>
# note I have made a spare copy of media01b.vdi.<br>
<br>
# The file backing virtual sda had no hardware problems.<br>
<br>
## various more layers here<br>
<br>
physical disk<br>
<br>
<br>
<br>
The physical disk at the bottom is failing. I used (g)ddrescue to copy as m=
uch of the media01b.vdi file as I could; the file is about 700G, and there =
were 2 chunks of 0x1000 bytes that could not be recovered and are now 0 fil=
led.<br>
<br>
<br>
<br>
The basic structure of the virtual disks appears intact: the partition tabl=
es are still there and the logical volumes can still be assembled.<br>
<br>
<br>
<br>
If it's worth getting into the details, here's what e2fsck, run ins=
ide another VM that has the problems disks temporarily inserted says.=C2=A0=
What do the individual block bitmap differences mean?=C2=A0 I'm guessi=
ng + and minus indicate whether the block was found in<br>
=C2=A0the scan only or in the file system tables on disk one, but I don'=
;t know which.=C2=A0 And what do the numbers mean?=C2=A0 Offsets in bytes? =
sectors? relative to ??<br>
<br>
<br>
<br>
root@wheezy02:~# e2fsck -vn /dev/media01-vg/root<br>
<br>
e2fsck 1.42.12 (29-Aug-2014)<br>
<br>
One or more block group descriptor checksums are invalid.=C2=A0 Fix? no<br>
<br>
<br>
<br>
Group descriptor 465 checksum is 0x5e7a, should be 0xa22b.=C2=A0 IGNORED.<b=
r>
<br>
Group descriptor 482 checksum is 0x69eb, should be 0x73a5.=C2=A0 IGNORED.<b=
r>
<br>
Group descriptor 485 checksum is 0xbd9b, should be 0x21c9.=C2=A0 IGNORED.<b=
r>
<br>
Group descriptor 496 checksum is 0xe550, should be 0x9a62.=C2=A0 IGNORED.<b=
r>
<br>
Group descriptor 508 checksum is 0xf4d0, should be 0x2466.=C2=A0 IGNORED.<b=
r>
<br>
/dev/media01-vg/root contains a file system with errors, check forced.<br>
<br>
Pass 1: Checking inodes, blocks, and sizes<br>
<br>
Pass 2: Checking directory structure<br>
<br>
Pass 3: Checking directory connectivity<br>
<br>
Pass 4: Checking reference counts<br>
<br>
Pass 5: Checking group summary information<br>
<br>
Block bitmap differences:=C2=A0 +15243264 +(15511418--15511423) +(15511488-=
-15511551) +(15523264--15523327) +(15812608--15813174) -(15813349--15813503=
) -(15813632--158\<br>
<br>
14054) +(15814656--15815176) +(15815680--15816191) -(15816505--15816703) -(=
15823872--15824895) -(15850505--15851519) -(15852544--15853567) -(15896583-=
-15898623) +\<br>
<br>
(16029735--16029759) +(16029786--16029823) +(16029852--16029855) +(16029884=
--16031743) -(16261152--16261954) -(16263740--16263743) -16459791 +(1645979=
5--16459799)\<br>
<br>
=C2=A0-(16459808--16459814) -(16459825--16459839) -(16460000--16460026) -(1=
6460288--16460799) +(16668689--16670719) +(17210175--17211391) +17498112 -(=
17498624--1749913\<br>
<br>
5) -(17499262--17500159) +17534976 -(17536000--17537023) -(17953505--179548=
15) +(18026714--18028543) +(18031327--18032639) -(18032653--18034687) +(180=
62252--18063\<br>
<br>
359) +(18655232--18657173) -(19314688--19316735) -(19331072--19333119) -194=
06880 -25174048 -(41954376--41955015) -(42999849--42999850) -(42999852--429=
99859) -(429\<br>
<br>
99861--42999862) -(43524128--43546654) -(45621280--45625870) -(45637632--45=
641520) -46669856 -(48242720--48246756) -(67436544--67446783)<br>
<br>
Fix? no<br>
<br>
<br>
<br>
Free blocks count wrong for group #473 (0, counted=3D134).<br>
<br>
Fix? no<br>
<br>
<br>
<br>
## quite a few more Free blocks wrong messages<br>
<br>
<br>
<br>
Free blocks count wrong (48434540, counted=3D48384585).<br>
<br>
Fix? no<br>
<br>
<br>
<br>
Inode bitmap differences:=C2=A0 -4849670<br>
<br>
Fix? no<br>
<br>
<br>
<br>
Free inodes count wrong for group #592 (8187, counted=3D8186).<br>
<br>
Fix? no<br>
<br>
<br>
<br>
Free inodes count wrong (16811016, counted=3D16811015).<br>
<br>
Fix? no<br>
<br>
<br>
<br>
Padding at end of block bitmap is not set. Fix? no<br>
<br>
<br>
<br>
<br>
<br>
/dev/media01-vg/root: ********** WARNING: Filesystem still has errors *****=
*****<br>
<br>
<br>
<br>
<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A056312 inodes used (0.33%, out of 16867328)<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 91 non-contiguous files (0.2%)<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 53 non-contiguous directories (0.1%)<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0# of inodes with ind/dind/t=
ind blocks: 0/0/0<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Extent depth histogram: 513=
85/43<br>
<br>
=C2=A0 =C2=A0 19012244 blocks used (28.19%, out of 67446784)<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 bad blocks<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 14 large files<br>
<br>
<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A046167 regular files<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 5101 directories<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 12 character device files<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 25 block device files<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 fifos<br>
<br>
<br>
<br>
<br>
<br>
</div></div><div class=3D"HOEnZb"><div class=3D"h5">From: <a href=3D"mailto=
:[email protected]">[email protected]</a> [<a href=3D"mailto:darkonc@gmail.=
com">[email protected]</a>] on behalf of Stephen Samuel [<a href=3D"mailto:=
[email protected]">[email protected]</a>]<br>
<br>
Sent: Thursday, November 19, 2015 8:00 AM<br>
<br>
To: Boylan, Ross<br>
<br>
Cc: <a href=3D"mailto:[email protected]">[email protected]</a><br>
<br>
Subject: Re: recovering corrupt file system<br>
<br>
<br>
<br>
<br>
<br>
<br>
well, the next place to go, if fsck isn't enough would be to to try deb=
ugfs(1)<br>
man debugfs.<br>
<br>
<br>
<br>
On Wed, Nov 18, 2015 at 8:39 PM, Boylan, Ross<br>
<<a href=3D"mailto:[email protected]">[email protected]</a>> wr=
ote:<br>
<br>
<br>
I guess some of the trouble was that the virtual disk was mounted read-only=
at the VM level.=C2=A0 When I mounted read/write I was able to do fsck, wh=
ich gave messages about replaying the logs and a couple messages about chan=
ging the inode counts (sorry, don't have<br>
=C2=A0the exact words).=C2=A0 Then I ran fsck -f, which didn't report a=
ny problems.=C2=A0 Then I mounted it, and everything seems OK.<br>
<br>
<br>
<br>
I'm still interested in the general question about how to diagnose and =
recover from file system errors, since I have another virtual machine that =
was backed by a failing real disk.<br>
<br>
________________________________________<br>
<br>
From: Boylan, Ross<br>
<br>
Sent: Wednesday, November 18, 2015 4:35 PM<br>
<br>
To:<br>
<a href=3D"mailto:[email protected]">[email protected]</a><br>
<br>
Subject: recovering corrupt file system<br>
<br>
<br>
<br>
<br>
Any recommendations for tools to diagnose and recover problems on an ext4 f=
ile system?<br>
<br>
<br>
<br>
In particular:<br>
<br>
root@jessie01:~# mount -o ro /dev/markov02/root /mnt/markov02<br>
<br>
mount: wrong fs type, bad option, bad superblock on /dev/mapper/markov02-ro=
ot,<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0missing codepage or helper program, or other err=
or<br>
<br>
<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0In some cases useful info is found in syslog - t=
ry<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0dmesg | tail or so.<br>
<br>
and e2fsck says<br>
<br>
root@jessie01:~# e2fsck /dev/markov02/root<br>
<br>
e2fsck 1.42.12 (29-Aug-2014)<br>
<br>
/dev/markov02/root: recovering journal<br>
<br>
Superblock needs_recovery flag is clear, but journal has data.<br>
<br>
<br>
<br>
markov02/root is an LVM volume, built on partitions from 2 disks in a virtu=
al machine.=C2=A0 The initial symptom was that the VM the disks were in ori=
ginally would only get as far as busybox when it started.=C2=A0 However, I =
think the filesystem was OK even after that,<br>
=C2=A0since it was visible in busybox and in another VM.=C2=A0 I think virt=
-manager might have overwritten on of the disks because I left "alloca=
te entire disk now" checked when I moved one of the disks between mach=
ines.<br>
<br>
<br>
<br>
I'm making copies of the virtual disks now.<br>
<br>
Ross Boylan<br>
<br>
<br>
<br>
_______________________________________________<br>
<br>
Ext3-users mailing list<br>
<br>
<a href=3D"mailto:[email protected]">[email protected]</a><br>
<br>
<a href=3D"https://www.redhat.com/mailman/listinfo/ext3-users" rel=3D"noref=
errer" target=3D"_blank">https://www.redhat.com/mailman/listinfo/ext3-users=
</a><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
--<br>
<br>
Stephen Samuel<br>
<a href=3D"http://www.bcgreen.com" rel=3D"noreferrer" target=3D"_blank">htt=
p://www.bcgreen.com</a>=C2=A0 Software, like love,<br>
<br>
<a href=3D"tel:778-861-7641" value=3D"+17788617641">778-861-7641</a>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 grows when you give it away<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
</div></div></blockquote></div><br><br clear=3D"all"><br>-- <br><div class=
=3D"gmail_signature">Stephen Samuel <a href=3D"http://www.bcgreen.com" targ=
et=3D"_blank">http://www.bcgreen.com</a>=C2=A0 Software, like love, <br>778=
-861-7641=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 grows when you give it away</div>
</div>
--001a11348b0499e7830524ff47c6--
--===============3186047637098440175==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Ext3-users mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/ext3-users
--===============3186047637098440175==--