Re: Troubleshoot full pool for 4.0 worth it?
Craig Barratt <[email protected]> Sat, 8 Apr 2017 10:54:41 -0700
| Newsgroups | gmane.comp.sysutils.backup.backuppc.devel,gmane.comp.sysutils.backup.backuppc.general |
|---|---|
| Message-ID | <CADSzEFhGHm6v=T9EX98bDmMu=T46sPk+Uo=xCnURh_A62nL-wQ@mail.gmail.com> |
--===============1687539459299936972==
Content-Type: multipart/alternative; boundary=001a113b0506fdf9ca054cab7054
--001a113b0506fdf9ca054cab7054
Content-Type: text/plain; charset=UTF-8
Bill,
/backuppc/cpool/0/0/0 is just one of the 4096 3.x pool directories (each of
the last three digits is a single hex value from 0..9a..f). So to see the
total storage remaining in the 3.x pool you should do this:
du -csh /backuppc/cpool/?/?/?
I'm not sure why some of the 3.x files didn't get migrated. You could pick
one that has more than one link (eg: 00002564f9012849e45bfa1f4fd47578
above) and find its inode:
ls -li /backuppc/cpool/0/0/0/00002564f9012849e45bfa1f4fd47578
then look for other files that have that same inode (replace NNN with the
inode printed by ls -i):
find /backuppc -inum NNN -print
But given your point about 0/0/0 being quite small, it's unlikely this can
explain 3TB of extra usage, and I suspect the du command above won't show
more than a few MB.
So another path is to use du to find which directories are so large. For
example:
du -hs /backuppc
If that number is reasonable, then it must be something outside /backuppc
that is using so much space.
Next:
du -hs /backuppc/cpool /backuppc/pool /backuppc/pc
Are any of those close to 3TB? If so, do the du inside those directories
to narrow things down.
Is it possible your excludes aren't work after the 4.x transition? For
example, on some linux systems /var/log/lastlog is a sparse file, and
backing it up will create a huge (regular) file.
You could also use find to look for single huge files, eg:
find /backuppc -size +1G -print
will list all files over 1G.
Craig
On Fri, Apr 7, 2017 at 1:08 AM, Bill Broadley <[email protected]> wrote:
>
> On 04/05/2017 03:25 PM, higuita wrote:
> > Hi
> >
> > On Tue, 4 Apr 2017 23:04:44 -0700, Bill Broadley <[email protected]>
> > wrote:
> >> -rw-r----- 1 backuppc backuppc 145 Feb 11 2015
> 00012f3df3fef9176f4a08f470d1f5e6
> > ^ |
> > This field is the number of hardlinks.
> > So you have entries >1, then you still have backups pointing to the v3
> pool.
>
> Odd.
>
> I ran the V3 to V4 migration script several times and it wasn't finding
> anything
> and running quickly. I was worried that my filesystem was corrupt
> somehow, it
> had been up for 360 some days. I umounted and fsck'd, not a single
> complaint.
>
> root@node1:/backuppc/cpool/0/0/0# ls -al | awk ' { print $2 } ' | grep -v
> "1" |
> wc -l
> 39
>
> I didn't have many with more than one link. The entire dir is small:
> root@node1:/backuppc/cpool/0/0/0# du -hs .
> 380K .
>
> I see similar elsewhere:
> root@node1:/backuppc/cpool/8/8/8# ls -al |wc -l; ls -al | awk ' { print
> $2 } ' |
> grep -v "1" | wc -l
> 59
> 31
>
> (59 files, 31 with links).
>
> I'm still seeing crazy disk usage, over 3TB, only about 650GB (total from
> the
> host status "full size" column) visible to backuppc.
>
> Keep in mind this happened with no changes to the server. 30 hosts backed
> up
> for a week or so, then suddenly much more disk is used. No host has larger
> backups, just a factor of 6 larger pool one night.
>
> I was using the v3 to v4 migration script from git since it wasn't in the
> release package yet (that's been fixed.
>
> I upgraded to backuppc 4.1.1, and the current versions of backuppc-xs and
> rsync-bpc. I ran the V3 to V4 migration script (now included in the
> release)
> again and it's doing some serious chewing (unlike before). It used to
> just fly
> through them all with "refCnt directory; skipping this backup".
>
> So maybe this will fix it.
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> BackupPC-devel mailing list
> [email protected]
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-devel
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
--001a113b0506fdf9ca054cab7054
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Bill,<div><br></div><div><span style=3D"font-size:12.8px">=
/backuppc/cpool/0/</span><wbr style=3D"font-size:12.8px"><span style=3D"fon=
t-size:12.8px">0/0 is just one of the 4096 3.x pool directories (each of th=
e last three digits is a single hex value from 0..9a..f).=C2=A0 So to see t=
he total storage remaining in the 3.x pool you should do this:</span><br></=
div><div><span style=3D"font-size:12.8px"><br></span></div><div><span style=
=3D"font-size:12.8px">=C2=A0 =C2=A0 du -csh=C2=A0</span><span style=3D"font=
-size:12.8px">/backuppc/cpool/?/?/?</span></div><div><span style=3D"font-si=
ze:12.8px"><br></span></div><div>I'm not sure why some of the 3.x files=
didn't get migrated.=C2=A0 You could pick one that has more than one l=
ink (eg:=C2=A0<span style=3D"font-size:12.8px">00002564f9012849e45bfa1f4fd4=
75</span><wbr style=3D"font-size:12.8px"><span style=3D"font-size:12.8px">7=
8 above) and find its inode:</span></div><div><span style=3D"font-size:12.8=
px"><br></span></div><div><span style=3D"font-size:12.8px">=C2=A0 =C2=A0ls =
-li=C2=A0</span><span style=3D"font-size:12.8px">/backuppc/</span><span sty=
le=3D"font-size:12.8px">cpool/0/0/0/</span><span style=3D"font-size:12.8px"=
>00002564f9012849e45bfa1f4fd475</span><wbr style=3D"font-size:12.8px"><span=
style=3D"font-size:12.8px">78</span></div><div><span style=3D"font-size:12=
.8px"><br></span></div><div><span style=3D"font-size:12.8px">then look for =
other files that have that same inode (replace NNN with the inode printed b=
y ls -i):</span></div><div><span style=3D"font-size:12.8px"><br></span></di=
v><div><span style=3D"font-size:12.8px">=C2=A0 =C2=A0find </span><span styl=
e=3D"font-size:12.8px">/backuppc -inum NNN -print</span></div><div><br></di=
v><div>But given your point about 0/0/0 being quite small, it's unlikel=
y this can explain 3TB of extra usage, and I suspect the du command above w=
on't show more than a few MB.</div><div><br></div><div>So another path =
is to use du to find which directories are so large.=C2=A0 For example:</di=
v><div><br></div><div>=C2=A0 =C2=A0 du -hs /backuppc</div><div><br></div><d=
iv>If that number is reasonable, then it must be something outside /backupp=
c that is using so much space.</div><div><br>Next:</div><div><br></div><div=
>=C2=A0 =C2=A0du -hs /backuppc/cpool /backuppc/pool /backuppc/pc</div><div>=
<br></div><div>Are any of those close to 3TB?=C2=A0 If so, do the du inside=
those directories to narrow things down.</div><div>=C2=A0</div><div>Is it =
possible your excludes aren't work after the 4.x transition?=C2=A0 For =
example, on some linux systems=C2=A0/var/log/lastlog is a sparse file, and =
backing it up will create a huge (regular) file.</div><div><br></div><div>Y=
ou could also use find to look for single huge files, eg:</div><div><br></d=
iv><div>=C2=A0 =C2=A0 find /backuppc -size +1G -print</div><div><br></div><=
div>will list all files over 1G.</div><div><br></div><div>Craig</div><div><=
br></div><div><br></div><div><br></div></div><div class=3D"gmail_extra"><br=
><div class=3D"gmail_quote">On Fri, Apr 7, 2017 at 1:08 AM, Bill Broadley <=
span dir=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D"_blank"=
>[email protected]</a>></span> wrote:<br><blockquote class=3D"gmail_quot=
e" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">=
<span class=3D""><br>
On 04/05/2017 03:25 PM, higuita wrote:<br>
> Hi<br>
><br>
> On Tue, 4 Apr 2017 23:04:44 -0700, Bill Broadley <<a href=3D"mailto=
:[email protected]">[email protected]</a>><br>
> wrote:<br>
</span><span class=3D"">>> -rw-r-----=C2=A0 1 backuppc backuppc=C2=A0=
=C2=A0 145 Feb 11=C2=A0 2015<br>
00012f3df3fef9176f4a08f470d1f5<wbr>e6<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0^=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|<br>
> This field is the number of hardlinks.<br>
> So you have entries >1, then you still have backups pointing to the=
v3 pool.<br>
<br>
</span>Odd.<br>
<br>
I ran the V3 to V4 migration script several times and it wasn't finding=
anything<br>
and running quickly.=C2=A0 I was worried that my filesystem was corrupt som=
ehow, it<br>
had been up for 360 some days.=C2=A0 I umounted and fsck'd, not a singl=
e complaint.<br>
<br>
root@node1:/backuppc/cpool/0/<wbr>0/0# ls -al | awk ' { print $2 } '=
; | grep -v "1" |<br>
wc -l<br>
39<br>
<br>
I didn't have many with more than one link.=C2=A0 The entire dir is sma=
ll:<br>
root@node1:/backuppc/cpool/0/<wbr>0/0# du -hs .<br>
380K=C2=A0 =C2=A0 .<br>
<br>
I see similar elsewhere:<br>
root@node1:/backuppc/cpool/8/<wbr>8/8# ls -al |wc -l; ls -al | awk ' { =
print $2 } ' |<br>
grep -v "1" | wc -l<br>
59<br>
31<br>
<br>
(59 files, 31 with links).<br>
<br>
I'm still seeing crazy disk usage, over 3TB, only about 650GB (total fr=
om the<br>
host status "full size" column) visible to backuppc.<br>
<br>
Keep in mind this happened with no changes to the server.=C2=A0 30 hosts ba=
cked up<br>
for a week or so, then suddenly much more disk is used.=C2=A0 No host has l=
arger<br>
backups, just a factor of 6 larger pool one night.<br>
<br>
I was using the v3 to v4 migration script from git since it wasn't in t=
he<br>
release package yet (that's been fixed.<br>
<br>
I upgraded to backuppc 4.1.1, and the current versions of backuppc-xs and<b=
r>
rsync-bpc.=C2=A0 I ran the V3 to V4 migration script (now included in the r=
elease)<br>
again and it's doing some serious chewing (unlike before).=C2=A0 It use=
d to just fly<br>
through them all with "refCnt directory; skipping this backup".<b=
r>
<br>
So maybe this will fix it.<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br>
<br>
------------------------------<wbr>------------------------------<wbr>-----=
-------------<br>
Check out the vibrant tech community on one of the world's most<br>
engaging tech sites, Slashdot.org! <a href=3D"http://sdm.link/slashdot" rel=
=3D"noreferrer" target=3D"_blank">http://sdm.link/slashdot</a><br>
______________________________<wbr>_________________<br>
BackupPC-devel mailing list<br>
<a href=3D"mailto:[email protected]">BackupPC-devel@list=
s.<wbr>sourceforge.net</a><br>
List:=C2=A0 =C2=A0 <a href=3D"https://lists.sourceforge.net/lists/listinfo/=
backuppc-devel" rel=3D"noreferrer" target=3D"_blank">https://lists.sourcefo=
rge.net/<wbr>lists/listinfo/backuppc-devel</a><br>
Wiki:=C2=A0 =C2=A0 <a href=3D"http://backuppc.wiki.sourceforge.net" rel=3D"=
noreferrer" target=3D"_blank">http://backuppc.wiki.<wbr>sourceforge.net</a>=
<br>
Project: <a href=3D"http://backuppc.sourceforge.net/" rel=3D"noreferrer" ta=
rget=3D"_blank">http://backuppc.sourceforge.<wbr>net/</a><br>
</div></div></blockquote></div><br></div>
--001a113b0506fdf9ca054cab7054--
--===============1687539459299936972==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
--===============1687539459299936972==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
BackupPC-devel mailing list
[email protected]
List: https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
--===============1687539459299936972==--