Re: Separating frequently-accessed LOCK and LOG* from $topDir/pc/$host

--===============8901859843725145618==
Content-Type: multipart/alternative; boundary="001a1140434a9778f705610f843f"

--001a1140434a9778f705610f843f
Content-Type: text/plain; charset="UTF-8"

Alex,

Thanks for the offer to prepare a pull request, but it doesn't seem worth
making the change.

Have you considered simply reducing your $Conf{WakeupSchedule} so your disk
drive wakes up less often?

I made the DDMMYYYY change in the docs.  Thanks.

Craig

On Tue, Dec 5, 2017 at 1:33 AM, Alexander Kobel <[email protected]> wrote:

> Hi,
>
> for a tiny BackupPC instance with on an SSD+HDD-server, I want the HDD to
> be in standby mode (spinned down) most of the time.  Hence, I only store
> __TOPDIR__ on it, and have everything else on the SSD. This almost works,
> except for the fact that unsuccessful pings write a log entry in
> __TOPDIR__/pc/$host/LOG.DDMMYYYY, and LOCK is touched.  Which happens on
> every wakeup when some client is not reachable; in other words, too often.
>
> AFAICS, from what's inside pc/, actually vital for the integrity of the
> backups are the nnn and refCnt dirs (and, to some extent, the backups and
> config.pl files).  Those are only accessed during backup or restore jobs,
> when the "vault" disk is active anyway.
> So, in short, I want to bring __TOPDIR__/pc/$host/LOG* and LOCK from the
> HDD to somewhere else.  Note that both LOG* and nnn are moving targets, so
> symlinks only work to a certain extent.
> (By the way: LOG.DDMMYYYY is actually LOG.MMYYYY, without DD, on my 4.1.3
> installation.)
>
>
> My usual approach is anything-sync-daemon (which uses an overlayfs to move
> some directory to tmpfs, and sync backs periodically and/or on user
> request).  However, this only works at directory level, and would require
> moving the entire __TOPDIR__/pc directory to tmpfs.  nnn can easily become
> large, and overlayfs is ill-suited for that.
> Another approach would be to move __TOPDIR__/pc to the SSD and rsync back
> each __TOPDIR__/pc/$host/ to the HDD in a DumpPostUserCmd and, perhaps,
> after each nightly.  That's better, but again, the nnn can grow large (at
> least in number of files) and might become ill-suited even for the SSD
> and/or rootfs.
>
>
> => Question 1.) Short of patching BackupPC, does anyone have a suggestion
> or alternate solution I didn't think of?
>
>
> It would be way easier to properly separate the two kinds of data if the
> location of the per-host LOG* and LOCK files would be configurable, e.g. to
> somewhere below __LOGDIR__.  This would match the logic for the global LOCK
> (which resides at __LOGDIR__), even though I'd typically expect lock files
> to live below /run/lock or /var/run/lock.
> Full backward compatibility could be kept, e.g., through a
> __PERHOSTLOGDIR__ variable defaulting to __TOPDIR__/pc/.
>
> I'd be willing to look into preparing a patch and pull request, if there
> is any interest.  But I understand that the scenario and motivation are
> negligible in an enterprise setup, and that changing the storage layout can
> a royal pain in the * for existing systems.  Hence,
>
>
> => Question 2.) Would such a change have any chance to be integrated in
> BackupPC?  If so, under what constraints for the interface?  Any
> recommendations for locations?  What about {Xfer,Restore,Archive}{ERR,LOG
> ,Info}*?
>
>
>
> Best,
> Alex
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> BackupPC-devel mailing list
> [email protected]
> List:    https://lists.sourceforge.net/lists/listinfo/backuppc-devel
> Wiki:    http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
>

--001a1140434a9778f705610f843f
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Alex,<div><br></div><div>Thanks for the offer to prepare a=
 pull request, but it doesn&#39;t seem worth making the change.</div><div><=
br></div><div>Have you considered simply reducing your=C2=A0$Conf{WakeupSch=
edule} so your disk drive wakes up less often?</div><div><br></div><div>I m=
ade the DDMMYYYY change in the docs.=C2=A0 Thanks.</div><div><br></div><div=
>Craig</div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote"=
>On Tue, Dec 5, 2017 at 1:33 AM, Alexander Kobel <span dir=3D"ltr">&lt;<a h=
ref=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>&=
gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 =
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
for a tiny BackupPC instance with on an SSD+HDD-server, I want the HDD to b=
e in standby mode (spinned down) most of the time.=C2=A0 Hence, I only stor=
e __TOPDIR__ on it, and have everything else on the SSD. This almost works,=
 except for the fact that unsuccessful pings write a log entry in __TOPDIR_=
_/pc/$host/LOG.DDMMYY<wbr>YY, and LOCK is touched.=C2=A0 Which happens on e=
very wakeup when some client is not reachable; in other words, too often.<b=
r>
<br>
AFAICS, from what&#39;s inside pc/, actually vital for the integrity of the=
 backups are the nnn and refCnt dirs (and, to some extent, the backups and =
<a href=3D"http://config.pl" rel=3D"noreferrer" target=3D"_blank">config.pl=
</a> files).=C2=A0 Those are only accessed during backup or restore jobs, w=
hen the &quot;vault&quot; disk is active anyway.<br>
So, in short, I want to bring __TOPDIR__/pc/$host/LOG* and LOCK from the HD=
D to somewhere else.=C2=A0 Note that both LOG* and nnn are moving targets, =
so symlinks only work to a certain extent.<br>
(By the way: LOG.DDMMYYYY is actually LOG.MMYYYY, without DD, on my 4.1.3 i=
nstallation.)<br>
<br>
<br>
My usual approach is anything-sync-daemon (which uses an overlayfs to move =
some directory to tmpfs, and sync backs periodically and/or on user request=
).=C2=A0 However, this only works at directory level, and would require mov=
ing the entire __TOPDIR__/pc directory to tmpfs.=C2=A0 nnn can easily becom=
e large, and overlayfs is ill-suited for that.<br>
Another approach would be to move __TOPDIR__/pc to the SSD and rsync back e=
ach __TOPDIR__/pc/$host/ to the HDD in a DumpPostUserCmd and, perhaps, afte=
r each nightly.=C2=A0 That&#39;s better, but again, the nnn can grow large =
(at least in number of files) and might become ill-suited even for the SSD =
and/or rootfs.<br>
<br>
<br>
=3D&gt; Question 1.) Short of patching BackupPC, does anyone have a suggest=
ion or alternate solution I didn&#39;t think of?<br>
<br>
<br>
It would be way easier to properly separate the two kinds of data if the lo=
cation of the per-host LOG* and LOCK files would be configurable, e.g. to s=
omewhere below __LOGDIR__.=C2=A0 This would match the logic for the global =
LOCK (which resides at __LOGDIR__), even though I&#39;d typically expect lo=
ck files to live below /run/lock or /var/run/lock.<br>
Full backward compatibility could be kept, e.g., through a __PERHOSTLOGDIR_=
_ variable defaulting to __TOPDIR__/pc/.<br>
<br>
I&#39;d be willing to look into preparing a patch and pull request, if ther=
e is any interest.=C2=A0 But I understand that the scenario and motivation =
are negligible in an enterprise setup, and that changing the storage layout=
 can a royal pain in the * for existing systems.=C2=A0 Hence,<br>
<br>
<br>
=3D&gt; Question 2.) Would such a change have any chance to be integrated i=
n BackupPC?=C2=A0 If so, under what constraints for the interface?=C2=A0 An=
y recommendations for locations?=C2=A0 What about {Xfer,Restore,Archive}{ER=
R,LOG<wbr>,Info}*?<br>
<br>
<br>
<br>
Best,<br>
Alex<br>
<br>
<br>------------------------------<wbr>------------------------------<wbr>-=
-----------------<br>
Check out the vibrant tech community on one of the world&#39;s most<br>
engaging tech sites, Slashdot.org! <a href=3D"http://sdm.link/slashdot" rel=
=3D"noreferrer" target=3D"_blank">http://sdm.link/slashdot</a><br>_________=
_____________________<wbr>_________________<br>
BackupPC-devel mailing list<br>
<a href=3D"mailto:[email protected]">BackupPC-devel@list=
s.<wbr>sourceforge.net</a><br>
List:=C2=A0 =C2=A0 <a href=3D"https://lists.sourceforge.net/lists/listinfo/=
backuppc-devel" rel=3D"noreferrer" target=3D"_blank">https://lists.sourcefo=
rge.net/<wbr>lists/listinfo/backuppc-devel</a><br>
Wiki:=C2=A0 =C2=A0 <a href=3D"http://backuppc.wiki.sourceforge.net" rel=3D"=
noreferrer" target=3D"_blank">http://backuppc.wiki.<wbr>sourceforge.net</a>=
<br>
Project: <a href=3D"http://backuppc.sourceforge.net/" rel=3D"noreferrer" ta=
rget=3D"_blank">http://backuppc.sourceforge.<wbr>net/</a><br>
<br></blockquote></div><br></div>

--001a1140434a9778f705610f843f--


--===============8901859843725145618==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
--===============8901859843725145618==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
BackupPC-devel mailing list
[email protected]
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

--===============8901859843725145618==--