Re: Suggestions for secondary cloud backups

--===============2265494324011618149==
Content-Type: multipart/alternative; boundary="000000000000f5ba8f0655cd8598"

--000000000000f5ba8f0655cd8598
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Sat, Jul 4, 2026 at 10:50=E2=80=AFAM Arno Lehmann via Bacula-users <
[email protected]> wrote:

> ...
>
> Keeping in mind that Pools are essentially for free, and Job definitions
> too, what keeps you from using Copy Jobs with the PoolUncopiedJobs
> selection scheme?


Phil said:

>
> ... I imagine it amounts to much the same thing.


I can think of exactly one scenario where a time-limited SQL query could be
better than the 'PoolUncopiedJobs' selection method. There could be other
advantages, but I am not aware of them.

If the local job has a longer retention period than the cloud job, you can
get to a state where nightly an old job is pruned from the cloud storage,
then the original local job is promptly selected for upload to the cloud
again, because it is a job in the pool which does not currently have a copy
in the cloud resource.

This could be even worse if the cloud storage uses any sort of object lock.

This error could be fairly obvious to detect in a simple misconfiguration
(local pool retention =3D 180 days, cloud pool retention =3D 90 days)

Such a problem could be far more tricky to detect if the local job
originally had a longer retention duration, local jobs were ran, but no
copies were made of those jobs. An admin (say, myself :eyeroll:) could run
a local full job while the job resource specifies the longer retention,
then lower the local job's retention to a shorter value before setting up
copy jobs / pools with matching retention. Because the copy pools will have
shorter retention than the instance of the local job which was ran early on
in the bacula installation's lifetime, the copies of the early local jobs
will be pruned from the cloud storage, but kept in the local storage. Every
time the copy control job is invoked while the longer-retention jobs exist
on the local storage but not on the cloud storage, the jobs will be
uploaded again. When the routine prune and truncate operations are ran
against the cloud storage, the cloud copies of these older jobs will be
deleted again, and the cycle begins anew. If object locking was enabled, as
was in my case, the cloud storage usage could balloon dramatically.

So for a copy job, it is very important that the retention periods used in
the copy pools match those specified in the local pools. Even if you are
not using object locking, this would be a waste of bandwidth and system
resources.

The main downside I can think of when using a time-bounded SQL query, is
the risk that something has gone wrong with cloud uploads for some reason,
and some older jobs which were never uploaded are not copied because the
SQL query excludes them because they are too old. This could break the
backup chain.

If using a time-limited SQL query like suggested by Phil, I do think it
could be wise to set up an admin job running an inverse query (select only
jobs which are OLDER than x days), and if any are found, report the job IDs
and exit 1 so the bacula admin job will show an error status to bring this
situation to an admin's attention.

I will reply with more information about my cloud copy jobs shortly.

Regards,
Robert Gerber
402-237-8692
[email protected]

--000000000000f5ba8f0655cd8598
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><div><br><div class=3D"gmail_quote gmail_=
quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Sat, Jul 4, 2026 =
at 10:50=E2=80=AFAM Arno Lehmann via Bacula-users &lt;<a href=3D"mailto:bac=
[email protected]">[email protected]</a>&gt;=
 wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px =
0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">...<br><=
br>
Keeping in mind that Pools are essentially for free, and Job definitions <b=
r>
too, what keeps you from using Copy Jobs with the PoolUncopiedJobs <br>sele=
ction scheme?</blockquote></div><br></div><div>Phil said:<br><div class=3D"=
gmail_quote gmail_quote_container"><blockquote class=3D"gmail_quote" style=
=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding=
-left:1ex"><br>... I imagine it amounts to much the same thing.</blockquote=
></div><br>I can think of exactly one scenario where a time-limited SQL que=
ry could be better than the &#39;PoolUncopiedJobs&#39; selection method. Th=
ere could be other advantages, but I am not aware of them.<br><br></div><di=
v>If the local job has a longer retention period than the cloud job, you ca=
n get to a state where nightly an old job is pruned from the cloud storage,=
 then the original local job is promptly selected for upload to the cloud a=
gain, because it is a job in the pool which does not currently have a copy =
in the cloud resource.<br><br></div><div>This could be even worse if the cl=
oud storage uses any sort of object lock.</div><div><br></div><div>This err=
or could=C2=A0be fairly obvious to detect in a simple misconfiguration (loc=
al pool retention =3D 180 days, cloud pool retention =3D 90 days)<br><br>Su=
ch a problem could be far more tricky to detect if the local job originally=
 had a longer retention duration, local jobs were ran, but no copies were m=
ade of those jobs. An admin (say, myself :eyeroll:) could run a local full =
job while the job resource specifies the longer retention, then lower the l=
ocal job&#39;s retention to a shorter value before setting up copy jobs / p=
ools with matching retention. Because the copy pools will have shorter rete=
ntion than the instance of the local job which was ran early on in the bacu=
la installation&#39;s lifetime, the copies of the early local jobs will be =
pruned from the cloud storage, but kept in the local storage. Every time th=
e copy control job is invoked while the longer-retention jobs exist on the =
local storage but not on the cloud storage, the jobs will be uploaded again=
. When the routine prune and truncate operations are ran against the cloud =
storage, the cloud copies of these older jobs will be deleted again, and th=
e cycle begins anew. If object locking was enabled, as was in my case, the =
cloud storage usage could balloon dramatically.</div><div><br></div><div>So=
 for a copy job, it is very important that the retention periods used in th=
e copy pools match those specified in the local pools. Even if you are not =
using object locking, this would be a waste of bandwidth and system resourc=
es.</div><div><br></div><div>The main downside I can think of when using a =
time-bounded SQL query, is the risk that something has gone wrong with clou=
d uploads for some reason, and some older jobs which were never uploaded ar=
e not copied because the SQL query excludes them because they are too old. =
This could break the backup chain.</div><div><br></div><div>If using a time=
-limited SQL query like suggested by Phil, I do think it could=C2=A0be wise=
 to set up an admin job running an inverse query (select only jobs which ar=
e OLDER than x days), and if any are found, report the job IDs and exit 1 s=
o the bacula admin job will show an error status to bring this situation to=
 an admin&#39;s attention.=C2=A0</div><div><br></div><div><div dir=3D"ltr" =
class=3D"gmail_signature"><div>I will reply with more information about my =
cloud copy jobs shortly.</div><div><br>Regards,</div><div>Robert Gerber</di=
v><div>402-237-8692</div><div><a href=3D"mailto:[email protected]" target=3D"_=
blank">[email protected]</a></div></div></div><br></div><br></div>

--000000000000f5ba8f0655cd8598--


--===============2265494324011618149==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline


--===============2265494324011618149==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users

--===============2265494324011618149==--