Re: Suggestions for secondary cloud backups
Rob Gerber <[email protected]> Sat, 4 Jul 2026 18:41:43 -0500
| Newsgroups | gmane.comp.bacula.user |
|---|---|
| Message-ID | <CAMKi9mSTewmJgsBOBtUDAs3HHJ6PiPg12xmmQYdJtjR_H5Xc+w@mail.gmail.com> |
--===============8568782620887316702==
Content-Type: multipart/alternative; boundary="000000000000ad48ea0655d197bf"
--000000000000ad48ea0655d197bf
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
message continues:
###########################
bacula-sd.conf excerpts:
I am using object locking, to prevent the deletion of cloud-based volumes
before their retention period has expired.
This is for ransomware mitigation. Implementation varies between cloud
providers, but generally it is much more difficult to delete an object
locked object from the cloud resource.
An object which is older than the object lock period can be deleted as
normal.
Bacula actually has no idea that I am using object locking. The easiest way
I found to set object locks was on a per-bucket basis.
I have to make sure that my Full, Diff, and Inc rentention periods align
with my bucket object lock durations.
Full, Diff, and Inc buckets each have their own object lock durations, to
match their pool retention periods.
For each bucket, bacula-sd.conf must have a cloud resource, autochanger,
and device resources dedicated to that bucket.
Additionally, Backblaze has an option to keep all versions of a file. I do
not want this, so I have configured it to only keep the latest version of
the file.
If an object lock applies to a file, Backblaze will still keep multiple
versions of the file.
If an object locked volume part is deleted by bacula, Backblaze will tell
bacula that the volume was deleted. The Backblaze bucket file browser will
still show you that a 0B volume part file has overlaid on the original part
file. Both files are accessible from the bucket file browser.
I don't know how other cloud storage providers handle these concepts, but I
assume they are likely present, and should be understood.
I only have a single local autochanger definition, and a number of device
resources dedicated to the local autochanger.
In other words, the local storage resources don't have devices and
autochangers dedicated to different job levels.
###########################
Cloud {
Name =3D "B2-TD-Full"
Driver =3D "Amazon" # this is the new cloud driver, available as of bacu=
la
CE 15.x. It is much more reliable than the old S3 driver.
HostName =3D "s3.us-west-004.backblazeb2.com"
BlobEndpoint =3D "https://s3.us-west-004.backblazeb2.com/" # this is
essential for non-AWS cloud providers
BucketName =3D "TD-Bacula-Full"
AccessKey =3D "REDACTED"
SecretKey =3D "REDACTED"
Protocol =3D "HTTPS"
UriStyle =3D "VirtualHost"
TruncateCache =3D "AfterUpload" # All local copies of the cloud volume
parts (except the header, *.part1) will be deleted after upload.
Upload =3D "No" # Copy volumes will not be uploaded after a job complete=
s.
Upload must be launched by a separate admin script, which runs a bconsole
command.
}
Cloud {
Name =3D "B2-TD-Diff"
...
BucketName =3D "TD-Bacula-Diff"
}
Cloud {
Name =3D "B2-TD-Inc"
...
BucketName =3D "TD-Bacula-Inc"
}
Autochanger {
Name =3D "B2-TD-Full"
Device =3D "B2-TD-Full-001"
Device =3D "B2-TD-Full-002"
Device =3D "B2-TD-Full-003"
Device =3D "B2-TD-Full-004"
Device =3D "B2-TD-Full-005"
Device =3D "B2-TD-Full-006"
ChangerDevice =3D "/dev/null"
ChangerCommand =3D "/dev/null"
}
Autochanger {
Name =3D "B2-TD-Diff"
...
}
Autochanger {
Name =3D "B2-TD-Inc"
...
}
Device {
Name =3D "B2-TD-Full-001"
Description =3D "B2-TD-Full"
MediaType =3D "B2-TD-Full"
DeviceType =3D "Cloud"
ArchiveDevice =3D "/mnt/synology/bacula/B2-TD-Full"
RemovableMedia =3D no
RandomAccess =3D yes
AutomaticMount =3D yes
LabelMedia =3D yes
Autochanger =3D yes
ReadOnly =3D no
MaximumFileSize =3D 5368709120
MaximumConcurrentJobs =3D 1
DriveIndex =3D 0
MaximumPartSize =3D 5368709120
Cloud =3D "B2-TD-Full"
VolumeEncryption =3D yes # Remove this line if you haven't set up
volume encryption. Only available in bacula CE 15.x and beyond.
}
Device {
Name =3D "B2-TD-Full-002"
...
}
Device {
Name =3D "B2-TD-Full-003"
...
}
Device {
Name =3D "B2-TD-Full-004"
...
}
Device {
Name =3D "B2-TD-Full-005"
...
}
Device {
Name =3D "B2-TD-Full-006"
...
AutoSelect =3D no # this drive will not be used by backups so it will
remain available for restores.
}
# (devices for the diff and inc storage resources not shown, but they must
exist and are the same as the Full devices, except for the level names)
#####################
bacula-sd.conf snippet ends
#####################
Please see next message.
Regards,
Robert Gerber
402-237-8692
[email protected]
On Sat, Jul 4, 2026 at 6:40=E2=80=AFPM Rob Gerber <[email protected]> wrote:
> Andrea,
> Sorry for the delay in reply. I'm going to have to post this in multiple
> pieces due to message length limits.
>
> One of my bacula installations runs on a synology appliance and uploads t=
o
> the Backblaze B2, an S3 compatible cloud service. You should be able to u=
se
> any S3 compatible cloud service.
> I separate out different job levels into their own pool for the local
> storage. This structure is mirrored in the cloud. I have a bucket for eac=
h
> cloud pool. Each cloud pool must have its own autochanger and device
> resources in bacula-sd.conf, and its own storage resource in
> bacula-dir.conf. Each local pool has an equivalent cloud pool. The local
> pools reference the cloud pools using 'NextPool=3D'.I use selection metho=
d
> PoolUncopiedJobs to choose which jobs are uploaded. I am using bacula 15.=
x,
> and the amazon cloud driver and volume encryption introduced in that
> version of bacula. You do not need to use volume encryption if you are no=
t
> concerned about this, but if you do you should know that each volume is
> encrypted with a unique key. If these keys are not backed up separately,
> recovery from your cloud volumes alone will be impossible.
>
> I delete all local copies of the copy jobs after upload. I do not upload
> at the time when the job is ran. I upload afterward with a copy volume pa=
rt
> sweeper script. I previously uploaded at time of copy job execution, but =
I
> found that bacula does not retry failed part uploads during that process,
> and bacula also does not throw an error when a part fails to upload.
> Without a part sweeper script to instruct bconsole to upload any cloud
> volume parts not previously uploaded, any such part would simply be left
> unuploaded forever. My part sweeper script also outputs to a log file, th=
en
> reads its own log to check for errors. The part sweeper script exits 1 on=
ce
> it's done all its work if it finds any failure to upload a cloud volume
> part.
>
> My basic psuedocode config structure (real examples further down):
> bacula-sd.conf:
> Cloud resource for bucket Cloud-Full (TruncateCache =3D "AfterUpload";
> Upload =3D "no")
> Cloud resource for bucket Cloud-Diff (TruncateCache =3D "AfterUpload";
> Upload =3D "no")
> Cloud resource for bucket Cloud-Inc (TruncateCache =3D "AfterUpload";
> Upload =3D "no")
> Autochanger resource for Cloud resource Cloud-Full
> Autochanger resource for Cloud resource Cloud-Diff
> Autochanger resource for Cloud resource Cloud-Inc
> Device resources for Cloud resource Cloud-Full
> Device resources for Cloud resource Cloud-Diff
> Device resources for Cloud resource Cloud-Inc
>
> Autochanger resource for local file based storage
> Device resources for local file based storage
>
> bacula-dir.conf:
> Storage for Cloud-Full
> Storage for Cloud-Diff
> Storage for Cloud-Inc
> Storage for local file based storage
> Pool Local-Full (MaximumVolumeJobs =3D 1; NextPool references the equival=
ent
> cloud pool)
> Pool Local-Diff (MaximumVolumeJobs =3D 1; NextPool references the equival=
ent
> cloud pool)
> Pool Local-Inc (MaximumVolumeJobs =3D 1; NextPool references the equival=
ent
> cloud pool)
> Pool Cloud-Full (MaximumVolumeJobs =3D 1)
> Pool Cloud-Diff (MaximumVolumeJobs =3D 1)
> Pool Cloud-Inc (MaximumVolumeJobs =3D 1)
> Schedule: Daily at 23:05. Used with backup jobs. Jobdefs selects the
> correct level.
> Schedule: Copy-End-Of-Day at 23:50. Copy jobs. Doesn't actually need to b=
e
> separate at this point. Priority ensures that this will run at the correc=
t
> point in the process.
> Schedule: NightlyAfterCopy at 23:54. Used to launch the part sweeper afte=
r
> copy jobs are finished. Not actually needed - priority ensures this will
> run at the correct point in the process.
> Jobdefs: Specifies local full, diff, inc pools; maximum interval between
> levels full and diff; no duplicate jobs allowed; ReRunFailedLevels =3D ye=
s
> admin-prune-allpools-job: Priority 1. Script with bconsole command to
> prune all eligible jobs.
> admin-truncate-volumes-job: Priority 2. Script with bconsole commands to
> truncate all pools on each storage. Minimizes cloud storage usage.
> Job 1
> Job 2
> admin-copy-control-launcher-job (lower priority than main backup jobs;
> launches the copy jobs after main jobs have finished. Otherwise the copy
> jobs will only queue up yesterdays jobs for copying, because todays backu=
p
> jobs have not finished at the time when the copy jobs were launched)
> Copy-Control-Full-job (one copy control job per pool. Manual schedule.
> Targets local pool for each level. Copy control job for each level preven=
ts
> cancellation of copy control jobs as duplicate)
> Copy-Control-Diff-job (one copy control job per pool. Manual schedule.
> Targets local pool for each level. Copy control job for each level preven=
ts
> cancellation of copy control jobs as duplicate)
> Copy-Control-Inc-job (one copy control job per pool. Manual schedule.
> Targets local pool for each level. Copy control job for each level preven=
ts
> cancellation of copy control jobs as duplicate)
> admin-cloud-volume-part-sweeper-job (Script with bconsole commands to
> verify all cloud volume parts have been uploaded. Also responsible to
> perform initial upload of copy job volumes (SD cloud resource specifies
> "Upload =3D no"))
>
> Message continues.
>
> Regards,
> Robert Gerber
> 402-237-8692
> [email protected]
>
>
>
--000000000000ad48ea0655d197bf
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div>message continues:<br><br><div>######################=
#####<br>bacula-sd.conf excerpts:<br><br></div><div>I am using object locki=
ng, to prevent the deletion of cloud-based volumes before their retention p=
eriod has expired.<br>This
is for ransomware mitigation. Implementation varies between cloud=20
providers, but generally it is much more difficult to delete an object=20
locked object from the cloud resource. <br>An object which is older than th=
e object lock period can be deleted as normal.<br>Bacula
actually has no idea that I am using object locking. The easiest way I=20
found to set object locks was on a per-bucket basis.<br>I have to make sure=
that my Full, Diff, and Inc rentention periods align with my bucket object=
lock durations.<br>Full, Diff, and Inc buckets each have their own object =
lock durations, to match their pool retention periods. <br>For each bucket,=
bacula-sd.conf must have a cloud resource, autochanger, and device resourc=
es dedicated to that bucket.<br>Additionally,
Backblaze has an option to keep all versions of a file. I do not want=20
this, so I have configured it to only keep the latest version of the=20
file. <br>If an object lock applies to a file, Backblaze will still keep mu=
ltiple versions of the file.<br>If
an object locked volume part is deleted by bacula, Backblaze will tell=20
bacula that the volume was deleted. The Backblaze bucket file browser=20
will still show you that a 0B volume part file has overlaid on the=20
original part file. Both files are accessible from the bucket file=20
browser.<br>I don't know how other cloud storage providers handle these=
=20
concepts, but I assume they are likely present, and should be=20
understood.<br><br></div><div><br></div><div>I only have a single local aut=
ochanger definition, and a number of device resources dedicated to the loca=
l autochanger. <br>In other words, the local storage resources don't ha=
ve devices and autochangers dedicated to different job levels.<br>#########=
##################<br><br></div><div>Cloud {<br>=C2=A0 Name =3D "B2-TD=
-Full"<br>=C2=A0
Driver =3D "Amazon" =C2=A0# this is the new cloud driver, availa=
ble as of=20
bacula CE 15.x. It is much more reliable than the old S3 driver.<br>=C2=A0 =
HostName =3D "<a href=3D"http://s3.us-west-004.backblazeb2.com" target=
=3D"_blank">s3.us-west-004.backblazeb2.com</a>"<br>=C2=A0 BlobEndpoint=
=3D "<a href=3D"https://s3.us-west-004.backblazeb2.com/" target=3D"_b=
lank">https://s3.us-west-004.backblazeb2.com/</a>" # this is essential=
for non-AWS cloud providers<br>=C2=A0 BucketName =3D "TD-Bacula-Full&=
quot;<br>=C2=A0 AccessKey =3D "REDACTED"<br>=C2=A0 SecretKey =3D =
"REDACTED"<br>=C2=A0 Protocol =3D "HTTPS"<br>=C2=A0 Uri=
Style =3D "VirtualHost"<br>=C2=A0
TruncateCache =3D "AfterUpload" =C2=A0# All local copies of the =
cloud volume=20
parts (except the header, *.part1) will be deleted after upload.<br>=C2=A0=
=20
Upload =3D "No" =C2=A0# Copy volumes will not be uploaded after a=
job=20
completes. Upload must be launched by a separate admin script, which=20
runs a bconsole command.<br>}<br>Cloud {<br>=C2=A0 Name =3D "B2-TD-Dif=
f"<br>=C2=A0 ...<br>=C2=A0 BucketName =3D "TD-Bacula-Diff"<b=
r>=C2=A0 }<br>Cloud {<br>=C2=A0 Name =3D "B2-TD-Inc"<br>=C2=A0 ..=
.<br>=C2=A0 BucketName =3D "TD-Bacula-Inc"<br>}<br><br><br>Autoch=
anger {<br>=C2=A0 Name =3D "B2-TD-Full"<br>=C2=A0 Device =3D &quo=
t;B2-TD-Full-001"<br>=C2=A0 Device =3D "B2-TD-Full-002"<br>=
=C2=A0 Device =3D "B2-TD-Full-003"<br>=C2=A0 Device =3D "B2-=
TD-Full-004"<br>=C2=A0 Device =3D "B2-TD-Full-005"<br>=C2=A0=
Device =3D "B2-TD-Full-006"<br>=C2=A0 ChangerDevice =3D "/d=
ev/null"<br>=C2=A0 ChangerCommand =3D "/dev/null"<br>}<br>Au=
tochanger {<br>=C2=A0 Name =3D "B2-TD-Diff"<br>=C2=A0 ...<br>}<br=
>Autochanger {<br>=C2=A0 Name =3D "B2-TD-Inc"<br>=C2=A0 ...<br>}<=
br><br><br>Device {<br>=C2=A0 Name =3D "B2-TD-Full-001"<br>=C2=A0=
Description =3D "B2-TD-Full"<br>=C2=A0 MediaType =3D "B2-TD=
-Full"<br>=C2=A0 DeviceType =3D "Cloud"<br>=C2=A0 ArchiveDev=
ice =3D "/mnt/synology/bacula/B2-TD-Full"<br>=C2=A0 RemovableMedi=
a =3D no<br>=C2=A0 RandomAccess =3D yes<br>=C2=A0 AutomaticMount =3D yes<br=
>=C2=A0 LabelMedia =3D yes<br>=C2=A0 Autochanger =3D yes<br>=C2=A0 ReadOnly=
=3D no<br>=C2=A0 MaximumFileSize =3D 5368709120<br>=C2=A0 MaximumConcurren=
tJobs =3D 1<br>=C2=A0 DriveIndex =3D 0<br>=C2=A0 MaximumPartSize =3D 536870=
9120<br>=C2=A0 Cloud =3D "B2-TD-Full"<br>=C2=A0
VolumeEncryption =3D yes =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Remove this line if=
you haven't set up
volume encryption. Only available in bacula CE 15.x and beyond.<br>}<br>De=
vice {<br>=C2=A0 Name =3D "B2-TD-Full-002"<br>=C2=A0 ...<br>}<br>=
Device {<br>=C2=A0 Name =3D "B2-TD-Full-003"<br>=C2=A0 ...<br>}<b=
r>Device {<br>=C2=A0 Name =3D "B2-TD-Full-004"<br>=C2=A0 ...<br>}=
<br>Device {<br>=C2=A0 Name =3D "B2-TD-Full-005"<br>=C2=A0 ...<br=
>}<br>Device {<br>=C2=A0 Name =3D "B2-TD-Full-006"<br>=C2=A0 ...<=
br>=C2=A0 AutoSelect =3D no # this drive will not be used by backups so it =
will remain available for restores.<br>}<br><br>#
(devices for the diff and inc storage resources not shown, but they=20
must exist and are the same as the Full devices, except for the level=20
names)<br><br>#####################<br>bacula-sd.conf snippet ends<br>#####=
################</div><div><br></div><div>Please see next message.</div><br=
></div><div><div dir=3D"ltr" class=3D"gmail_signature" data-smartmail=3D"gm=
ail_signature"><div>Regards,</div><div>Robert Gerber</div><div>402-237-8692=
</div><div><a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]=
et</a></div></div></div><br></div><br><div class=3D"gmail_quote gmail_quote=
_container"><div dir=3D"ltr" class=3D"gmail_attr">On Sat, Jul 4, 2026 at 6:=
40=E2=80=AFPM Rob Gerber <<a href=3D"mailto:[email protected]">[email protected]=
et</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margi=
n:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex=
"><div dir=3D"ltr"><div dir=3D"ltr"><div>Andrea,<br>Sorry for the delay in =
reply. I'm going to have to post this in multiple pieces due to message=
length limits.<br><br>One of my bacula=20
installations runs on a synology appliance and uploads to the Backblaze=20
B2, an S3 compatible cloud service. You should be able to use any S3=20
compatible cloud service.<br>I separate out different job levels into=20
their own pool for the local storage. This structure is mirrored in the=20
cloud. I have a bucket for each cloud pool. Each cloud pool must have=20
its own autochanger and device resources in bacula-sd.conf, and its own=20
storage resource in bacula-dir.conf. Each local pool has an equivalent=20
cloud pool. The local pools reference the cloud pools using=20
'NextPool=3D'.I use selection method PoolUncopiedJobs to choose whi=
ch jobs
are uploaded. I am using bacula 15.x, and the amazon cloud driver and=20
volume encryption introduced in that version of bacula. You do not need=20
to use volume encryption if you are not concerned about this, but if you
do you should know that each volume is encrypted with a unique key. If=20
these keys are not backed up separately, recovery from your cloud=20
volumes alone will be impossible. <br><br>I delete all local copies of=20
the copy jobs after upload. I do not upload at the time when the job is=20
ran. I upload afterward with a copy volume part sweeper script. I=20
previously uploaded at time of copy job execution, but I found that=20
bacula does not retry failed part uploads during that process, and=20
bacula also does not throw an error when a part fails to upload. Without
a part sweeper script to instruct bconsole to upload any cloud volume=20
parts not previously uploaded, any such part would simply be left=20
unuploaded forever. My part sweeper script also outputs to a log file,=20
then reads its own log to check for errors. The part sweeper script=20
exits 1 once it's done all its work if it finds any failure to upload a=
=20
cloud volume part.<br><br>My basic psuedocode config structure (real exampl=
es further down):<br>bacula-sd.conf:<br>Cloud resource for bucket Cloud-Ful=
l (TruncateCache =3D "AfterUpload"; Upload =3D "no")<br=
>Cloud resource for bucket Cloud-Diff (TruncateCache =3D "AfterUpload&=
quot;; Upload =3D "no")<br>Cloud resource for bucket Cloud-Inc =
=C2=A0(TruncateCache =3D "AfterUpload"; Upload =3D "no"=
)<br>Autochanger resource for Cloud resource Cloud-Full<br>Autochanger reso=
urce for Cloud resource Cloud-Diff<br>Autochanger resource for Cloud resour=
ce Cloud-Inc<br>Device resources for Cloud resource Cloud-Full<br>Device re=
sources for Cloud resource Cloud-Diff<br>Device resources for Cloud resourc=
e Cloud-Inc<br><br>Autochanger resource for local file based storage<br>Dev=
ice resources for local file based storage<br><br>bacula-dir.conf:<br>Stora=
ge for Cloud-Full<br>Storage for Cloud-Diff<br>Storage for Cloud-Inc<br>Sto=
rage for local file based storage<br>Pool Local-Full (MaximumVolumeJobs =3D=
1; NextPool references the equivalent cloud pool) =C2=A0 =C2=A0 <br>Pool L=
ocal-Diff (MaximumVolumeJobs =3D 1; NextPool references the equivalent clou=
d pool)<br>Pool Local-Inc =C2=A0(MaximumVolumeJobs =3D 1; NextPool referenc=
es the equivalent cloud pool)<br>Pool Cloud-Full (MaximumVolumeJobs =3D 1)<=
br>Pool Cloud-Diff (MaximumVolumeJobs =3D 1)<br>Pool Cloud-Inc =C2=A0(Maxim=
umVolumeJobs =3D 1)<br>Schedule: Daily at 23:05. Used with backup jobs. Job=
defs selects the correct level.<br>Schedule:
Copy-End-Of-Day at 23:50. Copy jobs. Doesn't actually need to be=20
separate at this point. Priority ensures that this will run at the=20
correct point in the process.<br>Schedule: NightlyAfterCopy at 23:54.=20
Used to launch the part sweeper after copy jobs are finished. Not=20
actually needed - priority ensures this will run at the correct point in
the process.<br>Jobdefs: Specifies local full, diff, inc pools; maximum
interval between levels full and diff; no duplicate jobs allowed;=20
ReRunFailedLevels =3D yes<br>admin-prune-allpools-job: Priority 1. Script w=
ith bconsole command to prune all eligible jobs.<br>admin-truncate-volumes-=
job:
Priority 2. Script with bconsole commands to truncate all pools on each
storage. Minimizes cloud storage usage. <br>Job 1<br>Job 2<br>admin-copy-c=
ontrol-launcher-job
(lower priority than main backup jobs; launches the copy jobs after=20
main jobs have finished. Otherwise the copy jobs will only queue up=20
yesterdays jobs for copying, because todays backup jobs have not=20
finished at the time when the copy jobs were launched)<br>Copy-Control-Full=
-job
(one copy control job per pool. Manual schedule. Targets local pool for
each level. Copy control job for each level prevents cancellation of=20
copy control jobs as duplicate)<br>Copy-Control-Diff-job (one copy=20
control job per pool. Manual schedule. Targets local pool for each=20
level. Copy control job for each level prevents cancellation of copy=20
control jobs as duplicate)<br>Copy-Control-Inc-job =C2=A0(one copy control=
=20
job per pool. Manual schedule. Targets local pool for each level. Copy=20
control job for each level prevents cancellation of copy control jobs as
duplicate)<br>admin-cloud-volume-part-sweeper-job (Script with=20
bconsole commands to verify all cloud volume parts have been uploaded.=20
Also responsible to perform initial upload of copy job volumes (SD cloud
resource specifies "Upload =3D no"))<br><br></div><div>Message c=
ontinues.</div><div><br></div><div><div dir=3D"ltr" class=3D"gmail_signatur=
e"><div>Regards,</div><div>Robert Gerber</div><div>402-237-8692</div><div><=
a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a></div>=
</div></div><br></div><br></div>
</blockquote></div>
--000000000000ad48ea0655d197bf--
--===============8568782620887316702==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
--===============8568782620887316702==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users
--===============8568782620887316702==--