Parallelizing Backup, Copy, and Restore for the Same Client (Best Practices?)

Hi all,

I have a client with ~2TB of data. A full backup job takes about 15 hours.
Once the backup finishes, a Copy Job is triggered (Selection Type =
PoolUncopiedJobs) to replicate the volume to a remote Storage Daemon —
which takes another 15 hours per volume in the best-case scenario. Finally,
for each backup, I run a Restore job on a replica server (same client
setup), which adds yet another 15 hours.

The trivial approach to parallelize these jobs (Backup, Copy, Restore) has
been to define a separate Storage + Device pair for each Job. The only
difference between them is the Name. All Devices share the same Archive
Device path — which I'm unsure is a good practice.

Is there a better way to parallelize Backup, Copy, and Restore jobs on the
same Pool/Volumes without defining a separate Storage/Device for each Job?
What would be considered a best-practice setup here?

TL;DR
I need to run Backup, Copy, and Restore jobs in parallel for the same
client (Restore happens on a replica server). Ideally without creating
separate Storage/Device/Pool for each job.

Bacula Version: 15.0.2 (21 March 2024)

This is the config that currently allows me to parallelize only Backup and
Restore. Following this logic, I’d need to create additional Storage +
Device just for the Copy jobs.

Client {
    Name = "mongodb01-fd"
    Address = xx.xx.xx.xx
    FDPort = 9102
    Catalog = MyCatalog
    Password = "PASSWORD"
    File Retention = 11 days
    Job Retention = 2 months
    AutoPrune = yes
    Heartbeat Interval = 1m
    TLS Require = yes
    TLS PSK Enable = yes
}

Job {
    Name = "mongodb01-Backup"
    JobDefs = "MongoDef"
    Pool = "mongodb01-baculaserver01-Pool"
    Level = Full
    Client = "mongodb01-fd"
    Storage = "mongodb01_baculaserver01_Storage"
    Schedule = "DisabledSchedule"  # This is how I disable this client so
it doesn't affect others using the same JobDef with valid Schedules. I
launch the backup manually using a bconsole command.
    @"/opt/bacula/etc/bacula-dir.conf.d/jobdefs/RunCopyToServer.jd"  # This
is how I trigger Copy Jobs for clients with small backups. It works well
within a 24-hour window. For this client, I should remove this line.
}

Pool {
    Name = "mongodb01-baculaserver01-Pool"
    Use Volume Once = yes
    Pool Type = Backup
    LabelFormat = "mongodb01-"
    AutoPrune = yes
    Recycle = yes
    VolumeRetention = 30 days
    Maximum Volumes = 365
    Maximum Volume Jobs = 1
    Recycle Oldest Volume = yes
    Next Pool = "mongodb01-baculaserver02-Pool"  # All copies go from
server01 to server02
}

Pool {
    Name = "mongodb01-baculaserver02-Pool"
    Use Volume Once = yes
    Pool Type = Backup
    LabelFormat = "mongodb01-"
    AutoPrune = yes
    Recycle = yes
    VolumeRetention = 30 days
    Maximum Volumes = 365
    Maximum Volume Jobs = 1
    Recycle Oldest Volume = yes
    Storage = "mongodb01_baculaserver02_Storage"
    Next Pool = "mongodb01-baculaserver01-Pool"  # If backup runs on
server02, the copy goes to server01. Normally, all main backups are done on
server01.
}

Storage {
  Name = "mongodb01_baculaserver01_Storage"
  Address = "baculaserver01.domain"
  SDPort = 9103
  Password = "xxxxxxxxxxxxxxxxxxxxxxx"
  Device = "mongodb01_baculaserver01_Device"
  Media Type = File1
  Maximum Concurrent Jobs = "10"
  Heartbeat Interval = 10
}

Storage {
  Name = "mongodb01_baculaserver01_Storage_2"
  Address = "baculaserver01.domain"
  SDPort = 9103
  Password = "xxxxxxxxxxxxxxxxxxxxxxxx"
  Device = "mongodb01_baculaserver01_Device_2"
  Media Type = File1
  Maximum Concurrent Jobs = "10"
  Heartbeat Interval = 10
}

Storage {
  Name = "mongodb01_baculaserver02_Storage"
  Address = "baculaserver02.domain"
  SDPort = 9103
  Password = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  Device = "mongodb01_baculaserver02_Device"
  Media Type = File1
  Maximum Concurrent Jobs = "10"
  Heartbeat Interval = 10
}

Device {
  Name = "mongodb01_baculaserver01_Device"
  Media Type = File1
  Archive Device = "/backup/bacula-storage/mongodb01/"
  Maximum Concurrent Jobs = 10
  LabelMedia = yes
  Random Access = Yes
  AutomaticMount = yes
  RemovableMedia = no
  AlwaysOpen = yes
}

Device {
  Name = "mongodb01_baculaserver01_Device_2"
  Media Type = File1
  Archive Device = "/backup/bacula-storage/mongodb01/"
  Maximum Concurrent Jobs = 10
  LabelMedia = yes
  Random Access = Yes
  AutomaticMount = yes
  RemovableMedia = no
  AlwaysOpen = yes
}

#This Device is stored in baculaserver02 where only a storage daemon is
running (not bacula-dir)
Device {
  Name = "mongodb01_baculaserver02_Device"
  Media Type = File1
  Archive Device = "/backup/bacula-storage/mongodb01/"
  LabelMedia = yes;
  Random Access = Yes;
  AutomaticMount = yes;
  RemovableMedia = no;
  AlwaysOpen = no;
}


Backup is launched via cron every 3 days using:
echo 'run job=mongodb01-Backup storage="mongodb01_baculaserver01_Storage_2"
yes' | docker exec -i bacula-dir /opt/bacula/bin/bconsole
Restore is triggered by another cron using the default Storage defined in
the Job.

Any advice or shared experience would be greatly appreciated.

Thanks,

_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users