Re: Parallelizing Backup, Copy, and Restore for the Same Client (Best Practices?)

Hi Dragan!

From what I can see in each Copy Job’s log, every job is grabbing a
different Volume and using a different Device to read from:

JobId 9314 --> Backup
Copy JobId 9448 grabbed JobId 7565 to copy : Using Device
"mongodb01_baculaserver01_Copy_Restore_1" to read
Copy JobId 9450 grabbed JobId 7752 to copy : Using Device
"mongodb01_baculaserver01_Copy_Restore_2" to read
Copy JobId 9452 grabbed JobId 8092 to copy : Using Device
"mongodb01_baculaserver01_Copy_Restore_3" to read
Copy JobId 9454 grabbed JobId 8433 to copy : Using Device
"mongodb01_baculaserver01_Copy_Restore_4" to read.
Copy JobId 9456 grabbed JobId 8912 to copy : Using Device
"mongodb01_baculaserver01_Copy_Restore_0" to read.

Here’s one of the logs as an example (9456):

mainbackupserver-dir JobId 9456: Copying using JobId=8912
Job=mongodb01-Backup.2025-07-11_00.20.51_46
mainbackupserver-dir JobId 9456: Start Copying JobId 9456,
Job=CopyJob_mongo.2025-07-15_19.33.05_39
mainbackupserver-dir JobId 9456: Connected to Storage
"mongodb01_baculaserver02_Storage" at baculaserver02:9103 with TLS
mainbackupserver-dir JobId 9456: Connected to Storage
"mongodb01_baculaserver01_Storage_Copy_Restore" at baculaserver01:9103 with
TLS
mainbackupserver-dir JobId 9456: Using Device
"mongodb01_baculaserver01_Copy_Restore_0" to read.
baculaserver01-sd JobId 9456: Ready to read from volume "mongodb01-2255" on
File device "mongodb01_baculaserver01_Copy_Restore_0"
(/backup/bacula-storage/mongodb01/).
baculaserver01-sd JobId 9456: Forward spacing Volume "mongodb01-2255" to
addr=318

So, at least based on the logs, it seems like the jobs are not competing
over the same Volume or Device.

On Wed, Jul 16, 2025 at 10:27 AM Dragan Milivojević <[email protected]>
wrote:

> Add limits for a volume so that it accepts only one job.
>
> On Wed, 16 Jul 2025 at 14:53, Leandro Saldivar via Bacula-users <
> [email protected]> wrote:
>
>> Hi Robert! Thanks for your time and input!
>>
>> What you suggested was exactly what came to my mind yesterday. I tested
>> it and saw something very strange:
>> I was able to start one Backup Job and 5 Copy Jobs in parallel, and
>> everything looked fine at first. However, on the remote SD, the file sizes
>> stopped growing independently, and a new file appeared that seems to be
>> aggregating all 5 Copy Jobs into a single volume:
>>
>> -rw-r----- 1 root root 4.6T  Jul 16 12:35 mongodb01-2510   ← this should
>> be 1.5T max, seems to be merging copies
>> -rw-r----- 1 root root 190K Jul 15 20:05 mongodb01-2499
>> -rw-r----- 1 root root 127K Jul 15 20:04 mongodb01-2495
>> -rw-r----- 1 root root  64K  Jul 15 20:04 mongodb01-2452
>> -rw-r----- 1 root root  318  Jul 15 20:04 mongodb01-2441
>> -rw-r----- 1 root root  44G Jul 15 20:03 mongodb01-2439
>> -rw-r----- 1 root root 1.5T  Jun 29 18:43 mongodb01-2273   ← expected
>> result
>> -rw-r----- 1 root root 1.5T  Jun 29 00:55 mongodb01-2268   ← expected
>> result
>>
>> So it seems the volumes from the parallel Copy Jobs are being merged
>> unintentionally.
>>
>> I added an Autochanger with 5 Devices, declared a Storage using the
>> Autochanger, and pointed my Copy Jobs to this new Storage resource.
>>
>> You can find my config here:
>> https://gist.github.com/lean2206/54b37b8d599243f4b95e047fa4a8b47a
>>
>> Any idea what might be causing this behavior? Meanwhile, until I find a
>> better approach, I'll run the Restore Job using a RunAfterJob from the
>> Backup, and trigger the Copy Jobs via cron — both using separate Devices,
>> in parallel. So far, this setup would require 2 Devices.
>>
>> Thanks again!
>>
>> On Tue, Jul 15, 2025 at 2:03 PM Rob Gerber <[email protected]> wrote:
>>
>>> Leandro,
>>>
>>> The following assumes you are using disk based backup, where the
>>> "drives" and "autochangers" are virtual.
>>>
>>> I would suggest creating multiple "drive" devices in your bacula-sd.conf
>>> for each SD destination, then tie them together with a single autochanger
>>> resource in those respective bacula-sd.conf files. In your bacula-dir.conf,
>>> in your storage resources, reference the respective autochanger resources.
>>> If your jobs are ran with the same priority, and all the various daemons
>>> have enough "allow simultaneous jobs" specified (not the actual option, I
>>> can't recall the exact wording used by Bacula), then you should see better
>>> parallelism. I believe you will then be limited by your disk / netwoek
>>> bandwidth, and single core performance on various Bacula tasks like hashing
>>> files.
>>>
>>> At least, unless I misunderstand something.
>>>
>>> I would provide config examples, but I am afk right now.
>>>
>>> Robert Gerber
>>> 402-237-8692
>>> [email protected]
>>>
>>> On Tue, Jul 15, 2025, 9:51 AM Leandro Saldivar via Bacula-users <
>>> [email protected]> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I have a client with ~2TB of data. A full backup job takes about 15
>>>> hours. Once the backup finishes, a Copy Job is triggered (Selection Type =
>>>> PoolUncopiedJobs) to replicate the volume to a remote Storage Daemon —
>>>> which takes another 15 hours per volume in the best-case scenario. Finally,
>>>> for each backup, I run a Restore job on a replica server (same client
>>>> setup), which adds yet another 15 hours.
>>>>
>>>> The trivial approach to parallelize these jobs (Backup, Copy, Restore)
>>>> has been to define a separate Storage + Device pair for each Job. The only
>>>> difference between them is the Name. All Devices share the same Archive
>>>> Device path — which I'm unsure is a good practice.
>>>>
>>>> Is there a better way to parallelize Backup, Copy, and Restore jobs on
>>>> the same Pool/Volumes without defining a separate Storage/Device for each
>>>> Job? What would be considered a best-practice setup here?
>>>>
>>>> TL;DR
>>>> I need to run Backup, Copy, and Restore jobs in parallel for the same
>>>> client (Restore happens on a replica server). Ideally without creating
>>>> separate Storage/Device/Pool for each job.
>>>>
>>>> Bacula Version: 15.0.2 (21 March 2024)
>>>>
>>>> This is the config that currently allows me to parallelize only Backup
>>>> and Restore. Following this logic, I’d need to create additional Storage +
>>>> Device just for the Copy jobs.
>>>>
>>>> Client {
>>>>     Name = "mongodb01-fd"
>>>>     Address = xx.xx.xx.xx
>>>>     FDPort = 9102
>>>>     Catalog = MyCatalog
>>>>     Password = "PASSWORD"
>>>>     File Retention = 11 days
>>>>     Job Retention = 2 months
>>>>     AutoPrune = yes
>>>>     Heartbeat Interval = 1m
>>>>     TLS Require = yes
>>>>     TLS PSK Enable = yes
>>>> }
>>>>
>>>> Job {
>>>>     Name = "mongodb01-Backup"
>>>>     JobDefs = "MongoDef"
>>>>     Pool = "mongodb01-baculaserver01-Pool"
>>>>     Level = Full
>>>>     Client = "mongodb01-fd"
>>>>     Storage = "mongodb01_baculaserver01_Storage"
>>>>     Schedule = "DisabledSchedule"  # This is how I disable this client
>>>> so it doesn't affect others using the same JobDef with valid Schedules. I
>>>> launch the backup manually using a bconsole command.
>>>>     @"/opt/bacula/etc/bacula-dir.conf.d/jobdefs/RunCopyToServer.jd"  #
>>>> This is how I trigger Copy Jobs for clients with small backups. It works
>>>> well within a 24-hour window. For this client, I should remove this line.
>>>> }
>>>>
>>>> Pool {
>>>>     Name = "mongodb01-baculaserver01-Pool"
>>>>     Use Volume Once = yes
>>>>     Pool Type = Backup
>>>>     LabelFormat = "mongodb01-"
>>>>     AutoPrune = yes
>>>>     Recycle = yes
>>>>     VolumeRetention = 30 days
>>>>     Maximum Volumes = 365
>>>>     Maximum Volume Jobs = 1
>>>>     Recycle Oldest Volume = yes
>>>>     Next Pool = "mongodb01-baculaserver02-Pool"  # All copies go from
>>>> server01 to server02
>>>> }
>>>>
>>>> Pool {
>>>>     Name = "mongodb01-baculaserver02-Pool"
>>>>     Use Volume Once = yes
>>>>     Pool Type = Backup
>>>>     LabelFormat = "mongodb01-"
>>>>     AutoPrune = yes
>>>>     Recycle = yes
>>>>     VolumeRetention = 30 days
>>>>     Maximum Volumes = 365
>>>>     Maximum Volume Jobs = 1
>>>>     Recycle Oldest Volume = yes
>>>>     Storage = "mongodb01_baculaserver02_Storage"
>>>>     Next Pool = "mongodb01-baculaserver01-Pool"  # If backup runs on
>>>> server02, the copy goes to server01. Normally, all main backups are done on
>>>> server01.
>>>> }
>>>>
>>>> Storage {
>>>>   Name = "mongodb01_baculaserver01_Storage"
>>>>   Address = "baculaserver01.domain"
>>>>   SDPort = 9103
>>>>   Password = "xxxxxxxxxxxxxxxxxxxxxxx"
>>>>   Device = "mongodb01_baculaserver01_Device"
>>>>   Media Type = File1
>>>>   Maximum Concurrent Jobs = "10"
>>>>   Heartbeat Interval = 10
>>>> }
>>>>
>>>> Storage {
>>>>   Name = "mongodb01_baculaserver01_Storage_2"
>>>>   Address = "baculaserver01.domain"
>>>>   SDPort = 9103
>>>>   Password = "xxxxxxxxxxxxxxxxxxxxxxxx"
>>>>   Device = "mongodb01_baculaserver01_Device_2"
>>>>   Media Type = File1
>>>>   Maximum Concurrent Jobs = "10"
>>>>   Heartbeat Interval = 10
>>>> }
>>>>
>>>> Storage {
>>>>   Name = "mongodb01_baculaserver02_Storage"
>>>>   Address = "baculaserver02.domain"
>>>>   SDPort = 9103
>>>>   Password = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>>>>   Device = "mongodb01_baculaserver02_Device"
>>>>   Media Type = File1
>>>>   Maximum Concurrent Jobs = "10"
>>>>   Heartbeat Interval = 10
>>>> }
>>>>
>>>> Device {
>>>>   Name = "mongodb01_baculaserver01_Device"
>>>>   Media Type = File1
>>>>   Archive Device = "/backup/bacula-storage/mongodb01/"
>>>>   Maximum Concurrent Jobs = 10
>>>>   LabelMedia = yes
>>>>   Random Access = Yes
>>>>   AutomaticMount = yes
>>>>   RemovableMedia = no
>>>>   AlwaysOpen = yes
>>>> }
>>>>
>>>> Device {
>>>>   Name = "mongodb01_baculaserver01_Device_2"
>>>>   Media Type = File1
>>>>   Archive Device = "/backup/bacula-storage/mongodb01/"
>>>>   Maximum Concurrent Jobs = 10
>>>>   LabelMedia = yes
>>>>   Random Access = Yes
>>>>   AutomaticMount = yes
>>>>   RemovableMedia = no
>>>>   AlwaysOpen = yes
>>>> }
>>>>
>>>> #This Device is stored in baculaserver02 where only a storage daemon is
>>>> running (not bacula-dir)
>>>> Device {
>>>>   Name = "mongodb01_baculaserver02_Device"
>>>>   Media Type = File1
>>>>   Archive Device = "/backup/bacula-storage/mongodb01/"
>>>>   LabelMedia = yes;
>>>>   Random Access = Yes;
>>>>   AutomaticMount = yes;
>>>>   RemovableMedia = no;
>>>>   AlwaysOpen = no;
>>>> }
>>>>
>>>>
>>>> Backup is launched via cron every 3 days using:
>>>> echo 'run job=mongodb01-Backup
>>>> storage="mongodb01_baculaserver01_Storage_2" yes' | docker exec -i
>>>> bacula-dir /opt/bacula/bin/bconsole
>>>> Restore is triggered by another cron using the default Storage defined
>>>> in the Job.
>>>>
>>>> Any advice or shared experience would be greatly appreciated.
>>>>
>>>> Thanks,
>>>> _______________________________________________
>>>> Bacula-users mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>>
>>> _______________________________________________
>> Bacula-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>
>

_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users