Re: bacula-fd Instances Timing Out on `status client`, despite Running Jobs

Bill Arlofski via Bacula-users <[email protected]> Wed, 21 May 2025 15:39:33 +0000
Newsgroups	gmane.comp.bacula.user
Message-ID	<[email protected]>
On 5/20/25 3:20 PM, Lloyd Brown wrote:
> Hi all,
> 
> I'm running into an issue with some bacula-fd instances and hoping someone can point me in the right direction.
> 
> In short: I have bacula-fd instances that are clearly running jobs (confirmed via strace), but they often time out when I run 
> status client=CLIENTNAME. They only seem reliably responsive when idle.
> 
> Details:
> 
>    *     Bacula version: 9.6.6 (yes, I know it's old — upgrade is planned).
>    *     Setup: Two hosts (`zhomebackup[1-2]`) running both SD and FD. A script at the beginning of each job snapshots NFS
>     shares, mounts them, and outputs file paths for backups.
>    *     Problem: These hosts struggle to handle more than 6–7 jobs effectively. Going beyond that causes a drop in aggregate
>     file scan rates.
>    *     Attempted solution: Spun up additional FD instances on separate ports (originally inside Docker, but now just
>     running natively on non-standard ports). These new instances are /intermittently/ responsive to `status client`, even
>     with only 1–3 jobs. The original FD (on the default port) remains responsive, even with 6–7 jobs.
> 
> I'm wondering if this could be a shared resource issue or some FD limitation I'm not accounting for. Or is there a better way 
> to scale job throughput?
> 
> I've attached a tarball containing systemd service files, FD configs, and relevant parts of the Director config, including an 
> example job definition.
> 
> Any insights would be greatly appreciated.
> 
> Thanks,
> Lloyd

Hello Lloyd,

For Bacula, each connection is counted as a 'Job'

This means on your FDs, once three jobs are running (six for the first one), the FD will not accept the new "job" connection 
for the `status client` which will appear as the symptom you are describing.

MaximumConcurrentJobs settings grepped from your configs:
----8<----
$ grep -ir maximum zhomebackup1/etc/bacula/bacula-fd.con*
zhomebackup1/etc/bacula/bacula-fd.conf:        Maximum Concurrent Jobs = 6
zhomebackup1/etc/bacula/bacula-fd.container1.conf:        #Maximum Concurrent Jobs = 20
zhomebackup1/etc/bacula/bacula-fd.container1.conf:        Maximum Concurrent Jobs = 3
zhomebackup1/etc/bacula/bacula-fd.container2.conf:        #Maximum Concurrent Jobs = 20
zhomebackup1/etc/bacula/bacula-fd.container2.conf:        Maximum Concurrent Jobs = 3
zhomebackup1/etc/bacula/bacula-fd.container3.conf:        #Maximum Concurrent Jobs = 20
zhomebackup1/etc/bacula/bacula-fd.container3.conf:        Maximum Concurrent Jobs = 3
----8<----

Just increase these FD settings, restart each FD and you should be fine.

A setting of `MaximumConcurrentJobs = 20` as shipped with the default/example configs is a good starting point for the FDs, 
and then you can manage the actual number of concurrent jobs triggered on each of the clients with the MaximumConcurrentJobs 
setting in the Client{} configurations on the Director which can then be adjusted up or down without requiring a restart of 
the FDs - just a bconsole 'reload' command is needed for the Director to pick up these changes.


Hope this helps,
Bill

-- 
Bill Arlofski
[email protected]

_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users
signature.asc (application/pgp-signature, 583 B)