Re: slow job with many files?

Bill Arlofski via Bacula-users <[email protected]> Tue, 27 Jan 2026 18:05:26 +0000
Newsgroups	gmane.comp.bacula.user
Message-ID	<[email protected]>
On 1/27/26 9:27 AM, Stefan G. Weichinger via Bacula-users wrote:
 >
> So I assume the number of files somehow kills the overall performance by
> doing the database/catalog inserts, right?

Hello Stefan,

No. You have `SpoolAttributes = yes` (the default setting) explicitly set in your JobDefs configuration.

This means that the SD writes the attribute data to its configured `WorkingDirectory` during a job, and when the job 
finishes, it sends this file to the Director which does an efficient batch insert of the attributes to the catalog.

If you disable attribute spooling, you will see a severe drop in performance because the SD will be sending each file's 
attributes to the Director as they are backed up and the Director will be inserting the attributes for each file into the 
catalog one at a time.


 > I might start the slower job now and watch "top" etc on the server ... could have done that already ;-)

Might want to run htop, or iotop on both client and server 
during this job.


 > And I see the NVMe maxxed out somehow, but with writes, not reads.

If this is on the client, it makes no sense at all. :)

During a backup, the FD is simply stat'ing, opening, reading, closing files. It is not writing anything, unless you are using 
Bacula Enterprise with Global Endpoint Deduplication and have set `dedup = bothsides` in the Fileset. :)


 > I compared the Filesets, both enabled zstd ... I now removed that and now the job is way faster.

Interesting, zstd is supposed to be better, faster, smaller, less CPU intense. Maybe the data it is trying to compress is not 
very compressible? In that case, disabling it would not show much increase in backup data storage use.

Maybe try with gzip and see if there is a difference?

You may also want to disable Communication Line Compression (CommCompression = no) in FD, SD, and Director to save some 
wasted CPU cycles.

And finally, you didn't mention what `signature` setting you were using
 in the Fileset(s). Take a look at that and maybe 
using a different one (less CPU intensive) may also help. This is not a security feature, so even md5 is OK to use. I 
typically use sha1, but sha256 and sha512 are also supported.


Best regards,
Bill

-- 
Bill Arlofski
[email protected]

_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users
signature.asc (application/pgp-signature, 583 B)