Re: slow job with many files?

Am 27.01.26 um 19:05 schrieb Bill Arlofski via Bacula-users:
> On 1/27/26 9:27 AM, Stefan G. Weichinger via Bacula-users wrote:
>  >
>> So I assume the number of files somehow kills the overall performance by
>> doing the database/catalog inserts, right?
> 
> Hello Stefan,
> 
> No. You have `SpoolAttributes = yes` (the default setting) explicitly 
> set in your JobDefs configuration.
> 
> This means that the SD writes the attribute data to its configured 
> `WorkingDirectory` during a job, and when the job finishes, it sends 
> this file to the Director which does an efficient batch insert of the 
> attributes to the catalog.
> 
> If you disable attribute spooling, you will see a severe drop in 
> performance because the SD will be sending each file's attributes to the 
> Director as they are backed up and the Director will be inserting the 
> attributes for each file into the catalog one at a time.

The hypothesis with the DB is wrong, I know. I saw it yesterday, the 
load on the server is low.

>  > And I see the NVMe maxxed out somehow, but with writes, not reads.
> 
> If this is on the client, it makes no sense at all. :)
> 
> During a backup, the FD is simply stat'ing, opening, reading, closing 
> files. It is not writing anything, unless you are using Bacula 
> Enterprise with Global Endpoint Deduplication and have set `dedup = 
> bothsides` in the Fileset. :)

definitely no enterprise stuff here

I wonder if it is related to btrfs on the client: there are 
btrfs-snapshots in that pool, etc / this might somehow lead to 
write-activities ... I can't explain it fully yet.

Now without compression it's not an issue anymore, as far as I saw 
yesterday.

>  > I compared the Filesets, both enabled zstd ... I now removed that and 
> now the job is way faster.
> 
> Interesting, zstd is supposed to be better, faster, smaller, less CPU 
> intense. Maybe the data it is trying to compress is not very 
> compressible? In that case, disabling it would not show much increase in 
> backup data storage use.
> 
> Maybe try with gzip and see if there is a difference?
> 
> You may also want to disable Communication Line Compression 
> (CommCompression = no) in FD, SD, and Director to save some wasted CPU 
> cycles.

might try, thanks
> And finally, you didn't mention what `signature` setting you were using
> in the Fileset(s). Take a look at that and maybe using a different one 
> (less CPU intensive) may also help. This is not a security feature, so 
> even md5 is OK to use. I typically use sha1, but sha256 and sha512 are 
> also supported.

The defs are shown in the first posting. *no* signature set. Is that 
wrong/problematic?

thanks, Stefan