Re: Strategies for backup of millions of files...

Hi Marco,

Am 13.09.2024 um 17:42 schrieb Marco Gaiarin:
> Mandi! Marco Gaiarin
>    In chel di` si favelave...
> 
>> EG, i'll come back on this on mid-september...
> 
> Still working on script.

I have not looked up the discussion leading here.

> 
> For a sake of mental health i want to mount snapshot on different mount
> point... so different file path.
> But path are defined in filesets, and i cannot define adifferent fileset for
> every mountpoint...
> 
> I know i can use StripPrefix/AddPrefix on restore, but there's some way to
> use a 'StripPrefix' like options on jobs?

The File Set Include Option "Strip Path = <integer>" may be useful.

What I usually prefer is a combination of a Run Script on the client to 
create a file list to back up, and a File= entry in the include list 
that reads from file or program, i.e. "\|" or "\<" ones.

However, you can do really interesting and slightly confusing things, too:

Job {
   Name = "magichomedirs:_home_b:10:27"
   Fileset = "magichomedirs"
   Level = Full
   Type = "Backup"
   Client = "homedirfileserver-fd"
   MaximumConcurrentJobs = 1
   Messages = "Standard"
   Pool = "short-lab"
   Priority = 10
}

FileSet {
   Name = magichomedirs
   Include {
     Options {
       Signature = SHA1
     }
     File = "\\|/opt/bacula/scripts/magicfileset.sh '%n'"
   }
}

# cat /opt/bacula/scripts/magicfileset.sh
#!/bin/bash

IFS=: read -r j p v b <<< $1
if [ -z "$b" ] ; then exit 0; fi
p=${p//_/\/}
# echo "Job $j prefix $p from $v to $b"
w=$(dirname "$p")
n=$(basename "$p")
# below is only one line, wrapped by mailer!
find "$w" -maxdepth 1 -type d -iname "$n"'*' | awk "/.*[0-9]+$/ 
{p=match(\$0, /[0-9]+\$/); s=0+substr(\$0, p); if (p && (s>=$v) && 
(s<$b)) print \$0}"

Which works as intended:

# ls /home/
arno  b004  b008  b012  b016  b020  b026  b030  b034  test
b001  b005  b009  b013  b017  b021  b027  b031  b035
b002  b006  b010  b014  b018  b024  b028  b032  b036
b003  b007  b011  b015  b019  b025  b029  b033  b037
# /opt/bacula/scripts/magicfileset.sh magichomedirs:_home_b:10:27
/home/b010
/home/b011
/home/b012
/home/b013
/home/b014
/home/b015
/home/b016
/home/b017
/home/b018
/home/b019
/home/b020
/home/b021
/home/b024
/home/b025
/home/b026
#

Usually, more explicit configuration is better -- you configure once, 
usually under no pressure and with time to experiment, but you need this 
stuff to have predictable effect when you are under pressure.

Cheers,

Arno

> 
> I hope i was clear...
> 

-- 
Arno Lehmann

IT-Service Lehmann
Sandstr. 6, 49080 Osnabrück


_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users