Re: Questions about delta encoding implementation and virtual files

Hi Eric!! 

Thank you so much for your answer. Give me please some minutes to read
your mail relaxedly. I'm not just interested in developing plugins!!! I
would love that Bacula would triumph over Veeam... and I wanted to
contribute for that!!! 

Let me please read carefully and please I emphasize on being extremely
thankful for your help really Eric :) :) 

I answer later :) :) 

Cheers!!

El 2022-03-08 09:56, Eric Bollengier escribió:

> ATENCION
> ATENCION
> ATENCION!!! Este correo se ha enviado desde fuera de la organizacion. No pinche en los enlaces ni abra los adjuntos a no ser que reconozca el remitente y sepa que el contenido es seguro.
> 
> Hello,
> 
> On 3/7/22 12:44, egoitz--- via Bacula-devel wrote: 
> 
>> Hi!,
>> 
>> After digging in plugin building documentation and checking the provided
>> examples, I have some doubts I have not been able to clarify by my own.
>> I describe them below in case you could give me a hand :). I would be
>> very thankful if you could help me a little clarifying this doubts :)
> 
> I very pleased to see someone interested to develop Bacula plugins,
> specially around data compression.
> 
>> - I'm working in trying to create an open source version of the delta
>> encoding plugin by using the bacula-fd plugin api. When working on it I
>> have seen Bacula's source is aware of delta and delta file
>> sequentiation.
> 
> Yes, Bacula can manage "delta" or "patches" for a given file. The first
> Full backup should take the entire file, the delta_seq will be 0.
> 
> During the first Incr or the Differential, the plugin can generate a
> "patch" (can be done with "diff", or "xdelta", or rsync, or whatever).
> 
> This new files is based on the original file, and the delta_seq will
> be set to 1 automatically (Accurate mode should be turned on). The plugin
> must set a variable in the save_pkt to indicates that it has saved a
> patch, and at the restore time, bacula must send back the version of
> the file included in the Full, then in Diff (if any) and all Incrementals.
> 
> On the plugin side, the restore code will be called for each part of
> the file.
> 
>> I have seen for instance, even a .bvfs command exists for
>> showing deltas of a file id. But, what  I have not found is that Bacula
>> works on that Delta files generation (patch generation, signatures,
>> etc...). I assume that Bacula in the non-fd part, acts just as a just
>> delta file holder keeping the files and stores the patch sequentiation
>> just that. Bacula keeps records of deltas in database (and file
>> storages) but only fd works with them (with probably a library like
>> librsync in the delta plugin) in the sense of applying patches over an
>> original file or even generating deltas when backup. Am I wrong?. Was
>> just for understanding the nice work done and what's already written and
>> free in Bacula's source for this purpose.
> 
> I think that you got the concept, the "delta" in Bacula means that you
> must restore all parts of a file, and not just the last copy.
> 
> With a program such as VMware, CBT helps the backup software to generate
> "patches" as well for example.
> 
>> - By the way, I have one question about virtual files. I have not seen
>> very clear (perhaps my problem as don't understand it) how to work with
>> them. I understand the concept, but have not seen a clear example of how
>> for instance in the backup you create a virtual file, how do you see it
>> in bvfs and finally... what you get after restoring. In page 36/146 of
>> Bacula 11 for developers pdf, you say "This will create a virtual file."
>> but really you are entering in the structure :
>> 
>> sp->type = FT_REG;
>> sp->statp.st_mode = 0700 | S_IFREG;
>> 
>> FT_REG and S_IFREG both are for regular files.... what exactly causes a
>> virtual file to be created?. Perhaps st_size -1?.
> 
> A virtual file is generated by plugins, they don't have to exist on disk.
> The name can be anything, it can also point to an existing file.
> 
> The plugin code will be executed to restore the "virtual file", the result
> can be a real file on disk, or a virtual machine on Proxmox for example.
> 
> In this example, we have a regular file, but it's a virtual file that may
> or may not exist on the filesystem.
> 
>> Are they relevant for what I'm trying to do?. It seems Bacula handles
>> delta sequentiation so... perhaps for this purpose I shouldn't need
>> "virtual files"?.
> 
> In your case, it will be virtual files that points to regular files.
> 
>> - I'm planning to implement delta encoding by checking the previous day
>> file signature done by librsync. Instead of looking at the filesystem it
>> would be nice if I could take a look at that signature in the last
>> backup done (yesterday backup). Could it be possible in some manner,
>> that if I see a file passed in EventHandleBackupFile() to check if
>> yesterdays signature exists in the backup of yesterday, and then read
>> the yesterday signature from the own backup?. I mean, instead of having
>> to leave the signature in the being backed server's filesystem.
> 
> You can store information in the save_pkt structure and the plugin can check
> the last version of that information with the accurate mode.
> 
> In general, you can use a couple of bytes with this technique.
> 
> I don't think you have enough space to store a file signature, you will
> have to use an other way to store it (a local file, a database record, ...)
> 
>> - The last one :) . For restoring, and for the code seen (for instance
>> in insert_missing_delta()) I assume Bacula detects we are restoring a
>> delta compressed file. Then I assume Bacula restores apart from the own
>> initial file, patches to arrive to the day we want to restore to. Am I
>> wrong?.
> 
> This is correct, bacula will send back to the plugin the data that was
> produced. Up to the plugin to reassemble the data. If one delta piece is
> missing, the restore will stop to the last correct one.
> 
>> Perhaps later in a post-restore job I could run a shell script
>> that tries to find patches pending to be applied to a parent file. I
>> suppose then I could apply and the backup would become finally restored.
>> Does some other more elegant way you could advise me?.
> 
> Normally, the plugin will receive patches one after the other, you can
> re-open the file on disk and apply the patch. It is also possible to
> store everything on disk and call a script to do the work at the end,
> it depends.
> 
> Good luck,
> 
> Best Regards,
> Eric

_______________________________________________
Bacula-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-devel