Re: Questions about delta encoding implementation and virtual files

Hi Eric!! 

Sorry for answering so late. I answer between lines in color green bold
for instance, for better clarification.

El 2022-03-08 09:56, Eric Bollengier escribió:

> Hello,
> 
> On 3/7/22 12:44, egoitz--- via Bacula-devel wrote: 
> 
>> Hi!,
>> 
>> After digging in plugin building documentation and checking the provided
>> examples, I have some doubts I have not been able to clarify by my own.
>> I describe them below in case you could give me a hand :). I would be
>> very thankful if you could help me a little clarifying this doubts :)
> 
> I very pleased to see someone interested to develop Bacula plugins,
> specially around data compression. 
> 
> I WROTE SOMETIME AGO TOO A VERY LITTLE PATCH FOR BACULA, FOR SPEEDING UP THE BVFS_CACHE ON POSTGRESQL (IT HAPPENED AT LEAST IN THE POSTGRESQL VERSION WE WERE USING AT LEAST IN THAT MOMENT). VERY HAPPY AND PROUD OF DOING THAT AND THIS NEW ONE :) . 
> 
> MY PLAN IS TO FIRST WRITE ABOUT DATA COMPRESSION BUT I WOULD LOVE TOO... TO KNOW BETTER THE BACULA FD PLUGIN API, IN ORDER TO WRITE SOME OTHER CUSTOM PLUGINS FOR US... THE DELTA ONE, IS IMPORTANT FOR US, BUT IT'S TOO THE FACT OF LEARNING HOW THE API WORKS. YOU KNOW, KNOWLEDGE ENDS UP BEING FLEXIBILITY AND SO... AND BECAUSE I WANTED TOO DOING IT TOO REALLY :) :) :) :) LOL 
> 
>> - I'm working in trying to create an open source version of the delta
>> encoding plugin by using the bacula-fd plugin api. When working on it I
>> have seen Bacula's source is aware of delta and delta file
>> sequentiation.
> 
> Yes, Bacula can manage "delta" or "patches" for a given file. The first
> Full backup should take the entire file, the delta_seq will be 0. 
> 
> YEP I SUPPOSED BY THE CODE READ... 
> 
> I ASSUME NOW IT'S JUST FOR REGISTRATION AND HELPING DEVELOPERS :) :) :) 
> 
> BUT BASICALLY PREVIOUSLY WANTED TO KNOW IF YOU WERE JUST STORING THE DELTA SEQUENCE, BECAUSE DELTA ENCONDING IS USEFUL IF YOU NEED IT OR YOU WERE STORING THAT DELTA DATA FOR LATER USING IT WITH SOME OTHER CODE THAT I DON'T HAVE OR WHATEVER.... 
> 
> During the first Incr or the Differential, the plugin can generate a
> "patch" (can be done with "diff", or "xdelta", or rsync, or whatever). 
> 
> OK, YES I SUPPOSE I WILL USE RDIFF DIRECTLY. THE BINARY BUILT AND PROVIDED WITH LIBRSYNC BY DEFAULT. IT SEEMS TO RUN PRETTY FINE EVEN ON WINDOWS WITH MSYS2.ORG PACKAGE SO.... 
> 
> This new files is based on the original file, and the delta_seq will
> be set to 1 automatically (Accurate mode should be turned on).  
> 
> YEP I HAVE READ BEFORE THIS TOO... 
> 
> The plugin
> must set a variable in the save_pkt to indicates that it has saved a
> patch,  
> 
> I ASSUME, YOU ARE TALKING ABOUT THE PRESENCE OF THE FO_DELTA FLAG IN SP->FLAGS? 
> 
> and at the restore time, bacula must send back the version of
> the file included in the Full, then in Diff (if any) and all Incrementals. 
> 
> YES THIS PART IT'S CLEAR TOO...
> 
> On the plugin side, the restore code will be called for each part of
> the file. 
> OK, SO EACH THE INITIAL FILE COPIED USING DELTA PLUS ALL IT'S PATCHES WILL GET RESTORED AS TOTALLY NORMAL FILES. FOR INSTANCE, WHEN BACULA IS GOING TO RESTORE A FILE WHICH IS BACKED UP USING DELTA ENCODING, BACULA WILL DO :

> - RESTORE INITIAL FILE

> - RESTORE FIRST PATCH

> - RESTORE SECOND PATCH

> - AND SO ON...?
> 
>> I have seen for instance, even a .bvfs command exists for
>> showing deltas of a file id. But, what  I have not found is that Bacula
>> works on that Delta files generation (patch generation, signatures,
>> etc...). I assume that Bacula in the non-fd part, acts just as a just
>> delta file holder keeping the files and stores the patch sequentiation
>> just that. Bacula keeps records of deltas in database (and file
>> storages) but only fd works with them (with probably a library like
>> librsync in the delta plugin) in the sense of applying patches over an
>> original file or even generating deltas when backup. Am I wrong?. Was
>> just for understanding the nice work done and what's already written and
>> free in Bacula's source for this purpose.
> 
> I think that you got the concept, the "delta" in Bacula means that you
> must restore all parts of a file, and not just the last copy.
> 
> With a program such as VMware, CBT helps the backup software to generate
> "patches" as well for example. 
> 
> YEP I HAVE READ ABOUT CBT TOO... AND EVEN I HAVE DONE SOME WORK FOR BACKING IP XCP-NG USING DELTAS OF VM :) (THERE IN XCP-NG WITHOUT CBT :) ) 
> 
>> - By the way, I have one question about virtual files. I have not seen
>> very clear (perhaps my problem as don't understand it) how to work with
>> them. I understand the concept, but have not seen a clear example of how
>> for instance in the backup you create a virtual file, how do you see it
>> in bvfs and finally... what you get after restoring. In page 36/146 of
>> Bacula 11 for developers pdf, you say "This will create a virtual file."
>> but really you are entering in the structure :
>> 
>> sp->type = FT_REG;
>> sp->statp.st_mode = 0700 | S_IFREG;
>> 
>> FT_REG and S_IFREG both are for regular files.... what exactly causes a
>> virtual file to be created?. Perhaps st_size -1?.
> 
> A virtual file is generated by plugins, they don't have to exist on disk.
> The name can be anything, it can also point to an existing file.
> 
> The plugin code will be executed to restore the "virtual file", the result
> can be a real file on disk, or a virtual machine on Proxmox for example.
> 
> In this example, we have a regular file, but it's a virtual file that may
> or may not exist on the filesystem. 
> 
> BUT, ONE THING....  I DON'T HAVE CLEAR, HOW BACULA DISTINGUISHES IF IT'S STORING A VIRTUAL FILE OR A NORMAL ONE. WHICH FIELD OR PARAMETER SHOULD YOU SET AND HOW IN 
> THE SAVE_PKT STRUCTURE?. FIRST I THOULD IT COULD BE USING A SIZE OF -1 IN THE SAVE_PKT STRUCTURE OR... USING SOME KIND OF FLAG OR... HOW DOES BACULA KNOW 
> IT'S DEALING IN THE STRUCTURES WITH A VIRTUAL OR NORMAL FILE? 
> 
> OK... I GOT THIS OTHER PART... SO I SUPPOSE IT DOES NOT APPLY FOR DELTA ENCODING BECAUSE HERE BACULA, HAS ALREADY DEFINED VARIABLES LIKE DELTA SEQUENCE AND SO... FOR HANDLING DELTA ENCONDING BACKUP AND RESTORE... BUT FOR EXAMPLE IF YOU WANTED TO GROUP FOR INSTANCE FIVE FILES TO GET RESTORED... YOU COULD SAY THEY ARE A VIRTUAL FILE AND WHEN YOU RESTORE THAT VIRTUAL FILE, THE FIVE FILES WILL APPEAR IN THE DISK.... 
> 
> COULD YOU PLEASE ERIC, TELL ME HOW YOU SET IN THE STRUCTURE YOU ARE DEALING WITH A VIRTUAL FILE?. 
> 
>> Are they relevant for what I'm trying to do?. It seems Bacula handles
>> delta sequentiation so... perhaps for this purpose I shouldn't need
>> "virtual files"?.
> 
> In your case, it will be virtual files that points to regular files. 
> 
> BUT YOU SAID BACULA WOULD CALL FOR INSTANCE THE ENDBACKUPFILE() FOR EACH FILE BACKED WITH DELTA ENCODING. I MEAN, WOULD IT RESTORE THE WAY DESCRIBED SOME LINES ABOVE?. BECAUSE IF IT WOULD DO THIS WAY : 
> 
>> - RESTORE INITIAL FILE
> 
>> - RESTORE FIRST PATCH
> 
>> - RESTORE SECOND PATCH
> 
>> - AND SO ON...?
> 
> I ASSUME THERE'S NO SENSE HERE ON USING VIRTUAL FILES, AM I WRONG?. HAVE I UNDERSTAND SOMETHING WRONG?. 
> 
>> - I'm planning to implement delta encoding by checking the previous day
>> file signature done by librsync. Instead of looking at the filesystem it
>> would be nice if I could take a look at that signature in the last
>> backup done (yesterday backup). Could it be possible in some manner,
>> that if I see a file passed in EventHandleBackupFile() to check if
>> yesterdays signature exists in the backup of yesterday, and then read
>> the yesterday signature from the own backup?. I mean, instead of having
>> to leave the signature in the being backed server's filesystem.
> 
> HERE COMES AN INTERESTING PART :) 
> 
> You can store information in the save_pkt structure and the plugin can check
> the last version of that information with the accurate mode. 
> 
> I WILL CHECK THE SIGNATURE FILE IS THE EXPECTED WAY WITH A SHA256 SUM. I'M NOT GOING TO COPY THE WHOLE SIGNATURE FILE... 
> 
> OK, SO IF I UNDERSTAND YOU PROPERLY... YOU MEAN : 
> 
> IF I'M GOING TO BACKUP SOMETHING PREVIOUSLY BACKED UP USING DELTA ENCODING OR I'M BACKING UP A PATCH FILE GENERATED BY DELTA ENCODING OR THE OWN SIGNATURE FILE OF A DELTA COPY, YOU SAY : 
> 
> - FIND THE SIGNATURE FILE OF THAT DELTA COPY GROUP, CALCULATE A SHA256 OF THE SIGNATURE FILE PRESENT IN DISK 
> - Copy to a new variable in save_pkt structure the calculated sha256
> 
> - When a function using save_pkt will get called and see, that the new now present, field inside structure with the sha256 of the file signature present in disk (of a delta copy group), has the same value as the one stored sha256 in a database for the signature file we're checking, go on with the patch or return a bRC_Error in that function?.
> 
> In general, you can use a couple of bytes with this technique. 
> 
> I DON'T UNDERSTAND YOU THIS LAST SENTENCE SORRY :)
> 
> I don't think you have enough space to store a file signature, you will
> have to use an other way to store it (a local file, a database record, ...) 
> 
> I WILL USE A SHA256 FOR INSTANCE... IT WOULD BE SMALLER AND IT WOULD PROVIDE SAME GUARANTEE FOR KNOWING THE SIGNATURE FILE IS NOT MODIFIED WITHOUT EXPECTED... (YOU KNOW FOR GENERATING PROPERLY THE PATCH)...
> 
>> - The last one :) . For restoring, and for the code seen (for instance
>> in insert_missing_delta()) I assume Bacula detects we are restoring a
>> delta compressed file. Then I assume Bacula restores apart from the own
>> initial file, patches to arrive to the day we want to restore to. Am I
>> wrong?.
> 
> This is correct, bacula will send back to the plugin the data that was
> produced. Up to the plugin to reassemble the data. If one delta piece is
> missing, the restore will stop to the last correct one. 
> 
> I SEE... SO I ASSUME WHEN RESTORING IT SAIS.... : 
> 
> - FILEA -> NORMAL FILE... 
> - FILEB -> NORMAL FILE... 
> - FILEC --> HEY! THIS HAS A DELTA_SEQ TO 0 SO THIS IS COPIED USING DELTA ENCODING!! LET'S FIRST OF ALL, RESTORE THIS ONE FILE BUT LATER WE WILL RESTORE ONE BY ONE EACH PATCH IF THIS DELTA GROUP. IS THIS CORRECT?. 
> 
> IF IT DOES THIS WAY... SURE... I COULD ONCE IN THE DISK RESTORED THE PATCH (IN THE ENDBACKUPFILE() FOR INSTANCE) APPLY THE PATCH TO THE FILE ALREADY RESTORED THAT CORRESPONDED TO DELTA_SEQ 0... 
> 
> WHEN APPLYING ALL PATCHES, I WILL REGENERATE THE SIGNATURE OF THE DELTA COPIED FILE.... 
> 
>> Perhaps later in a post-restore job I could run a shell script
>> that tries to find patches pending to be applied to a parent file. I
>> suppose then I could apply and the backup would become finally restored.
>> Does some other more elegant way you could advise me?.
> 
> Normally, the plugin will receive patches one after the other, you can
> re-open the file on disk and apply the patch. It is also possible to
> store everything on disk and call a script to do the work at the end,
> it depends. 
> 
> I SEE... SO THE SUPPOSED IDEA WRITTEN ABOVE... OK OK.... 
> 
> Good luck, 
> 
> THANK YOU SO MUCH FOR YOUR NICE HELP ERIC. I WOULD REALLY LOVE TO GET KNOWLEDGE OF THE BACULA PLUGIN API... :) 
> 
> CHEERS!!! 
> 
> Best Regards,
> Eric

_______________________________________________
Bacula-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-devel