Re: Questions about delta encoding implementation and virtual files

Good morning, 

As said yesterday, thank you so much for your answer. This is very
important for me, really. 

I answer below, in green bold for instance... for being able to be
distinguished better my comments...

El 2022-03-08 11:10, Radosław Korzeniewski escribió:

> ATENCION: Este correo se ha enviado desde fuera de la organización. No pinche en los enlaces ni abra los adjuntos a no ser que reconozca el remitente y sepa que el contenido es seguro.
> 
> Hello, 
> 
> pon., 7 mar 2022 o 12:44 egoitz--- via Bacula-devel <[email protected]> napisał(a): 
> 
>> - I'm working in trying to create an open source version of the delta encoding plugin by using the bacula-fd plugin api. When working on it I have seen Bacula's source is aware of delta and delta file sequentiation. I have seen for instance, even a .bvfs command exists for showing deltas of a file id. But, what  I have not found is that Bacula works on that Delta files generation (patch generation, signatures, etc...).
> 
> "delta files generation" - whatever it is is solely responsible for the plugin. 
> Bacula provides a delta sequencing mechanism only. 
> 
> THIS IS WHAT I WANTED TO CONFIRM :) :) YEP :) :) 
> 
>> I assume that Bacula in the non-fd part, acts just as a just delta file holder keeping the files and stores the patch sequentiation just that.
> 
> It is stored in the catalog, so during restore the Director knows that it has to include all incremental files in sequence instead of the latest one.  
> 
> I SEE... IT WAS IMPORTANT FOR ME TO CLARIFY THIS... 
> 
>> Bacula keeps records of deltas in database (and file storages) but only fd works with them (with probably a library like librsync in the delta plugin)
> 
> No, the delta sequencing - a.k.a. block level incrementals is handled by fd and dir where a fd/plugin is responsible for all diff + patches.  
> 
> I SEE... 
> 
>> in the sense of applying patches over an original file or even generating deltas when backup. Am I wrong?. Was just for understanding the nice work done and what's already written and free in Bacula's source for this purpose.
> 
> Yes, the fd/plugin is responsible for all "the dirty work" and Bacula "framework" helps organize it. 
> 
> VERY CLEAR NOW RADOSLAW... I NEEDED TO CONFIRM IT :) :) BECUASE YOU KNOW... PERHAPS THERE IS A NON-AVAILABLE CODE THAT DOES SOMETHING ELSE WITH THAT INFO STORED IN CATALOG... AND JUST... FOR HAVING A VISION OF HOW IS BEING USED THAT STORED DATA FOR LATER WRITTING IN SOME MANNER THE PENDING CODE :) . THANKS MATE :) 
> 
>> - By the way, I have one question about virtual files. I have not seen very clear (perhaps my problem as don't understand it) how to work with them. I understand the concept, but have not seen a clear example of how for instance in the backup you create a virtual file, how do you see it in bvfs and finally... what you get after restoring. In page 36/146 of Bacula 11 for developers pdf, you say "This will create a virtual file." but really you are entering in the structure : 
>> 
>> sp->type = FT_REG;
>> sp->statp.st_mode = 0700 | S_IFREG; 
>> 
>> FT_REG and S_IFREG both are for regular files.... what exactly causes a virtual file to be created?. Perhaps st_size -1?.
> 
> In this sense a "virtual" is a file which does not exist on a backup server and is created programmatically with a plugin API. This file is seen by a Bacula as an ordinary file with a slightly different backup stream id, so fd will know what tool to use to restore it. 
> You can fill "struct stat" with whatever acceptable values you want, where st_size should match as close as possible the real size of the saved file. The size == -1 will confuse users. 
> 
> OK, OK SO A VIRTUAL FILE IS WRITTEN IN THE STRUCT THE SAME MANNER AS A REGULAR FILE. YOU MEAN THEN, THERE'S NOTHING SHOULD BE DONE SPECIALLY IN THE STRUCTS, FOR BACULA TO KNOW WHETHER IT'S HANDLING A VIRTUAL OR A REGULAR FILE (AND OBVIOUSLY FOR ACTING ACCORDINGLY?. 
> 
> BUT THEN... I CAN'T REALLY UNDERSTAND HOW DOES BACULA DO THE CORRECT THING?. I MEAN... IT HAS SOME SORT OF TABLE, WITH THE FILEID, REFERENCING TO OTHER X FILEIDS TO BE RESTORED TO DISK OR SIMILAR?. 
> 
> PLEASE, LET ME SET AN EXAMPLE : 
> 
> I WANTED TO BACKED THE THREE FILES IN A VIRTUAL FILE, I SHOULD DO SOMETHING AS THIS? 
> 
> static const char *files[] = {
> "/filea",
> "/fileb",
> "/filec"
> };
> static int nb_files = 3; 
> STATIC BRC STARTBACKUPFILE(BPCONTEXT *CTX, STRUCT SAVE_PKT *SP)
> 
> {
> delta_test *self = get_self(ctx);
> if (!self) {
> return bRC_Error;
> }
> time_t now = time(NULL);
> sp->fname = (char *)"/name-of-the-virtual-for-grouping-the-three-files";
> sp->type = FT_REG;
> sp->statp.st_mode = 0700 | S_IFREG;
> sp->statp.st_ctime = now;
> sp->statp.st_mtime = now;
> sp->statp.st_atime = now;
> sp->statp.st_size = the-size-of-the-three-files-in-disk;
> sp->statp.st_blksize = 4096;
> sp->statp.st_blocks = 1;
> if (self->level == 'I' || self->level == 'D') {
> bRC state = bfuncs->checkChanges(ctx, sp);
> /* Should always be bRC_OK */
> sp->type = (state == bRC_Seen)? FT_NOCHG : FT_REG;
> sp->flags |= (FO_OFFSETS);
> }
> pm_strcpy(self->fname, files[]);
> return bRC_OK;
> }
> 
> LATER IN THIS CASE... IT'S NOT NOTHING TO DO WITH DELTAS... JUST LEAVE THE CORE CODE TO HANDLE IT? 
> 
> WHEN HAVING TO USE DELTAS... APART OBVIOUSLY OF SETTING THE FLAGS APPROPIATELY YOU DO IN THIS SAME FUNCTION : 
> 
> PM_STRCPY(SELF->FNAME, FILES[SELF->DELTA % NB_FILES]); 
> 
> I GUESS IT'S FOR SYING YOU WILL HAVE N FILES EACH TIME FOR ADDING TO VIRTUAL FILE AND THAT ARE DELTA FILES.... ISN'T IT? 
> 
> AM I WRONG? 
> 
>> Are they relevant for what I'm trying to do?. It seems Bacula handles delta sequentiation so... perhaps for this purpose I shouldn't need "virtual files"?.
> 
> No, the virtual files are available in command plugins api only and are used mainly for creating backups of different applications, i.e. running databases, where a standard file backup is useless, not optimal or simply unavailable. 
> 
> I SEE, I SEE MATE... 
> 
>> - I'm planning to implement delta encoding by checking the previous day file signature done by librsync. Instead of looking at the filesystem it would be nice if I could take a look at that signature in the last backup done (yesterday backup).
> 
> What "signature" are you thinking of?  
> There is an "accurate catalog query api" in Bacula but as far I know it is not handling checksums (md5, sha, etc). 
> You can extend this code if you wish. 
> 
> I JUST WANTED TO BE SURE, THAT THE SIGNATURE FILE HAS NOT BEEN MODIFIED SINCE I LEFT IT ON DISK WHEN GENERATED (OR REGENERATED). I DON'T REALLY MIND USING CHECKSUMS OR NOT, IF SOMETHING CAN SAY ME: "RELAX, NOBODY HAS CHANGED THE SIGNATURE FILE". I JUST WANTED TO BE SURE THE FILE IS NOT CHANGED SINCE I MODIFIED. DOES THE ACURATE API CAN HELP WITH THAT?. IF IT CAN, IT'S MORE THAN ENOUGH FOR ME. I'LL TAKE A LOOK AT ACCURATE API :) TO SEE IF IT COULD HELP ME :) :) 
> 
> . I don't really mind what the signature is... just to know if someone has "broken" the file with the signature... because then the patch won't allow later 
> 
>> Could it be possible in some manner, that if I see a file passed in EventHandleBackupFile() to check if yesterdays signature exists in the backup of yesterday, and then read the yesterday signature from the own backup?. I mean, instead of having to leave the signature in the being backed server's filesystem.
> 
> Do you want to store your librsync signatures in a catalog database? Did you count the required size?  
> 
> A CHECKSUM LIKE A SHA256 WOULD BE ENOUGH FOR ME. I DON'T REALLY NEED THE WHOLE SIGNATURE.... 
> 
>> - The last one :) . For restoring, and for the code seen (for instance in insert_missing_delta()) I assume Bacula detects we are restoring a delta compressed file.
> 
> Block incremental is far from compressed, IMVHO. BEE Plugins (i.e. vSphere, Proxmox, XENServer among others) heavily use delta sequencing for VM Image incremental backups. This functionality is not compression which you can apply to generated backup stream, i.e. LZO, GZIP or GED.  
> 
> WELL YES... TOTALLY AGREE.... I HAVE BADLY CALLED DELTA COMPRESSION.... LETS SAY JUST... DELTA ENCODING.... PERHAPS I SHOULD HAVE SAID "WE ARE RESTORING A DELTA ENCODED FILE." 
> 
>> Then I assume Bacula restores apart from the own initial file, patches to arrive to the day we want to restore to. Am I wrong?.
> 
> I do not understand your question, so I describe how it works - for fd command plugin: 
> - fd command plugin during backup generates delta sequence for a file and saves incremental information - in most cases as an offset to changed block and its new content, but it can do anything else, i.e. a QEMU Incremental Plugin generates an incremental qcow2 image in this step 
> -  during restore Bacula will send selected fd command plugin the same information and data what it generates during backup, so for XENServer Plugin it simply get offset and raw data to perform destination image patching, but for QEMU Plugin it receives a qcow2  incremental image which is then used for qcow2 image patching. 
> 
> SORRY FOR EXPLAINING WRONGLY. WHAT I TRIED TO SAY IS : "HOW THE BACKUP WILL LEAVE THE SOURCE FILE PLUS PATCHES IN THE DISK AFTER RESTORATION?. FOR INSTANCE : 
> 
> - BACULA CHECKS IF SOMETHING IS DELTA ENCODED IN THE DIRECTORY HE HAS BEEN ASKED TO GET RESTORED. 
> - IF EXIST, WITH A LOOP IT RESTORES ALL PATCHES UNTIL NO MORE PATCHES TO RESTORE EXIST. 
> - LATER CONTINUE WITH NOT DELTA ENCODED FILES?. 
> 
>> Perhaps later in a post-restore job I could run a shell script that tries to find patches pending to be applied to a parent file. I suppose then I could apply and the backup would become finally restored. Does some other more elegant way you could advise me?.
> 
> Do not run external scripts for that. It will hurt you more than you think. :) 
> 
> ADVISE LISTENED MATE :) :) 
> 
> best regards 
> 
> THANK YOU SO MUCH RADOSLAW!!! BEST REGARDS MY FRIEND :) :) -- 
> Radosław Korzeniewski
> [email protected]

_______________________________________________
Bacula-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-devel