Re: Streamlined application/script development [Re: coding an Application in perl]

On 01/09/17 03:05 PM, Chapman Flack wrote:
> On 09/01/2017 01:35 PM, Jean-Louis Martineau wrote:
>
> > amsvnmakehotcopy:
> > 5. It could have an INCREMENTAL property to make the hot copy faster.
>
> I see that svnadmin has a global '--incremental' option, but I never
> thought to try whether it works for 'hotcopy'. It seems not to:
>
> Subcommand 'hotcopy' doesn't accept option '--incremental'
> Type 'svnadmin help hotcopy' for usage.

It works for me.

>
> > 1. No need for a SVNREPOSITORY repository.
> > 2. The diskdevice must be the SVNREPOSITORY value.
> > 3. Need a TMPDIR property of where to do the temporary hotcopy.
> > 4. The hotcopy must be done in $TMPDIR/quote_string(diskdevice), that
> >    way, you can have multiple amsvnmakehotcopy that use the same TMPDIR.
> > 6. Must OUTPUT the DIRECTORY property
>
> I can remember now, the reason I didn't do it that way, came from
> this thread:
>
> https://marc.info/?l=amanda-hackers&m=146596522530684 
> <https://marc.info/?l=amanda-hackers&m=146596522530684>
>
> For some reason, I can't see the reply in MARC, but a copy came
> in my mail and looks like it also went to amanda-hackers. You had
> replied that the property set by a pre-foo script is only passed
> to the application's foo command. If I wanted the script to hotcopy
> to a temporary directory in pre-dle-estimate, and use the same copy
> for estimate and backup, I would need to run the script at
> pre-dle-backup time also, even though it would do nothing except
> output the same property.
Another reason why I hate estimate.
One of the problem is the time between the estimate and the backup, in 
can be many hours, so you do an old backup, its often better to take a 
new snapshot in pre-dle-backup even if the estimate is less accurate.
Also if the backup is not run, do you have some cleanup mechanism to 
delete old snaphost?

If someone run in 'estimate server', it is better to do the snapshot in 
pre-dle-backup.
>
> That's possible (though it seems unnecessarily complicated) as long
> as the directory name is something fully predictable like
> quote_string(diskdevice). That does make it easy to have several DLEs
> using the script on different repositories ... but it would also make
> it hard to have the same repository mentioned in two DLEs (daily/
> monthly or onsite/offsite). In that case, the collision would have to
> be avoided by setting TMPDIR differently in one DLE. But I am not
> sure that is more clear than setting a different diskdevice, and using
> the repository property.
You can use the config name in the path, but why?
Since the TMPDIR is a hotcopy, I think you can share it for multiple 
backup set, but that would require a lock so nothing update it while you 
do the backup.
The lock should be done anyway since a user can run two backup in 
parallel for the same config.
>
> All collisions would be avoided by using a randomly generated
> name in TMPDIR, and using the DIRECTORY property to output that.
> However, in that case, the script would need local state storage,
> so the script in pre-dle-estimate could save the generated name
> and later in pre-dle-backup could output it again.


>
> The same issues came up in amlvmsnapshot, since pre-dle-estimate
> needs create a snapshot with some name, and mount the snapshot
> somewhere, but post-dle-backup needs to unmount and destroy it.
> The sure way to avoid collisions is to generate the names,
> but then they need to be saved somehow between pre-dle-estimate
> and post-dle-backup.
>
> Maybe a lot of things would be easier if the server would actually
> remember properties set by early script stages and pass them on
> to later stages of the application and scripts.
They could be remember on the client too.
>
> These questions also figure into the 'future work' design of some
> kind of configurable permission-granting mechanism. Consider
> amlvmsnapshot, which needs privilege to create and destroy snapshots
> and mount and unmount them. It would be ideal to imagine configuring
> a permission rule saying "this script can issue umount and lvremove
> commands, only if the arguments are the same as for the lvcreate and
> mount it issued earlier" but that means the information needs to
> be saved somewhere between stages (and in some verifiable fashion,
> so a script can't just lie about it).
>
> By the way, in your reply on that thread, you said you had updated
> the Script_API wiki page, but today I don't see the changes; is it
> possible you never saved, or they got reverted somehow?
I don't know what happened, I update the page.
>
> > amopaquetree:
> >
> > 2. I like the idea of backing up only a section of a file, but the cost
> >    of keeping a copy is huge.
>
> Everything is relative ... it is large, and that's documented, so an
> admin can decide whether it makes sense for a given situation.
>
> If you've got a database that is truly huge, you probably won't use
> this approach, but if it is merely large, and you have enough
> local disk to fit 2 ✕ large, and that reduces the size of your
> increments and network bandwidth to 0.0005 ✕ large and you store
> a lot of those, you might take the bait. (Those are roughly the
> actual numbers I'm seeing with a database at work where we're using
> the technique.)
I agree, it can be useful for someone.
>
> > amgtar:
> >
> > 1. We could add an OPAQUE property to amgtar to not list all files in
> >    the index.
>
> I'm not sure whether opaqueness was really the major selling point
> for amopaquetree ... it was more like, well, that's how rsync batches
> work, but there are kinds of job where it's ok, and the space saving
> makes it worthwhile. If there were a way to index an rsync batch and
> individually restore from it, I would have been happy to do that (and
> then name the app something else). So I am not sure it would be
> important to modify other apps to let them be opaque. :)
>
> > amgrowingfile:
> >
> > 1. Could keep a md5sum of the backed section of the file to verify it
> >    was not modified.
> > 2. If modified, it should output the following line to the MESG stream
> >    and exit, it tell amdump to retry the dump immediately at level 0
> >      * sendbackup: retry delay 0 level 0 message \"File was reset\"\n
>
> I did think about doing something like that (guess you saw my
> ### verify prior size and digest here comment). Just didn't take
> the time to implement it. It should probably be optional anyway;
> there could be a property for it. If you want it, every backup
> takes time O(size of whole file); if you trust the process that
> produces the file, then you don't need it, and the backup takes
> O(size of increment) time only. If you chattr +a the file, then
> you can be sure it is only appended.
Unless remove and recreated.
>
> > amgrowingzip
> >
> > 2. Is there a race if you do the backup at the same time a process
> >    append to the archive? Maybe it will not find the central directory?
>
> That's why the FLOCK property is there.
>
> -Chap
This message is the property of CARBONITE, INC. and may contain confidential or privileged information.
If this message has been delivered to you by mistake, then do not copy or deliver this message to anyone.  Instead, destroy it and notify me by reply e-mail