Re: Streamlined application/script development [Re: coding an Application in perl]

On 09/01/2017 01:35 PM, Jean-Louis Martineau wrote:

> amsvnmakehotcopy:
> 5. It could have an INCREMENTAL property to make the hot copy faster.

I see that svnadmin has a global '--incremental' option, but I never
thought to try whether it works for 'hotcopy'. It seems not to:

  Subcommand 'hotcopy' doesn't accept option '--incremental'
  Type 'svnadmin help hotcopy' for usage.

> 1. No need for a SVNREPOSITORY repository.
> 2. The diskdevice must be the SVNREPOSITORY value.
> 3. Need a TMPDIR property of where to do the temporary hotcopy.
> 4. The hotcopy must be done in $TMPDIR/quote_string(diskdevice), that
>    way, you can have multiple amsvnmakehotcopy that use the same TMPDIR.
> 6. Must OUTPUT the DIRECTORY property

I can remember now, the reason I didn't do it that way, came from
this thread:

https://marc.info/?l=amanda-hackers&m=146596522530684

For some reason, I can't see the reply in MARC, but a copy came
in my mail and looks like it also went to amanda-hackers. You had
replied that the property set by a pre-foo script is only passed
to the application's foo command. If I wanted the script to hotcopy
to a temporary directory in pre-dle-estimate, and use the same copy
for estimate and backup, I would need to run the script at
pre-dle-backup time also, even though it would do nothing except
output the same property.

That's possible (though it seems unnecessarily complicated) as long
as the directory name is something fully predictable like
quote_string(diskdevice). That does make it easy to have several DLEs
using the script on different repositories ... but it would also make
it hard to have the same repository mentioned in two DLEs (daily/
monthly or onsite/offsite). In that case, the collision would have to
be avoided by setting TMPDIR differently in one DLE. But I am not
sure that is more clear than setting a different diskdevice, and using
the repository property.

All collisions would be avoided by using a randomly generated
name in TMPDIR, and using the DIRECTORY property to output that.
However, in that case, the script would need local state storage,
so the script in pre-dle-estimate could save the generated name
and later in pre-dle-backup could output it again.

The same issues came up in amlvmsnapshot, since pre-dle-estimate
needs create a snapshot with some name, and mount the snapshot
somewhere, but post-dle-backup needs to unmount and destroy it.
The sure way to avoid collisions is to generate the names,
but then they need to be saved somehow between pre-dle-estimate
and post-dle-backup.

Maybe a lot of things would be easier if the server would actually
remember properties set by early script stages and pass them on
to later stages of the application and scripts.

These questions also figure into the 'future work' design of some
kind of configurable permission-granting mechanism. Consider
amlvmsnapshot, which needs privilege to create and destroy snapshots
and mount and unmount them. It would be ideal to imagine configuring
a permission rule saying "this script can issue umount and lvremove
commands, only if the arguments are the same as for the lvcreate and
mount it issued earlier" but that means the information needs to
be saved somewhere between stages (and in some verifiable fashion,
so a script can't just lie about it).

By the way, in your reply on that thread, you said you had updated
the Script_API wiki page, but today I don't see the changes; is it
possible you never saved, or they got reverted somehow?

> amopaquetree:
> 
> 2. I like the idea of backing up only a section of a file, but the cost
>    of keeping a copy is huge.

Everything is relative ... it is large, and that's documented, so an
admin can decide whether it makes sense for a given situation.

If you've got a database that is truly huge, you probably won't use
this approach, but if it is merely large, and you have enough
local disk to fit 2 ✕ large, and that reduces the size of your
increments and network bandwidth to 0.0005 ✕ large and you store
a lot of those, you might take the bait. (Those are roughly the
actual numbers I'm seeing with a database at work where we're using
the technique.)

> amgtar:
> 
> 1. We could add an OPAQUE property to amgtar to not list all files in
>    the index.

I'm not sure whether opaqueness was really the major selling point
for amopaquetree ... it was more like, well, that's how rsync batches
work, but there are kinds of job where it's ok, and the space saving
makes it worthwhile. If there were a way to index an rsync batch and
individually restore from it, I would have been happy to do that (and
then name the app something else). So I am not sure it would be
important to modify other apps to let them be opaque. :)

> amgrowingfile:
> 
> 1. Could keep a md5sum of the backed section of the file to verify it
>    was not modified.
> 2. If modified, it should output the following line to the MESG stream
>    and exit, it tell amdump to retry the dump immediately at level 0
>      * sendbackup: retry delay 0 level 0 message \"File was reset\"\n

I did think about doing something like that (guess you saw my
### verify prior size and digest here comment). Just didn't take
the time to implement it. It should probably be optional anyway;
there could be a property for it. If you want it, every backup
takes time O(size of whole file); if you trust the process that
produces the file, then you don't need it, and the backup takes
O(size of increment) time only. If you chattr +a the file, then
you can be sure it is only appended.

> amgrowingzip
> 
> 2. Is there a race if you do the backup at the same time a process
>    append to the archive? Maybe it will not find the central directory?

That's why the FLOCK property is there.

-Chap