Re: coding an Application in perl

On 02/23/2016 10:55 AM, Jean-Louis Martineau wrote:
> Hi,
> 
> Thanks for your interest in writing application for amanda.
> What kind of applications do you want to write?

Well, I have the following several ideas for a sequence of
increasingly specialized apps.

I like the idea that the apps are Perl classes, and I think they
could make even more extensive use of object design and inheritance,
so that a new app could be written in very little code, inheriting
from a base that does a lot of the work.

The first one I would propose would have a name something like
amMonotonicFile. It would be like amraw, the user object is a
single file, but with the further restriction that the file is
only ever appended to (log or audit files, for example).

Unlike amraw, amMonotonicFile can support levels > 0; the saved
state is very simple (the length of the file at the lower level
backup) and an increment is just the tail from that point to the
current end.

Some filesystems allow setting a flag to actually enforce that
a file is never modified except by appending. Even where that
flag is not available, the saved state can include both the
previous length and a checksum, to verify that the assumption
of monotonicity has not been violated.

This application might seem of limited use by itself, but can
also make a good base object for similar applications.

The next would be called something like amMonotonicZip. Again
the user object is a single file, now restricted to be a zip
file that only has new entries added at the end. Appending a
new zip entry is not strictly an append at the file level; it
is a seek to the end, backspace over the zip directory, write
the new entry and the new directory. Not all zip tools actually
do this, some may rewrite the whole file no matter what you
change, so with them a monotonic zip is not possible. But there
are some tools (like the zipfile module in Python) that do work
exactly that way, and can be used to grow a purely monotonic
zip file.

The only differences between monotonicFile and monotonicZip
are that the saved state would not be the previous exact end
of file, but the previous offset of the zip directory, and
when restoring such a file, the pointer has to back up to
that point before writing the next increment.

The monotonicZip application should be a very few lines of
code to subclass monotonicFile and override how the offset
is computed and how recovery is done.

What is a monotonic zip file good for? Any system that
produces a constant large number of small files in sequence
that you want to save for some reason. We have this with
write-ahead log files from a database. Depending on the
database traffic, at times of light load these files may
be only a few hundred bytes compressed, but generated in
large numbers per day. They can be much smaller than the
filesystem block size, so keeping them all around as small
files would waste tremendous space in internal fragmentation.
Appending them to a monotonic zip is much more efficient.
We have one with a long period of steadily generated log
data, tens of thousands of individual files, in a zip that
slowly grows, still in the sub-gigabyte range so far.
When it is written using tools that truly only touch it near
the end (like Python zipfile), most of the blocks never have
to be reallocated or rewritten, a special benefit if it is
stored on a copy-on-write filesystem.

A monotonic-zip app *could* produce index data for the files
within the zip, but I do not plan to (at first, anyway); it
will just treat the entire zip as one user object.

Another building-block application would be something like
amOpaqueTree ... think of a directory tree where the files
and names are all under some software control and not
individually meaningful to you. A database data directory,
subversion or git repository, etc. You are not usually looking
for just file 1865a73f9 out of this directory, you just want
to recover the whole tree as it was at the time of backup.

Compact incremental backups of such a tree can be made by
the well-known rsync tool, using its --only-write-batch
option. The "state" to save between levels can be large;
the first saved state requires space equivalent to the
tree being backed up. Later saved states are much smaller,
thanks to the --link-dest option. I have already been using
this technique for particular jobs, implemented in a Python
script, but it would be more generally useful as an Amanda
application, which would just need translating it to Perl.

> Yes, 'num_blocks * block_size' if the number of KB .

Ok. And by contrast, the <size> field in a DAR command is
actually expressed in bytes?

> Forget about collection, it was never implemented.
> COLLECTION NO means the application do not have collection, which is
> TRUE for all applications.

Ok, that makes sense; maybe that can be said on the Operations
page (right now the row for COLLECTION in the table just has
a blank Description column)

> I improved the
> http://wiki.zmanda.com/index.php/Application_API/Operations page and I
> added a new http://wiki.zmanda.com/index.php/Application_API/DAR page

That helps, thanks!

Actually, I would like to know more about the --record option.
The Operations page does not say much about it, but I assume it
is related to 'record' described for a DLE in amanda.conf, so it
has to do with the application saving local state for future
levels > 0? Are there any guidelines or conventions I should
know about that?

What is the relationship between --record and
--state-stream / --recover-dump-state-file?  I guess they are
both about saving state (in an application-determined format),
but locally for --record vs. on the server for --state-stream?

Is --state-stream only intended for use with DAR? Suppose there
is an application that does not support DAR, but does want to
support levels > 0, and so needs a way to save state. Does such
an application always use --record, or could it also work with
--stream-state? Is it a free design choice to use one method or
the other based on desire to keep the state locally or on the
server?   ... or have I misunderstood the purposes of
these different options?

Thanks,
-Chap