star-1.5a64 aborts on reading archives with _small_ garbage tails

Hi,
 
> Well, it seems that you did convince me as star did never complain
> if the read size modulo 512 was not zero but the amount itself
> was > 512.

You can imagine how puzzled i was when i encountered
this effect just in the moment when i tested encryption
and decryption of publicly stored backups.
That happened after lots of successfull tests without
encryption. Just bad luck.

 
> In addition, there was a problem related to calling
> 	rpm2cpio | star -xp

There is use for such ? Google ... 
... wow. There is use for everything. 


>>  I spoil a copy by appending 123 bytes :
>
> Why do you do this?

The small tail is a volume (resp. media) checksum tag. 

It looks roughly like

  magic_number  pos_in_stream  date_and_checksum_record

For media verification one can compute the MD5 of the stream
on the fly while watching for such a tag. Then one compares
the computed MD5 with the MD5 read from date_and_checksum_record.

magic_number is quite unique by an own MD5 based inner
redundancy of the whole tag. So if you find one, then you
recognize it. The question remains: is it part of the backup
data or is it a real tail tag intended for comparison ?

The number  pos_in_stream  does this distinction. It has
to match exactly the bytecount where the tag starts. 
Hardly any filetree format starts with a plain file's data.
So there is always an offset which invalidates pos_in_stream
of any tag copies in the backup.
Thus tags are quite unlikely to be mistaken for valid when
they are part of the payload data of a backup archive
or filesystem image.
A false positive is not overly fatal, after all, because
a message about mismatching MD5s would be amended later by
the hopefully encountered real tag which hopefully matches.
The time information from the tags should help the user to
find out what's going on.

A tag does not verify any inner details of the backup but it
can verify the media without any reference to the original
computer system, to the format of the backup, and without
the need to know any secrets like encryption keys.
Just put in the media and run scdbackup_verify -auto 

If backups are checked regularly then one gets an early
warning if they begin to go bad. So one can make a new
backup or start recovery efforts with the deteriorating one.


In fashionable OO terms : i derive my backup formats by
inheriting from existing backup formats and adding some
own features.

In your role as star's father, you became grandpa :))


You remember August 2004, cdwrite@ ?
"Are there objections against having the only track of a CD
 ending with 300+ kB of _ non-zero _ pad bytes rather than the
 padding provided by  cdrecord padsize=...  or  mkisofs -pad  ?"

I wanted to make use of the CD padding for storing 
checksums which would have been recognizable by their
sheer mass and redundancy. Not a safe idea to replace the
zeros by garble, we finally decided. Idea trashed.

But the part with the checksum on media lived on.


There may be two further kinds of tails :

Checksum lists which cover the backup in 64 kB blocks.
Usually they are exploited by reading one deterioring
media several times, in the hope that it works correctly
at random locations (surpringly successful with DVD-RW).
One may also produce several media with identical copies
of a backup image. These can be read one by one or 
simultaneously in order to pick correct blocks out of
several alternatives.
Finally one may produce permuted copies which expose
different data blocks to specially endangered parts of
media.

I use the block checksums to learn about the patterns 
how DVD-RW go bad (if they don't just die from OPC failure).
They pass the way randomly and irreproducibly, i found out.
Directly after re-writing, notabene, not while sitting on 
the shelf and serving as read-only data storage. (At least
for the 11 months since the first ones were burned.)

The other, super large tail format are repeated backup images
(plus checksum lists) which allow to store one image
several times on the same media. Again the  pos_in_stream
trick applies.

If you ever are bored enough to read 3000+ lines of
plain ASCII :  http://scdbackup.webframe.org/README-0.8.5
appendice VERIFY, REDUNDANCY, CRYPTO (and SNAPHOT, thank you).
You might be interested in occurences of "star_as_afio_wrapper".

 
> Note that the fix I applied set a hard EOF flag whenever such a
> condition accurs. So star (the archiver process) will continue to 
> work until the logical EOF is seen of the data ends.

I will not nibble at the end of the star archive.
star will see a complete archive with no prepended or
injected data.


> The way I implemented the fix will e.g. prevent you from
> being able to continue with multi-volume handling after this
> condition did accur.

This should currently not apply to me because of my
own multi-volume handling. It has to cope with 
ISO-9660, with CD paddings and such things. Usually
volume and media relate 1:1 , but there are 1:n and
m:1 situations.
Normally star will not get aware of anything else but a
single big star volume.

Nevertheless, a valuable hint for my further plans. 

I'm still not done with adopting star, although all features
of my afio adapter are now available with star too.
As completition i plan to offer a way how to make star
backup a whole filesystem under its own reponsibility and
having my stuff only taking care for writing to multiple
media.

Thus i will get a quick incremental desaster recovery option,
which is currently still missing in my package. Your
full restore is unbeatably simple to operate if one has the
task to restore a single complete filesystem.

That all could be done with star alone, of course, but my
users will then be able to use the output channels which
they have configured for my program.
Also i will be able to add my redundancy and encryption
features.


Have a nice day :)

Thomas