Re: Re: last archive slice

On July 24, 2004 11:22 pm, Denis Corbin wrote:
> Johnathan Burchill wrote:
> | Hi Denis,
>
> Hello Johnathan,
>
> | In dar version 2.1.3, libdar mistakes the last slice number sometimes.
> |
> | For example, create an archive with slice sizes of 3 MB. Say this
> | results in 6 slices. Now create the archive, same basename, same
> | directory, and allow the library to overwrite the existing archive,
> | but use 5 MB slices. Suppose it results in 3 slices.
> |
> | Now try to list the archive. I get the following user interaction
>
> question:
> | "/opt/user/backups/dar_backups/test6.6.dar is a file from another set
> | of backup file, please provide the correct file."
> |
> | Then the user has to delete slices, one by one starting with the
> | highest number, and retry the list command until it works.
> |
> | Is this the expected behaviour?
>
> yes.
>
> | Shouldn't libdar see that the third slice,
> | in this example, would be the last archive slice? I have to admit,
> | it's confusing for a user who does this to know that the archive
> | consists of only three slices when six are on disk.
>
> This is a user mistake to hide under the same basename in the same
> directory slices of different archives.
>
> There is no use to keep last slices of an old archive, it only waste
> your disk space: If you miss the first slice, you won't be able to
> restore anything from your old archive. Else, if you definitively want
> to keep several archives with the same name, it is not a very good idea
> to mix their respective slices toghether in the same directory... ;-) no
> ?

That is exactly the problem I was trying to address. The slices _are_ 
absolutely useless, and just waste disk space, so why not have libdar 
remove them?

>
> | You wouldn't have to change the slice size to get this problem.
>
> Suppose you
>
> | did a full backup and got 12 CD-R-sized slices. Then during the course
> | of the following week, you delete a bunch of directories that you
> | don't need anymore, and do a full backup which has 11 CD-R-sized
> | slices. Dar will miss the fact that the last slice  #11, but will
> | still ask the user for the 12th slice.
>
> what would be the 12th slice useful for ? why would it have to stay on
> disk for ?
>
> | I see the following solutions:
> |
> | 1) libdar deletes all slices on a disk if they have the same storage
> | directory and basename as the new one being created. It might do this
> | before creating the archive, or after creating the archive. Perhaps
> | doing this after will be more efficient, since you only delete slices
> | that haven't been overwritten, if there are any.
>
> I do not agree. libdar deletes nothing. 'rm' command does. ('rm' or
> unlink() system call)

Libdar does delete files on disk if the appropriate conditions are met 
during a restoration, but I quibble :).

But libdar does overwrite archive slices! Is this not a form of deletion?
Consider the echo command:

echo "hello" > file1
echo "bye" > file1

What is file1?

cat file1
bye

It is not, "byelo", yet this is the analogous behavior that dar does when 
creating a new archive with slices, while replacing one that already 
exists. As you intend this to be the behaviour, I suggest that it is not 
the obvious or intuitive one, and at least should be documented. Apologies 
in advance if it is already in the manpage.

I conceptualize disk archives as objects, whether they are made up of 
slices or not. Making a new backup in a directory that contains an archive 
of the same name should replace the archive as a whole, not on a 
slice-by-slice basis. I do not see anything "clever" or assuming about 
that for libdar.

An archive with one slice, made with a slice size of 0, would be completely 
replaced by the new archive. By your logic you should overwrite the 
initial portion of the old archive with the new one, not completely 
replace it.

Same thing for when someone saves a file in a document processor. You don't 
expect to have to first remove the one on disk before saving what is in 
memory to the same filename.

In fact, in the API, libdar::archive is a class for which you can 
instantiate archive objects. The objects aren't individual slices, they 
are entire archives. Overwriting an old archive object with a new archive 
object with the same name, same directory, should completely replace the 
old archive object.

>
> | 2) The search algorithm gets fixed to stop on the first slice with a
> | "T" type terminating character in the header. Am I correct in
> | interpreting
>
> the
>
> | "T" to mean that the slice is the last one in the archive? I assume
> | that libdar figures out the last slice by relying on the filename, not
> | the header, although I haven't checked the sourcecode.
>
> Assuming in libdar the mistakes of the users is not a good idea. I can
> already hear some users complaining that libdar does not remove unuseful
> slices of older archives, while it does not complain anymore that extra
> slices exist and do only waste disk space.

I understand your point now. 

I am not suggesting though that libdar just stop reporting that there are 
extra useless slices, I am suggesting that it do something to stop that 
situation from happening. I will make a weaker request then, that at least 
libdar warn the user of this situation during the creation process. 

For instance, would it not make sense for libdar to report during the 
creation process that there are extra useless slices, report which ones 
they are, and ask the user whether they should be removed, or warn the 
user that they should be removed manually? You eliminate the assumption by 
asking the question.

After all, you do ask if your user intends to get into the "endless loop" 
by backing up a directory that contains the archive, with no exclusion 
file filters. You don't let it happen without the question.

>
> I don't like MS-office like programs that tend to have the prentention
> to be more clever than their users. They either get much restrictive and
> blindly forbid operation needed by more clever users than the program
> developers, or automatically do stupid things or have stupid questions
> to the user, that even less clever users find borring.

I wholeheartedly agree! :)

>
> | 3) A combination of 1) and 2).
>
> no again. If a user wants to clean its previous archive it is simple:
>
> ~  rm old_archive.*.dar
> ~  dar -c old_arhive ...
>
> nothing more nothing less. Simple, everybody understands, no hidden
> features from dar or libdar, no hidden surprises. (lib)dar is only a
> backup software.
>
> | I see a similar problem when creating archives with KDar. I have a
>
> progress
>
> | bar that periodically updates which slice is currently being written,
> | and what that slice's size is. When the "pause" between slices option
> | is chosen, and libdar asks if it okay to continue writing the next
> | slice, first it writes the "non-terminating" character to the slice
> | header, and then waits for the user to answer the question. In the
> | case where the
>
> user
>
> | is overwriting an older archive, the statusbar code runs through the
>
> files
>
> | on disk until it finds the one with the "T" in the header. This will
> | be the last one in the older archive, and the statusbar will jump to
> | that slice instead of the current one.
>
> To my point of view, this is here again a user mistake. Once you
> overwrite the first slice of the old archive, all the other slices of
> this archives are useless. Why not removing first all the slices of the
> old archive you have planned to overwrite ?

I agree, if the user removes the old archive before creating the new one, 
this situation will never arise in KDar.

What I do not understand is why, given that you admit the remaining slices 
become useless once the first one is overwritten, libdar does not remove 
those useless extra slices automatically. 

Can you imagine some situation where, for a more clever user, those extra 
slices are not useless? 

Perhaps this is a stupid question! :) 

>
> | Perhaps libdar should write the "N" to the termination type only if
> | the user says it's okay to continue, and write an "I" to the
> | termination type if the user cancels. That way we know that the "last"
> | archive slice is actually invalid, i.e. the creation process was
> | aborted, or failed in
>
> some
>
> | way.
>
> perhaps, KDAR could propose a "remove old archive slice" option ? ;-)
> option, that can simply rely on the rm command or unlink system call...
>

Agreed. Except I still think if you allow the user to overwrite an archive, 
the entire archive should be replaced, not just individual slices.

> | For the status indicator, the best solution would be for libdar to
> | have a "currentSlice" method, which could be called at any time to
> | determine the slice number of the current one being written or read.
>
> a new callback function ? ;-)

There's no need for a callback function. The method just returns the slice 
number, a libdar::infinint. I envision usage as such:

//(start a new creation thread):

libdar::archive = newArchive;
createArchiveThread *createThread = new 
createArchiveThread( newArchive, ...); 
createThread->start();

//(occasionaly check the current slice number):
while ( createThread->running() )
{
	libdar::infinint currentSlice = theArchive->currentSlice();
	//report the current slice to the user
	updateStatusBar( currentSlice );
	//Sleep and check again
	sleep( DURATION );
}

>
> | Should I file a bug report for any of this?
>
> I don't think so, do you still ?
>

I do see now that the issue is whether the user or the library should be 
removing slices. Fair enough, your philosophy of design is keep it simple 
and unassuming. Just because I cannot imagine any situation in which the 
user will ever need those "useless" extra slices, doesn't mean that 
someone else won't.

I will implement a "do not overwrite entire archive" option, that defaults 
to being off.

[...]
>
> Cheers,
> Denis.

Cheers,
JB

>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by BEA Weblogic Workshop
> FREE Java Enterprise J2EE developer tools!
> Get your free copy of BEA WebLogic Workshop 8.1 today.
> http://ads.osdn.com/?ad_id=4721&alloc_id=10040&op=click
> _______________________________________________
> Dar-libdar_api mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

-- 
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
[email protected]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQBBA2BnrXMSOtiWrO4RAvcNAKDc9IgZEtvOn9yW8s9pZAR+xbKZRQCfTjt4
C9uwh6boAVS3Y7z5Lfy6akY=
=/sVj
-----END PGP SIGNATURE-----