Re: Re: infinint, libdar

On Mon, 2004-05-03 at 11:38, Denis Corbin wrote:
> Wesley Leggette wrote:
> >[...]
> > In the terminateur
> > ------------------------------------------------------
> >[...]
> > Using this code, I considered the following cases, where "|" indicates
> > what we know to be the end of the infinint and the beginning of the size
> > field:
> > 
> > A normal two integer backwards infinint offset
> > .... ... | 0xC0
> >  - no 0xFF found, read backward one to find excess bitfield, which indicates two
> > 
> > .... ... | 0xC0 0xFF
> > - one 0xFF found, read backward one to find the excess bitfield, which indicates two more
> > 
> > .... ... | 0xFF
> > - one 0xFF found, read backward one to find the excess bitfield, WHICH IN THIS CASE WILL
> >   READ INTO THE ACTUALL INFININT, AND SCREW UP THE OFFSET
> 
> Yes. This is a bug. The bug should appear (leading to "incoherent 
> structure" or the like, and failing to read an archive), if the total 
> "Term Cap" is a multiple of 8*4 bytes long, which gives a infinint 
> multiple of 32 bytes length (=256 bits). But infinint size (when saved 
> in file) is usually a multiple of 4 bytes plus one or several bytes of 
> preambule, 5 bytes, 9 bytes, 13 bytes, 17 bytes, ... 34 bytes, ... etc. 
> None here are multiple of 32 bits... so for theses first numbers the 
> "Term Cap" does not end at byte boundary.
> 
> The problem will arise when the preamble will be a multiple of 4 bytes, 
> then we *may* have the total length of the infinint multiple of 32 
> bytes. This does not appears, before integers of 96 bits long: the 
> preamble is four bytes long (the integer is 12 bytes), and the offset of 
> the catalogue is 2^96 bytes from the beginning, thus your archive is 7.9 
> E28 bytes long so far ! :-) this is 7.1 E16 Terabytes... The biggest 
> archive I have heard about is just 1.4 Terabyte ! But the bug will be 
> fixed, sure !
> 
> Thanks for having found it ! Even if it will not be seen before a few 
> million years ! ;-)))
> 
> 
> 
> > 
> > The way I see it, this could be fixed by always outputing an excess bitfield, which would be
> > set to 0x00 if the infinint takes up an even group of 8 integers. But I didn't see the code
> > to do something like this. Is this a bug, or am I just missing something?
> 
> That's the fix I was planning to do.
> 
> > 
> > 
> > 
> > 
> > In the archive level header
> > ------------------------------------------------------
> > 
> > +---------+------+---------------+------+
> > | edition | algo | command line  | flag |
> > |         |      |               |      |
> > +---------+------+---------------+------+
> > 
> > 
> > - command line		This is obviously written as a C style string, but I am missing
> > 			how it is used in dar itself. All the archives I generate just
> > 			have "N/A" set in that field. (I admit, I've been studying the
> > 			libdar code it detail, but haven't touched the dar stuff yet).
> 
> this was used before version 1.3.0 it was containing the command-line 
> used to generate the archive. But with -B option which opens 
> command-line to included files it had less interest... so now, new 
> version of dar (>= 2.0.0) put "N/A" in place of the command-line, but 
> can still read what have been put by ancient versions.
> 
> > 
> > - EA flag		SAVED_EA_ROOT = 0x80		how is this option used in dar?
> > 			SAVED_EA_USER = 0x40		how is this option used in dar?
> 
> This flag tells wether EA have been saved or not (if a file has not EA 
> and the EA have been saved, any EA will be erased at restoration time, 
> unless -u or -U is used).
> 
> > 
> > 			SCRAMBLED = 0x20
> > 			With this, I was thinking of adding gpg encryption and maybe other
> > 			formats to my implementation. Any thoughts on changing this to
> > 			match the algo format in dar edition 04 so different encryption
> > 			schemes could be used? ;)
> 
> As you see there is room for other encrytion protocols, gpg does not fit 
> the global need, it would avoid seeking in the archive or need to 
> decrypt the whole archive just to read the contents... That's a long 
> discussion I already had, but there will be something more interesting 
> than "scrambling" one day (that's not the top priority... tread safe 
> library is).

I'm a little confused as to what to dilema is with gpg'ing files
individually as they're added to the archive. Can you clarify this for
me?

Also, would it be too much to ask that a seperate encryption type flag
be added to the next file format revision of dar? I would like to try
adding various encryption schemes to my library, but I want to keep the
library writing the standard dar format.



> 
> > 
> > 
> > Again, thanks for helping me out with this. Basically, I'm attempting to put together the
> > peices for a real linux backup program (you know, background libraries, stuff like that), and
> > at first I was just going to pipe stuff to tar and rar and whatever else seemed good. But 
> > dar caught my eye, and I got really interested in the file format itself.
> 
> Strange usually, the fact that dar uses its own file format is not 
> appreciate ! ;-) I still don't see well why you need to re-implement the 
> libdar library, while you can use it directly, or if it is just a "real 
> linux backup program" you want, do dar or kdar miss this goal ? 8-)

It's just for a school project, really. I was first working on the
backup program, but I'm more interested in the file format itself now.
The class is in java, so thus the choice of the language. I'm
implementing it in java first, and lately I've been interested in mono,
so I'll implement it in C# next.

Yeah, I know... It's weird. 

As far as dar and kdar, I use Gnome. So there you go. My program will be
a GTK app. gdar really. In the end, I'll be using kdar as a basis for my
implementation. I'll be focusing a lot on the features of dar_manager as
well (great utility!).


> 
> > 
> > I actually had a big long conversation with a friend the other day about why the catalog has to
> > be stored at the end of the archive. I was the fact that dar archives can't really be read to or
> > written to pure one way pipes like tar archives can (they can, right?). But then I was thinking that
> > it doesn't seem like such a big deal since the whole idea of dar is that it would be written to
> > random access mediums in the first place, so why not take advantage of that, right?
> 
> :-) The catalogue is built all along the backup process. It contains the 
> offset of each file. So it cannot be dumped before all files have been 
> saved. And saving it at the beggining would suppose we know the size of 
> the catalogue, while we don't even know how much file will be saved...
> 
> Saving the corresponding information after each file, would explode the 
> catalogue all along the archive, make dar archive like tar ones, where 
> you need to read them all even just to know what is in them...
> 
> Last point, dar is able to write an archive to pipe. It is not able to 
> read it from pipe. Well, it could but it would require to feed the whole 
> archive to pipe, and to merge all slices togheter for that... This is 
> what tar does, but if you have to restore a few bytes, you would spent 
> hours reading the archive through pipe ? I tried with tar, and prefered 
> something faster. The two pipes and dar_slave has the advantage to stay 
> relatively simple while it does not require you to modify an existing 
> archive split over several disks, and it will transfer only what is 
> required to restore what was asked to, so it *is* fast.

I'm not sure if I'm on the right track here, but if you've got a split
archive that you're writing to pipe, how is the slice header changed to
non terminal at the end of writing the archive? Or can you only write an
archive to pipe if it's not sliced up?

> 
> 
> > 
> > 
> > Thanks,
> > Wesley Leggette
> > 
> > 
> >[...]
> 
> Cheers,
> 
> Denis.
> 




-------------------------------------------------------
This SF.Net email is sponsored by Sleepycat Software
Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to 
deliver higher performing products faster, at low TCO.
http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3