Re: infinint, libdar

Wesley Leggette wrote:
>[...]
> In the terminateur
> ------------------------------------------------------
>[...]
> Using this code, I considered the following cases, where "|" indicates
> what we know to be the end of the infinint and the beginning of the size
> field:
> 
> A normal two integer backwards infinint offset
> .... ... | 0xC0
>  - no 0xFF found, read backward one to find excess bitfield, which indicates two
> 
> .... ... | 0xC0 0xFF
> - one 0xFF found, read backward one to find the excess bitfield, which indicates two more
> 
> .... ... | 0xFF
> - one 0xFF found, read backward one to find the excess bitfield, WHICH IN THIS CASE WILL
>   READ INTO THE ACTUALL INFININT, AND SCREW UP THE OFFSET

Yes. This is a bug. The bug should appear (leading to "incoherent 
structure" or the like, and failing to read an archive), if the total 
"Term Cap" is a multiple of 8*4 bytes long, which gives a infinint 
multiple of 32 bytes length (=256 bits). But infinint size (when saved 
in file) is usually a multiple of 4 bytes plus one or several bytes of 
preambule, 5 bytes, 9 bytes, 13 bytes, 17 bytes, ... 34 bytes, ... etc. 
None here are multiple of 32 bits... so for theses first numbers the 
"Term Cap" does not end at byte boundary.

The problem will arise when the preamble will be a multiple of 4 bytes, 
then we *may* have the total length of the infinint multiple of 32 
bytes. This does not appears, before integers of 96 bits long: the 
preamble is four bytes long (the integer is 12 bytes), and the offset of 
the catalogue is 2^96 bytes from the beginning, thus your archive is 7.9 
E28 bytes long so far ! :-) this is 7.1 E16 Terabytes... The biggest 
archive I have heard about is just 1.4 Terabyte ! But the bug will be 
fixed, sure !

Thanks for having found it ! Even if it will not be seen before a few 
million years ! ;-)))



> 
> The way I see it, this could be fixed by always outputing an excess bitfield, which would be
> set to 0x00 if the infinint takes up an even group of 8 integers. But I didn't see the code
> to do something like this. Is this a bug, or am I just missing something?

That's the fix I was planning to do.

> 
> 
> 
> 
> In the archive level header
> ------------------------------------------------------
> 
> +---------+------+---------------+------+
> | edition | algo | command line  | flag |
> |         |      |               |      |
> +---------+------+---------------+------+
> 
> 
> - command line		This is obviously written as a C style string, but I am missing
> 			how it is used in dar itself. All the archives I generate just
> 			have "N/A" set in that field. (I admit, I've been studying the
> 			libdar code it detail, but haven't touched the dar stuff yet).

this was used before version 1.3.0 it was containing the command-line 
used to generate the archive. But with -B option which opens 
command-line to included files it had less interest... so now, new 
version of dar (>= 2.0.0) put "N/A" in place of the command-line, but 
can still read what have been put by ancient versions.

> 
> - EA flag		SAVED_EA_ROOT = 0x80		how is this option used in dar?
> 			SAVED_EA_USER = 0x40		how is this option used in dar?

This flag tells wether EA have been saved or not (if a file has not EA 
and the EA have been saved, any EA will be erased at restoration time, 
unless -u or -U is used).

> 
> 			SCRAMBLED = 0x20
> 			With this, I was thinking of adding gpg encryption and maybe other
> 			formats to my implementation. Any thoughts on changing this to
> 			match the algo format in dar edition 04 so different encryption
> 			schemes could be used? ;)

As you see there is room for other encrytion protocols, gpg does not fit 
the global need, it would avoid seeking in the archive or need to 
decrypt the whole archive just to read the contents... That's a long 
discussion I already had, but there will be something more interesting 
than "scrambling" one day (that's not the top priority... tread safe 
library is).

> 
> 
> Again, thanks for helping me out with this. Basically, I'm attempting to put together the
> peices for a real linux backup program (you know, background libraries, stuff like that), and
> at first I was just going to pipe stuff to tar and rar and whatever else seemed good. But 
> dar caught my eye, and I got really interested in the file format itself.

Strange usually, the fact that dar uses its own file format is not 
appreciate ! ;-) I still don't see well why you need to re-implement the 
libdar library, while you can use it directly, or if it is just a "real 
linux backup program" you want, do dar or kdar miss this goal ? 8-)

> 
> I actually had a big long conversation with a friend the other day about why the catalog has to
> be stored at the end of the archive. I was the fact that dar archives can't really be read to or
> written to pure one way pipes like tar archives can (they can, right?). But then I was thinking that
> it doesn't seem like such a big deal since the whole idea of dar is that it would be written to
> random access mediums in the first place, so why not take advantage of that, right?

:-) The catalogue is built all along the backup process. It contains the 
offset of each file. So it cannot be dumped before all files have been 
saved. And saving it at the beggining would suppose we know the size of 
the catalogue, while we don't even know how much file will be saved...

Saving the corresponding information after each file, would explode the 
catalogue all along the archive, make dar archive like tar ones, where 
you need to read them all even just to know what is in them...

Last point, dar is able to write an archive to pipe. It is not able to 
read it from pipe. Well, it could but it would require to feed the whole 
archive to pipe, and to merge all slices togheter for that... This is 
what tar does, but if you have to restore a few bytes, you would spent 
hours reading the archive through pipe ? I tried with tar, and prefered 
something faster. The two pipes and dar_slave has the advantage to stay 
relatively simple while it does not require you to modify an existing 
archive split over several disks, and it will transfer only what is 
required to restore what was asked to, so it *is* fast.


> 
> 
> Thanks,
> Wesley Leggette
> 
> 
>[...]

Cheers,

Denis.




-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE. 
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click