Re: open archive per level

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 22/10/2013 21:26, Tobias wrote:
> Hi,

Hello Tobias,

> 
>> Last point is implementation constraint: The way to drop catalogue
>> contents to archive or read it from archive is done by a recursive
>> method (dump() for dropping, and a specific constructor for reading)
>> common to all catalogue object types (directories, files, etc.). The
>> directory constructor should act differently having the knowledge of a
>> target path to read, whether:
>> a) - we are a directory containing the path to the target (only retain
>> sub entry of the target, temporarily create others and destroy them
>> afterward
>> b) - we are the last directory of the target path (act as today, read
>> all entries)
>> c) - we are the subdirectory of the target path (create temporary
>> objects up to its EOD)
>> d) - we are a directory "after" the last component of the target path
>> has been completely read (in which case we can stop the catalogue reading)
> 
> That's exactly the point.
> What I thought about was:
> When we are at case a) or c) we need only to know:
> 1) is it a file -> skip

yes, but in order to skip over you must known how far to skip forward
... this is only possible by reading an analyzing the following entries
in the catalogue dump... that's to say quite the same CPU processing as
building the corresponding object, but without memory requirement, I agree.

> 2) is it a directory -> increase "depth counter" -> parse directory
> And this is the point where we could save CPU cycles and memory, because
> we don't need to extract and store all information about the inode.

You don't have to allocate memory, right, but you have to parse the
catalogue dump up to the time you read the balanced End Of Directory
mark along the dump, meaning you've reach the end of that directory.

> But I don't know exactly how the "read" constructor reads the catalogue
> and what the structure built up in memory looks like.

What is important here is to understand the way the catalogue object
dumps its data into the archive. Each inode object it contains
(directory, file, pipe, symlink, hardlink, etc.) dumps in turn one after
the other the data that will be necessary to rebuilt the catalogue from
the adhoc constructor in the future. Unfortunately, there is no
Type-Length-Value information, so you cannot simply skip over a given
entry (knowing its length) you have to "reading" it in depth: filename
length vary, date, UID, GID are integers stored in a specific format
named "infinint" that can be arbitrarily long and does not suffer any
limitations, some other fields are present only under certain
circumstances, etc.

A TLV-like structure was not necessary. Moreover it would have brought
some sever limitations because is would have implies fixed length for
the Type and Length fields, which might be a huge constraint when saving
an arbitrarily large directory, for example.

> 
> What is the most time consuming part when open the archive?
> a) Is it just the reading and decompressing of the catalogue stored in the archive?

yes, it is.

> b) The interpreting of the data (what kind of inode, name, permissions, ...)?

no, this is negligible

> c) Or to build up the catalogue in memory?

this much less important than decompressing, but comes in second point
after it.

> We could only influence c) and maybe b).

Right. For c) there is already optimization layer that asks to the
system larger blocks of memory and split them for internal memory
allocation/reallocation. You can try disabling it passing
- --disable-special-alloc to ./configure and recompiling dar/libdar (make
clean / make) to see the performance impact.

There is also a caching layer in dar to avoid asking many small portion
of data from the filesystem, and ask a larger file portion each time.
This reduces the context switch and allow dar to work efficiently over a
the network where latency is important (using ssh for example).

> This depends on the format of the inode.
> Can we only read a part of the attributes?

Only if it is the last thing we expect to read from the catalogue dump
in the archive.

> 
> 
>> It will make the implementation more complicated and more subject to
>> bugs, this is a certainty. Will it boost the process? Probably but at
>> different level of gain depending on the "target path" location and
>> extension inside the archive, and assuming this is one shot operation on
>> that particular archive (seems not realistic for a GUI, more the case of
>> a script or CLI software).
> 
> Yes, this is complicated and maybe the performance boost is so small, 
> that we will loos it by opening a subdirectory.
>

I agree,

By the way, if you have any questions or need clarifications using
libdar don't hesitate to ask on this mailing-list.

> 
> 
> Regards,
> Tobias
> 

Regards,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIVAwUBUmlLiggxsL0D2LGCAQLBDhAAxX5DMtBD+hDKIkGgKfhDcvgCqEnzn6W1
xYgLrll/vG5y4ZkUDFnilFV4Xc93GWgdMmbf6V2zeX+lmDfPQhNA6bJdXmfpNp54
6nKK3pTWifD2rK+h6fsqjk9HIFGKOEjTLMNeohiYBWb7d5xRd6pgxkLlZgPMpAxh
VLyyWoWsCnCVsPFtLpjqbOcOZ36YrzNxL0W4nULIqQs9E8SgjNiiYw8AdqTNh51j
cOqeLcYjW6xgRiKcZrIFaomiKDyPQOTdgcj5KKdXQhG/tflMWsQ2P7ccNGXNR+to
yEtdhPB4vBSD78yYlwQlF8LOY4UKQWGC3TJoOPNuk5KKJe41AcfgMV+bfBsTCuba
QymzCqqynxMJaZqbiEl5QhIemKuYIFjfBAF43PWg19VTSuizZZDs9PHhOZTGvHhw
HXhb1kJC1BE4p4rw7wrp4gNcUuabuLtTPvitxivrDyebpTt8uPrc/qbAscKRyhHr
Nvxh2D60XomPbv6hnNmiwjOZJMy2L1IHHRR857WS+eQ2vDeqwmDtV40tG3xSSIL+
k76KXsIHXHNccpOX1x3Yxbv+t/iw6f47kmUUb47xDCn0Job1kYALQ28WGRuNLySE
8tqapU5UypctWSkIBFBOlOwjPb1vbOXrka1q7nHNIqp4HfcUzgOEi3z/ljnfFLlQ
DgJ7JozohMM=
=YXkK
-----END PGP SIGNATURE-----

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk