Re: open archive per level

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Le 20/10/2013 23:24, Tobias wrote:
> I understand the problem.
> 
> 1: I think memory shouldn't be the problem, so we could store the
> uncompressed catalogue in memory.

Actually, the decompression is done on-fly while reading the catalogue
and creating corresponding objects on-fly (directory objects, files
objects, etc.), so no memory is used to store "uncompressed" copy of the
catalogue dump.

here it would require copying the portion of the archive where resides
the catalogue dump into memory and then only build some of the
catalogue's objects from it.

> 
> 2: This example is from your documentation of the Dar 5 archive format:
> - toto
>    | titi
>    | tutu
>    | tata
>    |  | blup
>    |  +---
>    | boum
>    | coucou
>    +---
> 
> +-------+------+------+------+------+-----+------+--------+-----+
> | toto  | titi | tutu | tata | blup | EOD | boum | coucou | EOD |
> |       |      |      |      |      |     |      |        |     |
> +-------+------+------+------+------+-----+------+--------+-----+
> 
> When parsing the filesystem tree, would it be possible to skip the
> subdirectories when building up the catalogue in memory?
> Based on the example. You want to list the content of toto. 
> So you go toto, titi, tutu, tata. Now you are leaving the desired level
> and your going to search for the next EOD to come back to the toto
> level. So your only reading blup but don't add it to the struct in
> memory.

In the catalogue there is no "marks" or TLV-like structure that can be
used to skip over a given entry. So to know what is the next entry after
toto you have read tata entry completely, you need to understand all the
next entries up to the directory balanced EOD entry that means you have
reached the end of tata contents and are back into toto contents.

Worse, the length of a particular object is strongly dependent on its
contents. The objective was to provide flexibility and allow new fields
and new objects types (classes) to answer future needs. This is one of
the strength of the current design which allowed 8 new evolutions of
archive format without breaking backward compatibility.

The consequence is that to know when you are back to "toto" level you
need to build temporary objects up to the EOD meaning the end of
directory "tata". This work of object creation costs the same as today,
having to destroy them just afterward brings extra work, thus more time
to reach that directory (toto) contents than today.

This is only when the EOD corresponding to toto would be met that you
could stop parsing the rest of the catalogue. Here in the example this
means the end of the catalogue! OK, this example is particular, assuming
we would like to read the content of tata we would could stop reading
the archive before reading "boum". You see that even in that opposite
case we already have read more than the half of the catalogue...

Last point is implementation constraint: The way to drop catalogue
contents to archive or read it from archive is done by a recursive
method (dump() for dropping, and a specific constructor for reading)
common to all catalogue object types (directories, files, etc.). The
directory constructor should act differently having the knowledge of a
target path to read, whether:
a) - we are a directory containing the path to the target (only retain
sub entry of the target, temporarily create others and destroy them
afterward
b) - we are the last directory of the target path (act as today, read
all entries)
c) - we are the subdirectory of the target path (create temporary
objects up to its EOD)
d) - we are a directory "after" the last component of the target path
has been completely read (in which case we can stop the catalogue reading)

Assuming all this worse the effort, from the GUI you will get (let's
hope faster than today) the content of a given directory. What if the
user asks to expand another subdirectory? Woudn't we have to redo all
the process with a more complicated situation where some objects (root
directory for example) already exist and must not be duplicated?
What about the compressed archive in memory, would it have to stay in
memory up to the time the whole archive would be read (and how to know
it would be completely read?), or have it to be uncompressed each time?
You'll probably prefer the first solution because you are focused on CPU
time. But one of the drawback of dar is that it may be fond of memory
when the archive contains a lot of small files, here we'll probably not
doubled the memory requirement but will not help that many users that
find it annoying on embedded systems for example...



> Of course this is a really simple example. In reale world you have some
> more subdirs and determining the point where your back on the desired
> level will not that easy. 
> But do you understand what I mean?

I guess so, yes.

> Shouldn't this boost up the process, because you don't have to add
> hundreds of (in this moment uninteresting) inodes to the struct in
> memory?

It will make the implementation more complicated and more subject to
bugs, this is a certainty. Will it boost the process? Probably but at
different level of gain depending on the "target path" location and
extension inside the archive, and assuming this is one shot operation on
that particular archive (seems not realistic for a GUI, more the case of
a script or CLI software).

> 
> Ok, I admit when you want to build up the whole tree, this technique
> costs more CPU cycles. But then I only want to extract one directory
> from the archive I don't wont to wate over half a minute until the
> catalogue is read.

I understand the need. I hope you understand the implications and
complexity doing so, waiting for your objections :-)

> 
> Tobias
> 

Regards,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIVAwUBUmTYAAgxsL0D2LGCAQI4jQ/+Kjb2JEwrNk3kcWbHV7O7M9HmJ3S5XTEz
JJs364PZzudY97xXA+WeSF8J8sRpbQiF5wfbRAguynggjJEyOwMks8p93HfHrCX5
Pz4edoD0xxKG+40/un1kYqUtnGL5ShVRg+6Ao+ilQwXpQN8HlNkYlMdC20uzNoUL
uByLQjfl14h7KTvhmh+RsbRg4xn0npSnBY8X6H1ow+ZoycWwFKy0DDuN3DUmnsI3
hWkBXx3tTfhfFhcgS1c/90JrMBBubZ3n/QsTvJgSmxLvmuE7nd2LnT2KQz3XeroH
cccCFHOVY9FOn3bNbshZPOJDPmLvxD9HyiHBiOl+a4DfpmR/cNZRsH8od1dZjhie
U/YPy5HRk24lJLTGj3iB2VUMgi4+oEUDLYhQGGUnkKq4kKrpg1taZND2MabZi78n
H5604asOKRtsroEDAzMXPFRqeJ+zio0T7tFORGtu7J5ix0hEmtcffl27OUGWccVH
BgcP8ZUky/Uq+AMmU+Cxa0NuBb6R8L07L0nKarMMf8Cqe0Zp5F9QTwP00gX4CrjC
U64XGPR+nVRiHBYW8boH+84ZXFnedJ9yNkbB1Yx3nCGcggjOl1b+nhb33SW+UZd+
tYwa8IQxVS/IwK0exOYsQe+AaLHtiDlEBhh/+K/NBybvl1vjI/yuDCBBcKh3Y4NQ
NaPK3ixsYxs=
=SpXC
-----END PGP SIGNATURE-----

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk