Draft dar 3 specification
Wesley Leggette <lists-+42/j/[email protected]> Sat, 05 Mar 2005 09:49:49 +0100
| Newsgroups | gmane.comp.sysutils.backup.dar.general |
|---|---|
| Message-ID | <1110012589.26300.48.camel@localhost> |
Attached is a draft specification of the dar file format version 3 (not 4, the latest version). It's not complete yet. Right now, I'm still working on the data section specifics. I'm trying to understand and document how the compression and encryption affects the binary data and extended attribute data. This is the section I need assistance with now. Everything else is pretty complete. There are small details I'd like to add, but I'm pretty clear on how they work. -- Wesley Leggette <lists-+42/j/[email protected]>
spec.html
(text/html, 44.5 KB)
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.3.3: http://docutils.sourceforge.net/" />
<title></title>
<link rel="stylesheet" href="default.css" type="text/css" />
</head>
<body>
<div class="document">
<div class="section" id="dar-format-specification">
<h1><a name="dar-format-specification">Dar Format Specification</a></h1>
<table class="field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field"><th class="field-name">Author:</th><td class="field-body">Wesley Leggette</td>
</tr>
<tr class="field"><th class="field-name">Revision:</th><td class="field-body">$Revision: 1.1 $</td>
</tr>
<tr class="field"><th class="field-name">Copyright:</th><td class="field-body">GNU General Public License</td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="dar-format-version-3">
<h1><a name="dar-format-version-3">Dar Format Version 3</a></h1>
<p><strong>Dar</strong> was designed by Denis Corbin as part of the <a class="reference" href="http://dar.linux.free.fr/">Dar Archive</a> application
(DAR). All materials related to DAR are copyright Denis Corbin and are
available under the terms of the GNU General Public License.</p>
<p>Dar is implemented by the Dar Library (libdar), which is available only under
the terms of the GNU General Public License.</p>
<div class="section" id="summary">
<h2><a name="summary">Summary</a></h2>
<p>summary summary</p>
</div>
<div class="section" id="archive-structure">
<h2><a name="archive-structure">Archive Structure</a></h2>
<pre class="literal-block">
|-- first slice size ----------------------------|
+-------------+---------+------------------------+
| slice + ext | archive | file data + EA |
| header | header | |
+-------------+---------+------------------------+
^ ^
zero offset at archive level offset (A)
|-- slice size ---------------------------------------|
+--------+--------------------------------------------+
| slice | file data + EA |
| header | |
+--------+--------------------------------------------+
^
offset (A)
+--------+--------------------------------------------+
| slice | file data + EA |
| header | |
+--------+--------------------------------------------+
|-- final slice size -----------------------------|
+--------+---------------------+-----------+------+
| slice | file data + EA | catalogue | term |
| header | | | |
+--------+---------------------+-----------+------+
^
catalog offset
</pre>
<p>Dar archives are written sequentially with one exception: During writing, the
<tt class="literal"><span class="pre">last</span> <span class="pre">flag</span></tt> byte is initially set to <tt class="literal"><span class="pre">FLAG_TERMINAL</span></tt> and is reset to
<tt class="literal"><span class="pre">FLAG_NON_TERMINAL</span></tt> on transition to a new slice.</p>
<p>When reading, implementations should read the first slice header, the archive
header, then the terminator on the last slice and the catalog. The implementation
may then proceed with the desired operation. Besides the mandatory access of the
first and last slices, other slices may be requested only as needed.</p>
<p>All offsets stored in the archive level are based from a zero offset at the end
of the first slice header before the archive header.</p>
<p>The abstracted archive level offset is resolved using several values.
<tt class="literal"><span class="pre">FIRST_SLICE_SIZE</span></tt> is queried from the file descriptor of that slice,
<tt class="literal"><span class="pre">SLICE_SIZE</span></tt> is set to the value stored in the slice header's size
extension (if present) or set equal to the <tt class="literal"><span class="pre">FIRST_SLICE_SIZE</span></tt>. <tt class="literal"><span class="pre">LAST_SLICE_SIZE</span></tt>
and <tt class="literal"><span class="pre">LAST_SLICE_NUMBER</span></tt> are queried from the file descriptor and the file name
respectively of the last slice file. <tt class="literal"><span class="pre">FIRST_SLICE_OFFSET</span></tt>
is set to the current position after reading the first slice header.
<tt class="literal"><span class="pre">SLICE_OFFSET</span></tt> is set equal to <tt class="literal"><span class="pre">SLICE_HEADER_SIZE</span></tt> (the size of a slice
header without extensions, currently 16 bytes). Any archive level offset can be
translated with these values</p>
<div class="section" id="slice-level">
<h3><a name="slice-level">Slice Level</a></h3>
<pre class="literal-block">
+--------+-------------------------------------------+
| header | data |
| | |
+--------+-------------------------------------------+
</pre>
<p>Dar archives are placed in a sort of logical envelope called a slice, allowing
the archive to be spanned across one or more files. Each slice consists of a
slice header followed by data.</p>
<div class="section" id="slice-header">
<h4><a name="slice-header">Slice Header</a></h4>
<pre class="literal-block">
+-------+----------+------+-----------+................+
| magic | internal | last | extension | extension |
| num. | name | flag | flag | data |
+-------+----------+------+-----------+................+
</pre>
<p>The slice header, located at the beginning of every slice file, identifies
the file as a dar archive, matches slices together, and indicates other
basic file information.</p>
<dl>
<dt><tt class="literal"><span class="pre">magic</span> <span class="pre">number</span></tt></dt>
<dd>Fixed value indicating file type.</dd>
<dt><tt class="literal"><span class="pre">internal</span> <span class="pre">name</span></tt></dt>
<dd>Unique identifier for a given archive. This machine generated name should
be guaranteed to be unique. The name is used to verify that two slices
belong to the same archive.</dd>
<dt><tt class="literal"><span class="pre">last</span> <span class="pre">flag</span></tt></dt>
<dd>Indicates if the slice is the last of the archive.</dd>
<dt><tt class="literal"><span class="pre">extension</span> <span class="pre">flag</span></tt></dt>
<dd>Indicates if an extension field is present. Currently, only one extension
field can be present at any time.</dd>
<dt><tt class="literal"><span class="pre">(extension</span> <span class="pre">data)</span></tt></dt>
<dd>Extension data.</dd>
</dl>
<table border class="table">
<colgroup>
<col width="25%" />
<col width="31%" />
<col width="44%" />
</colgroup>
<tbody valign="top">
<tr><td>magic number</td>
<td><p class="first">4 byte field</p>
<p class="last">MSB order</p>
</td>
<td><p class="first"><tt class="literal"><span class="pre">SAUV_MAGIC_NUMBER</span> <span class="pre">=</span> <span class="pre">123</span></tt></p>
<p class="last"><tt class="literal"><span class="pre">0x{</span> <span class="pre">00</span> <span class="pre">00</span> <span class="pre">00</span> <span class="pre">7B</span> <span class="pre">}</span></tt></p>
</td>
</tr>
<tr><td>internal name</td>
<td>10 byte field</td>
<td>In libdar, internal name is
generated from system time
appended by process id.</td>
</tr>
<tr><td>last flag</td>
<td>1 byte field</td>
<td><p class="first"><tt class="literal"><span class="pre">FLAG_NON_TERMINAL</span> <span class="pre">=</span> <span class="pre">'N'</span> <span class="pre">0x{</span> <span class="pre">4E</span> <span class="pre">}</span></tt></p>
<p class="last"><tt class="literal"><span class="pre">FLAG_TERMINAL</span> <span class="pre">=</span> <span class="pre">'T'</span> <span class="pre">0x{</span> <span class="pre">54</span> <span class="pre">}</span></tt></p>
</td>
</tr>
<tr><td>extension flag</td>
<td>1 byte field</td>
<td><p class="first"><tt class="literal"><span class="pre">EXTENSION_NO</span> <span class="pre">=</span> <span class="pre">'N'</span> <span class="pre">0x{</span> <span class="pre">4E</span> <span class="pre">}</span></tt></p>
<p class="last"><tt class="literal"><span class="pre">EXTENSION_SIZE</span> <span class="pre">=</span> <span class="pre">'S'</span> <span class="pre">0x{</span> <span class="pre">53</span> <span class="pre">}</span></tt></p>
</td>
</tr>
<tr><td>extension data</td>
<td>Variable size field</td>
<td>Extension specified data.</td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="extensions">
<h4><a name="extensions">Extensions</a></h4>
<pre class="literal-block">
+-----------+----------------+
| extension | extension |
| flag | data |
+-----------+----------------+
</pre>
<p>Currently the only valid slice level header extension is the <tt class="literal"><span class="pre">EXTENSION_SIZE</span></tt>
header, which indicates that the first slice is set to be different than the
other slices. The first file size is determined from the file itself
and the size of other slices is indicated in the extension data. Last slice
size is also determined from that slice file itself.</p>
<pre class="literal-block">
EXTENSION_SIZE
{
char flag = 'S';
infinint size; // size of following slices
}
</pre>
</div>
</div>
<div class="section" id="archive-level">
<h3><a name="archive-level">Archive Level</a></h3>
<pre class="literal-block">
+---------+-----------------------------------------+-----------+------+
| archive | data | catalogue | term |
| header | | | |
+---------+-----------------------------------------+-----------+------+
</pre>
<p>An archive may be split into multiple slices. The slices are handled
exclusively at the slice level. The archive level is accessed by logically
concatenated; the zero offset starts after the first slice header and
continues, jumping over each slice header.</p>
</div>
<div class="section" id="archive-header">
<h3><a name="archive-header">Archive Header</a></h3>
<pre class="literal-block">
+---------+-------+---------------+------+
| format | comp. | command line | flag |
| version | algo | | |
+---------+-------+---------------+------+
</pre>
<dl>
<dt><tt class="literal"><span class="pre">format</span> <span class="pre">version</span></tt></dt>
<dd>Indicates the format version of this archive.</dd>
<dt><tt class="literal"><span class="pre">compress</span> <span class="pre">algorithm</span></tt></dt>
<dd>Indicates the compression algorithm used.</dd>
<dt><tt class="literal"><span class="pre">command</span> <span class="pre">line</span></tt></dt>
<dd>The command line string used when originally making the archive. Usage of
this field is <strong>DEPRECATED</strong>.</dd>
<dt><tt class="literal"><span class="pre">flag</span></tt></dt>
<dd>Indicates if root or user Extended Attributes are saved; indicates if the
scramble weak encryption algorithm is used. If Extended Attributes are
stored it is optional to restore these values to disk. This flag was added
in format version 02.</dd>
</dl>
<table border class="table">
<colgroup>
<col width="25%" />
<col width="31%" />
<col width="44%" />
</colgroup>
<tbody valign="top">
<tr><td>format version</td>
<td><p class="first">3 byte field</p>
<p class="last">null terminated string</p>
</td>
<td><p class="first"><tt class="literal"><span class="pre">SUPPORTED_VERSION</span> <span class="pre">=</span> <span class="pre">"03"</span></tt></p>
<p class="last"><tt class="literal"><span class="pre">0x{</span> <span class="pre">30</span> <span class="pre">33</span> <span class="pre">00</span> <span class="pre">}</span></tt></p>
</td>
</tr>
<tr><td>compress algorithm</td>
<td>1 byte field</td>
<td><p class="first"><tt class="literal"><span class="pre">none</span> <span class="pre">=</span> <span class="pre">'n'</span> <span class="pre">0x{</span> <span class="pre">6E</span> <span class="pre">}</span></tt></p>
<p><tt class="literal"><span class="pre">zip</span> <span class="pre">=</span> <span class="pre">'p'</span> <span class="pre">0x{</span> <span class="pre">70</span> <span class="pre">}</span></tt></p>
<p><tt class="literal"><span class="pre">gzip</span> <span class="pre">=</span> <span class="pre">'z'</span> <span class="pre">0x{</span> <span class="pre">7A</span> <span class="pre">}</span></tt></p>
<p class="last"><tt class="literal"><span class="pre">bzip2</span> <span class="pre">=</span> <span class="pre">'y'</span> <span class="pre">0x{</span> <span class="pre">79</span> <span class="pre">}</span></tt></p>
</td>
</tr>
<tr><td>command line</td>
<td><p class="first">Variable length field</p>
<p class="last">null terminated string</p>
</td>
<td><p class="first"><tt class="literal"><span class="pre">default</span> <span class="pre">=</span> <span class="pre">"N/A"</span></tt></p>
<p class="last"><tt class="literal"><span class="pre">0x{</span> <span class="pre">4E</span> <span class="pre">2F</span> <span class="pre">41</span> <span class="pre">00</span> <span class="pre">}</span></tt></p>
</td>
</tr>
<tr><td>flag</td>
<td>1 byte field</td>
<td><p class="first"><tt class="literal"><span class="pre">SAVED_EA_ROOT</span> <span class="pre">=</span> <span class="pre">0x80</span></tt></p>
<p><tt class="literal"><span class="pre">SAVED_EA_USER</span> <span class="pre">=</span> <span class="pre">0x40</span></tt></p>
<p class="last"><tt class="literal"><span class="pre">SCRAMBLED</span> <span class="pre">=</span> <span class="pre">0x20</span></tt></p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="stored-data">
<h3><a name="stored-data">Stored Data</a></h3>
<p><strong>FIXME: This section may be incorrect.</strong></p>
<pre class="literal-block">
....--+---------------------+----+------------+-----------+----+---....
| file data | EA | file data | file data | EA |
| (may be compressed) | | (no EA) | | |
....--+---------------------+----+------------+-----------+----+---....
</pre>
<p>Data is stored as a stream of file contents followed by Extended Attributes
if present. Offsets to file data and Extended Attributes are stored in the
catalog at the end of the archive. Data is not stored in a solid format,
meaning that each file and set of Extended Attributes are encrypted and
compressed seperately and padded until the next block. Block sizes, which
are generally irrelevant, are determined by specific compression and
encryption algorithms.</p>
<pre class="literal-block">
+----------------------------------------------------------+
| plain file data |
| |
+----------------------------------------------------------+
+----------------------------------------------------------+...+
| plain file data | |
| | |
+----------------------------------------------------------+...+
+------------------------------------------+
| compressed file data |
| |
+------------------------------------------+
+------------------------------------------+....+
| compressed file data | |
| | |
+------------------------------------------+....+
+-----------------------------------------------+
| encrypted file data |
| |
+-----------------------------------------------+
+-----------------------------------------------+----------------+
| encrypted file data | encrypted EA |
| | |
+-----------------------------------------------+----------------+
</pre>
<p>In the example above, a file is padded until it is a multiple of the compression
algorithm's block size. After compression, the resulting data is padded again,
then encrypted. Any Extended Attributes are processed in the same way and dumped
after the encrypted file data.</p>
<div class="section" id="ea-data-storage-format">
<h4><a name="ea-data-storage-format">EA Data Storage Format</a></h4>
<p><strong>FIXME: This section may be incorrect.</strong></p>
<p>qsdlf</p>
</div>
</div>
<div class="section" id="catalogue">
<h3><a name="catalogue">Catalogue</a></h3>
<p>The catalog contains all inode, directory structure, and hard link information.
The directory structure is stored in a simple way: The inode of a directory
comes first, the inode of the files it contains, then a special end of directory
entry.</p>
<p>Consider the following tree:</p>
<pre class="literal-block">
- toto
| titi
| tutu
| tata
| | blup
| +---
| boum
| coucou
+---
</pre>
<p>The following sequence would be generated for the catalog:</p>
<pre class="literal-block">
+-------+------+------+------+------+-----+------+--------+-----+
| toto | titi | tutu | tata | blup | EOD | boum | coucou | EOD |
| | | | | | | | | |
+-------+------+------+------+------+-----+------+--------+-----+
</pre>
<p>The first catalog entry in a dar archive is a special "root" directory entry.
This entry serves as a marker; its presence must be verified and it must not
be restored by implementations. The
root directory entry's values are shown below. See the format
information for <tt class="literal"><span class="pre">directory</span></tt> entries for more information.</p>
<pre class="literal-block">
ROOT_DIRECTORY
{
char signature = 'd';
char* name_string = "root";
char EA_flag = ea_none; // 0x00
uint16_t UID = 0;
uint16_t GID = 0;
uint16_t permissions = 0;
infinint atime = 0;
infinint mtime = 0;
}
</pre>
<div class="section" id="entry-types">
<h4><a name="entry-types">Entry Types</a></h4>
<p>The first byte of catalog entries indicate the entry type and its status. An
entry has a status of <tt class="literal"><span class="pre">s_saved</span></tt> if full file data is saved in the archive.
This is the normal situation. When an archive catalog is isolated from a given
archive all entries in the extracted catalog do not have a saved status.</p>
<p>Saved status is indicated by unsetting the most significant bit of the entry
type byte. An entry does not have a saved status when this bit is set:</p>
<pre class="literal-block">
+-------------------------------+
| 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
+---+---+---+---+---+---+---+---+
| 0 | ASCII character | saved status
| 1 | ASCII character | unsaved status
+---+---+---+---+---+---+---+---+
</pre>
<p>Entry formats can be arranged in a hierarchial format. Higher level entry types
are contained as the header of deriving types. The highest level type is the
<tt class="literal"><span class="pre">entree</span></tt>, which contains the entry signature.</p>
<pre class="literal-block">
- entree
| EOD
| nomme
| | hard link label
+- | deleted file marker
| inode
| | symbolic link
+- | device
| | character special device
| | block special device
| +---
| directory
| ignored directory
| file
| | file label
| +---
| socket
| pipe
+---
</pre>
<p>The entry signature, while part of the <tt class="literal"><span class="pre">entree</span></tt> entry type, is defined by the
entry that is stored in the catalog. There are 11 valid entry types. This
includes the 7 standard Posix file types and 4 catalog specific types.</p>
<table border class="table">
<colgroup>
<col width="33%" />
<col width="22%" />
<col width="21%" />
<col width="24%" />
</colgroup>
<thead valign="bottom">
<tr><th>Entry Type</th>
<th>Signature</th>
<th>Saved Status</th>
<th>Unsaved Status</th>
</tr>
</thead>
<tbody valign="top">
<tr><td>regular file</td>
<td><tt class="literal"><span class="pre">f</span></tt></td>
<td><tt class="literal"><span class="pre">0x66</span></tt></td>
<td><tt class="literal"><span class="pre">0xE6</span></tt></td>
</tr>
<tr><td>symbolic link</td>
<td><tt class="literal"><span class="pre">l</span></tt></td>
<td><tt class="literal"><span class="pre">0x6C</span></tt></td>
<td><tt class="literal"><span class="pre">0xEC</span></tt></td>
</tr>
<tr><td>character device</td>
<td><tt class="literal"><span class="pre">c</span></tt></td>
<td><tt class="literal"><span class="pre">0x63</span></tt></td>
<td><tt class="literal"><span class="pre">0xE3</span></tt></td>
</tr>
<tr><td>block device</td>
<td><tt class="literal"><span class="pre">b</span></tt></td>
<td><tt class="literal"><span class="pre">0x62</span></tt></td>
<td><tt class="literal"><span class="pre">0xE2</span></tt></td>
</tr>
<tr><td>pipe device</td>
<td><tt class="literal"><span class="pre">p</span></tt></td>
<td><tt class="literal"><span class="pre">0x70</span></tt></td>
<td><tt class="literal"><span class="pre">0xF0</span></tt></td>
</tr>
<tr><td>socket device</td>
<td><tt class="literal"><span class="pre">s</span></tt></td>
<td><tt class="literal"><span class="pre">0x73</span></tt></td>
<td><tt class="literal"><span class="pre">0xF3</span></tt></td>
</tr>
<tr><td>directory</td>
<td><tt class="literal"><span class="pre">d</span></tt></td>
<td><tt class="literal"><span class="pre">0x64</span></tt></td>
<td><tt class="literal"><span class="pre">0xE4</span></tt></td>
</tr>
<tr><td>end of directory</td>
<td><tt class="literal"><span class="pre">z</span></tt></td>
<td><tt class="literal"><span class="pre">0x7A</span></tt></td>
<td><tt class="literal"><span class="pre">0xFA</span></tt></td>
</tr>
<tr><td>deleted file marker</td>
<td><tt class="literal"><span class="pre">x</span></tt></td>
<td><tt class="literal"><span class="pre">0x78</span></tt></td>
<td><tt class="literal"><span class="pre">0xF8</span></tt></td>
</tr>
<tr><td>hard link label</td>
<td><tt class="literal"><span class="pre">h</span></tt></td>
<td><tt class="literal"><span class="pre">0x68</span></tt></td>
<td><tt class="literal"><span class="pre">0xE8</span></tt></td>
</tr>
<tr><td>regular file label</td>
<td><tt class="literal"><span class="pre">e</span></tt></td>
<td><tt class="literal"><span class="pre">0x65</span></tt></td>
<td><tt class="literal"><span class="pre">0xE5</span></tt></td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="entree-format">
<h4><a name="entree-format">Entree Format</a></h4>
<pre class="literal-block">
+-----+
| sig |
| |
+-----+
</pre>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature character encoding entry type and status.</dd>
</dl>
</div>
<div class="section" id="eod-format">
<h4><a name="eod-format">EOD Format</a></h4>
<pre class="literal-block">
+-----+
| sig |
| |
+-----+
</pre>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature is '<tt class="literal"><span class="pre">z</span></tt>'. Indicates <tt class="literal"><span class="pre">end</span> <span class="pre">of</span> <span class="pre">directory</span></tt> entry.</dd>
</dl>
</div>
<div class="section" id="nomme-format">
<h4><a name="nomme-format">Nomme Format</a></h4>
<pre class="literal-block">
+-----+-------------+
| sig | name_string |
| | |
+-----+-------------+
</pre>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature is defined by the specific entry type.</dd>
<dt><tt class="literal"><span class="pre">name_string</span></tt></dt>
<dd>Null terminated string containing the name of the object (path not included).</dd>
</dl>
</div>
<div class="section" id="hard-link-label-format">
<h4><a name="hard-link-label-format">Hard Link Label Format</a></h4>
<pre class="literal-block">
+--------------+-----------+
| NOMME | etiquette |
| header dump | |
+--------------+-----------+
</pre>
<p>When more than one hard link for a given file is saved in an archive the first
occurance of the file will be saved as a special "label" entry matching the
file type. For example, regular files with multiple hard links will first be
saved as a <tt class="literal"><span class="pre">file</span> <span class="pre">label</span></tt> entry. Later occurances, regardless of the file
type, will be saved as a <tt class="literal"><span class="pre">hard</span> <span class="pre">link</span> <span class="pre">label</span></tt> entry with a matching label number.</p>
<p>Currently regular files are the sole file type for whom multiple hard links can
be saved.</p>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature is '<tt class="literal"><span class="pre">h</span></tt>'.</dd>
<dt><tt class="literal"><span class="pre">etiquette</span></tt></dt>
<dd>Infinint containing a number that is unique for each inode saved in the
archive. This number is not the same as the filesystem inode number.
Implementations must keep an inode table to match label numbers during
save and restore operations.</dd>
</dl>
</div>
<div class="section" id="deleted-file-marker-format">
<h4><a name="deleted-file-marker-format">Deleted File Marker Format</a></h4>
<pre class="literal-block">
+--------------+------+
| NOMME | orig |
| header dump | sig |
+--------------+------+
</pre>
<p>Files deleted since a given reference archive are tracked with <tt class="literal"><span class="pre">deleted</span> <span class="pre">file</span>
<span class="pre">marker</span></tt> entries. The original catalog entry is removed and replaced with this
type of entry. The signature of the original entry is appended.</p>
<p><strong>FIXME: How are deleted directories handled?</strong></p>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature is '<tt class="literal"><span class="pre">x</span></tt>'.</dd>
<dt><tt class="literal"><span class="pre">original_signature</span></tt></dt>
<dd>The signature of the original file before it was deleted.</dd>
</dl>
</div>
<div class="section" id="inode-format">
<h4><a name="inode-format">Inode Format</a></h4>
<pre class="literal-block">
+--------------+------+-----+-----+------+-------+--------\
| NOMME | EA | UID | GID | perm | atime | mtime /
| header dump | flag | | | | | \
+--------------+------+-----+-----+------+-------+--------/
\............+.........+..........+
/ *ea_offset | *ea_crc | ea_ctime |
\ | | |
/............+.........+..........+
</pre>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature is defined by the specific inode entry type.</dd>
<dt><tt class="literal"><span class="pre">EA</span> <span class="pre">flag</span></tt></dt>
<dd>1 byte inode flag; values are <tt class="literal"><span class="pre">ea_none</span> <span class="pre">=</span> <span class="pre">0x00</span></tt> when no Extended Attributes are
stored, <tt class="literal"><span class="pre">ea_partial</span> <span class="pre">=</span> <span class="pre">0x01</span></tt> when partial Extended Attribute information
is stored, <tt class="literal"><span class="pre">ea_full</span> <span class="pre">=</span> <span class="pre">0x02</span></tt> when Extended Attributes are fully stored.
Extended Attributes are "partially" stored when the catalog is isolated.
In such cases, only the inode change time is stored, not the attribute
data.</dd>
<dt><tt class="literal"><span class="pre">UID</span></tt></dt>
<dd>16-bit word containing Posix user identification number.</dd>
<dt><tt class="literal"><span class="pre">GID</span></tt></dt>
<dd>16-bit word containing Posix group identification number.</dd>
<dt><tt class="literal"><span class="pre">permissions</span></tt></dt>
<dd>16-bit word containing Posix mode and permissions. See <tt class="literal"><span class="pre">man</span> <span class="pre">2</span> <span class="pre">chmod</span></tt> for
more information.</dd>
<dt><tt class="literal"><span class="pre">atime</span></tt></dt>
<dd>Infinint containing time of last access. See <tt class="literal"><span class="pre">man</span> <span class="pre">2</span> <span class="pre">stat</span></tt> for more information.</dd>
<dt><tt class="literal"><span class="pre">mtime</span></tt></dt>
<dd>Infinint containing time of last modification. See <tt class="literal"><span class="pre">man</span> <span class="pre">2</span> <span class="pre">stat</span></tt> for more
information.</dd>
<dt><tt class="literal"><span class="pre">ea_offset</span></tt></dt>
<dd>Infinint containing the offset to Extended Attribute data. This field is present
only when full Extended Attribute information is stored.</dd>
<dt><tt class="literal"><span class="pre">ea_crc</span></tt></dt>
<dd>16-bit word containing CRC sum of Extended Attribute data. This field is present
only when full Extended Attribute information is stored.</dd>
<dt><tt class="literal"><span class="pre">ea_ctime</span></tt></dt>
<dd>Infinint containing the time of last Extended Attribute modification. On Linux,
this is the same as the inode's <tt class="literal"><span class="pre">ctime</span></tt>. This field is always present when
Extended Attributes are saved.</dd>
</dl>
</div>
<div class="section" id="symbolic-link-format">
<h4><a name="symbolic-link-format">Symbolic Link Format</a></h4>
<pre class="literal-block">
+---------------------+----------------------------------------+
| INODE | target_string |
| header dump | |
+---------------------+----------------------------------------+
</pre>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature is '<tt class="literal"><span class="pre">l</span></tt>'</dd>
<dt><tt class="literal"><span class="pre">target_string</span></tt></dt>
<dd>Null terminated string containing the symbolic link target as it is saved on
the file system.</dd>
</dl>
</div>
<div class="section" id="device-format">
<h4><a name="device-format">Device Format</a></h4>
<pre class="literal-block">
+---------------------\.......+.......+
| INODE / major | minor |
| header dump \ | |
+---------------------/.......+.......+
</pre>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature is '<tt class="literal"><span class="pre">c</span></tt>' when saving a character special device file, '<tt class="literal"><span class="pre">b</span></tt>'
for block special device files.</dd>
<dt><tt class="literal"><span class="pre">major</span></tt></dt>
<dd>16-bit word containing major device number. This ensures compatibility when
systems move to 16-bit major and minor device numbers. This field is not
present when device data is not saved, which occurs when the catalog is
isolated.</dd>
<dt><tt class="literal"><span class="pre">minor</span></tt></dt>
<dd>16-bit word containing minor device number. This ensures compatibility when
systems move to 16-bit major and minor device numbers. This field is not
presend when device data is not saved, which occurs when the catalog is
isolated.</dd>
</dl>
</div>
<div class="section" id="directory-format">
<h4><a name="directory-format">Directory Format</a></h4>
<pre class="literal-block">
+---------------------\...................\-----+
| INODE / catalog / EOD |
| header dump \ entries \ |
+---------------------/.................../-----+
</pre>
<p>Directories are stored as <tt class="literal"><span class="pre">inode</span></tt> entries with a signature '<tt class="literal"><span class="pre">d</span></tt>'. The name
of the directory, without path information, is stored in the <tt class="literal"><span class="pre">nomme</span></tt> header.
The directory contents are dumped sequentially after the directory entry, after
which an <tt class="literal"><span class="pre">end</span> <span class="pre">of</span> <span class="pre">directory</span></tt> entry must be inserted. For an empty directory an
<tt class="literal"><span class="pre">end</span> <span class="pre">of</span> <span class="pre">directory</span></tt> entry is inserted directly after the directory entry.</p>
<p>In a dar catalog, the</p>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature is '<tt class="literal"><span class="pre">d</span></tt>'.</dd>
</dl>
</div>
<div class="section" id="ignored-directory-format">
<h4><a name="ignored-directory-format">Ignored Directory Format</a></h4>
<pre class="literal-block">
+---------------------+-----+
| INODE | EOD |
| header dump | |
+---------------------+-----+
</pre>
<p>The inclusion of stub entries for pruned directories is not required for
implementations and can be implemented as an optional parameter. If stubs are
included, they are treated like empty directories.</p>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature is '<tt class="literal"><span class="pre">d</span></tt>'.</dd>
</dl>
</div>
<div class="section" id="file-format">
<h4><a name="file-format">File Format</a></h4>
<pre class="literal-block">
+---------------------+-------\.........+..............\------+
| INODE | size / offset | storage_size / crc |
| header dump | \ | \ |
+---------------------+-------/.........+............../------+
</pre>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature is '<tt class="literal"><span class="pre">f</span></tt>'.</dd>
<dt><tt class="literal"><span class="pre">size</span></tt></dt>
<dd>Infinint containing the original size of the file.</dd>
<dt><tt class="literal"><span class="pre">offset</span></tt></dt>
<dd>Infinint containing the offset to the file data. This field is not present when
the file data is not saved, which occurs when the catalog is isolated.</dd>
<dt><tt class="literal"><span class="pre">storage_size</span></tt></dt>
<dd>Infinint containing the new size of the file saved in the archive. This field
is not present when the file data is not saved, which occurs when the catalog
is isolated.</dd>
<dt><tt class="literal"><span class="pre">crc</span></tt></dt>
<dd>16-bit word containing CRC sum of the original file data.</dd>
</dl>
</div>
<div class="section" id="file-label-format">
<h4><a name="file-label-format">File Label Format</a></h4>
<pre class="literal-block">
+--------------------------------------+-----------+
| FILE | etiquette |
| header dump | |
+--------------------------------------+-----------+
</pre>
<p>When more than one hard link for a given regular file is saved in an archive
the first occurance of the file will be appended with a label indicating an
archive wide unique number. Later occurances will be saved as a <tt class="literal"><span class="pre">hard</span> <span class="pre">link</span>
<span class="pre">label</span></tt> entry with the matching label number.</p>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature is '<tt class="literal"><span class="pre">e</span></tt>'.</dd>
<dt><tt class="literal"><span class="pre">etiquette</span></tt></dt>
<dd>Infinint containing a number that is unique for each inode saved in the
archive. This number is not the same as the filesystem inode number.
Implementations must keep an inode table to match label numbers during
save and restore operations.</dd>
</dl>
</div>
<div class="section" id="socket-format">
<h4><a name="socket-format">Socket Format</a></h4>
<pre class="literal-block">
+---------------------+
| INODE |
| header dump |
+---------------------+
</pre>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature is '<tt class="literal"><span class="pre">s</span></tt>'.</dd>
</dl>
</div>
<div class="section" id="pipe-fifo-format">
<h4><a name="pipe-fifo-format">Pipe (FIFO) Format</a></h4>
<pre class="literal-block">
+---------------------+
| INODE |
| header dump |
+---------------------+
</pre>
<dl>
<dt><tt class="literal"><span class="pre">signature</span></tt></dt>
<dd>Signature is '<tt class="literal"><span class="pre">p</span></tt>'.</dd>
</dl>
</div>
</div>
<div class="section" id="terminator">
<h3><a name="terminator">Terminator</a></h3>
<pre class="literal-block">
+-------------------+----------+----------+----------+
| infinint | 0x00 | bitfield | 0xFF |
| catalog position | padding | term cap | term cap |
+-------------------+----------+----------+----------+
3. 4. 2. 1.
(2)--------------------------> <------------------(1)
</pre>
<p>The terminator stores the position of the beginning of the catalog. It is the
last object written during archive creation. After processing the slice and
archive headers the reader must seek to the end and read the terminator, then
the catalog.</p>
<p>The terminator is read in two stages. First, the terminator cap is read
starting from EOF to the bitfield. The cap is used to determine the size and
calculate the offset of the infinint. The infinint stores the archive level
offset of the catalog.</p>
<ol class="arabic">
<li><p class="first">Terminator cap stage</p>
<blockquote>
<ol class="arabic">
<li><p class="first"><tt class="literal"><span class="pre">N</span></tt> <tt class="literal"><span class="pre">0xFF</span></tt> bytes. One byte indicates 8 blocks added to the offset
where the blocksize is 1 byte.</p>
</li>
<li><p class="first"><tt class="literal"><span class="pre">1</span></tt> bitfield indicating 0 to 7 additional blocks in excess
of 8 block groups:</p>
<pre class="literal-block">
+-------------------------------+
| 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
+---+---+---+---+---+---+---+---+
| 1 | - | - | - | - | - | - | - | 1 extra blocks
| 1 | 1 | - | - | - | - | - | - | 2 extra blocks
| 1 | 1 | 1 | - | - | - | - | - | 3 extra blocks ...
| | | | | | | | |
| 1 | 1 | 1 | 1 | 1 | 1 | 1 | - | 7 extra blocks
+-------------------------------+
</pre>
</li>
</ol>
</blockquote>
</li>
<li><p class="first">Catalog position indicator stage</p>
<blockquote>
<ol class="arabic" start="3">
<li><p class="first">Infinint containing the offset of the beginning of the catalog. Offset
zero is the start of the archive level, the position after the end of
the first slice header:</p>
<pre class="literal-block">
+---------+-------------------------------+-----------+------+
| archive | data | catalogue | term |
| header | | | |
+---------+-------------------------------+-----------+------+
^ ^
zero offset stored cat. offset
</pre>
</li>
<li><p class="first"><tt class="literal"><span class="pre">N</span></tt> <tt class="literal"><span class="pre">0x00</span></tt> bytes. Aligns end of infinint to 8 byte block boundary.</p>
</li>
</ol>
</blockquote>
</li>
</ol>
</div>
</div>
<div class="section" id="file-system-specific-features">
<h2><a name="file-system-specific-features">File System Specific Features</a></h2>
<p>blah blah blah</p>
<div class="section" id="posix-metadata">
<h3><a name="posix-metadata">Posix Metadata</a></h3>
<p>q flkjqs lkfjsdl kfj</p>
</div>
<div class="section" id="dos-metadata">
<h3><a name="dos-metadata">Dos Metadata</a></h3>
<p>qsldkjf lqsjflk jfmlqsk j</p>
</div>
<div class="section" id="extended-attributes">
<h3><a name="extended-attributes">Extended Attributes</a></h3>
<p>hkhkhkjhkhkj</p>
</div>
</div>
<div class="section" id="binary-format">
<h2><a name="binary-format">Binary Format</a></h2>
<p>When not otherwise specified, all binary information is stored in the ways
described below. Formats and encodings follow the appropriate standards
with the exception of the variable length integer format <tt class="literal"><span class="pre">infinint</span></tt> and
the weak encryption algorithm <tt class="literal"><span class="pre">scramble</span></tt>.</p>
<div class="section" id="endian-encoding">
<h3><a name="endian-encoding">Endian Encoding</a></h3>
<p>Multiple byte integer values are always written in network byte order (MSB).
All values are unsigned unless otherwise indicated.</p>
</div>
<div class="section" id="time-format">
<h3><a name="time-format">Time Format</a></h3>
<p>All time values use the definition of Posix time as per POSIX.1. Values are
encoded in variable length <tt class="literal"><span class="pre">infinint</span></tt> form.</p>
</div>
<div class="section" id="character-encoding">
<h3><a name="character-encoding">Character Encoding</a></h3>
<p>All characters are stored in UTF-8. Strings are null-terminated arrays.</p>
</div>
<div class="section" id="infinint-format">
<h3><a name="infinint-format">Infinint Format</a></h3>
<pre class="literal-block">
+....+....+....+----+....+....+....+....+
| 00 | 00 | 00 | BB | XX | XX | XX | XX |
+....+....+....+----+....+....+....+....+
1. 2. 3.
(1)-----------------| (2)---------------|
</pre>
<p>Many of the numbers stored in dar archives have specific limits. For example,
because on most Posix systems UID and GID are 16-bit words those values are
stored in that way in the archive. For other values, integers are stored in
<tt class="literal"><span class="pre">infinint</span></tt> form. The integer data is placed after a length cap.</p>
<ol class="arabic">
<li><p class="first">Length preamble</p>
<blockquote>
<ol class="arabic">
<li><p class="first"><tt class="literal"><span class="pre">N</span></tt> <tt class="literal"><span class="pre">0x00</span></tt> bytes. One byte for each group of 8 blocks in payload
where the blocksize is 4 bytes (one integer).</p>
</li>
<li><p class="first"><tt class="literal"><span class="pre">1</span></tt> bitfield indicating the number of integers beyond the 8 integer
group boundry. Indicates 0 to 7 additional integers. One bit must be set:</p>
<pre class="literal-block">
+-------------------------------+
| 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
+---+---+---+---+---+---+---+---+
| 1 | - | - | - | - | - | - | - | 1 extra integers
| - | 1 | - | - | - | - | - | - | 2 extra integers
| - | - | 1 | - | - | - | - | - | 3 extra integers ...
| | | | | | | | |
| - | - | - | - | - | - | - | 1 | 0 extra integers
+-------------------------------+
</pre>
</li>
</ol>
</blockquote>
</li>
<li><p class="first">Payload</p>
<blockquote>
<ol class="arabic simple" start="3">
<li><tt class="literal"><span class="pre">N</span></tt> bytes containing data. Stored in network byte order (MSB).</li>
</ol>
</blockquote>
</li>
</ol>
</div>
<div class="section" id="scrambling-encryption">
<h3><a name="scrambling-encryption">Scrambling Encryption</a></h3>
<p>Scrambling is a weak block cipher algorithm that uses a passphrase to form
a variable length key. The UTF-8 passphrase is translated directly into the
symmetric key and reiteratively matching each byte of the key with the bytes
of the plain text. The cipher text is formed by adding the matched byte of the
key to the plain text data (plain + pass % 256).</p>
<p>For example, take "example" as a passphrase and "source plain text data" as
the plain text. A 56-bit symmetric key is formed directly from the passphrase
and the plain text is divided into four blocks:</p>
<pre class="literal-block">
blocks: -- 1 --------------- | -- 2 --------------- | -- 3 --------------- | --
data: 73 6F 75 72 63 65 20 70 6C 61 69 6E 20 74 65 78 74 20 64 61 74 61
key: 65 78 61 6D 70 6C 65 65 78 61 6D 70 6C 65 65 78 61 6D 70 6C 65 65
result: D8 E7 D6 DF D3 D1 85 D5 E4 C2 D6 DE 8C D9 CA F0 D5 8D D4 CD D9 C6
</pre>
<p>Decryption is performed by subtracting the key in from the cipher text in the
same manner.</p>
<p>This scramble algorithm was implemented before support for strong encryption
was available. Scramble is not secure and its use is not recommended.</p>
</div>
<div class="section" id="standard-encryption-algorithms">
<h3><a name="standard-encryption-algorithms">Standard Encryption Algorithms</a></h3>
<p>qsdfqsdfqs</p>
</div>
<div class="section" id="standard-compression-algorithms">
<h3><a name="standard-compression-algorithms">Standard Compression Algorithms</a></h3>
<p>qsdfqsqdf</p>
</div>
</div>
<div class="section" id="format-limitations">
<h2><a name="format-limitations">Format Limitations</a></h2>
<p>dflkjqlfj qsfjls jk</p>
<ul class="simple">
<li>slice size validation -- if a file is missized, fails during reading, no quick way to check</li>
</ul>
</div>
</div>
</div>
</body>
</html>