Request to create a Dump Tag registry

AuriStor has been investigating extending the Dump Stream to support the 
functionality proposed in April 2011 as RPC Refresh (Gerrit 4573) as 
well as additional data and metadata types implemented by AuriStorFS.  
To avoid unnecessary collisions and to maximize interoperability I 
propose the creation of a Dump Tag Registry.

The Dump Format is poorly documented and many tags and sub-tags have 
been squatted upon over the years without creation of a registry.  The 
original Dump Format did not provide any mechanism for backward 
compatibility.  The structure of the data following a Tag or Sub-Tag 
could not be determined independent of any particular implementation.   
For example, when OpenAFS introduced the D_VNODE 'h' Sub-Tag to 
represent the lengths of vnodes larger than 2GB any dump stream that 
included the D_VNODE 'h' sub-tag could not be parsed by an earlier 
version of IBM AFS or OpenAFS.  It can be argued that in this particular 
case the ability to understand 64-bit lengths should be considered 
critical and therefore parsing the dump stream should fail.   The same 
case cannot be made for the D_VOLUMEHEADER 'V' (volume update counter) 
sub-tag that was introduced in October 2005.  An inability to parse and 
restore this counter should not prevent a backup dump stream from being 
used to recover the contents of a Volume.

In March 2005 Peter Somogyi proposed a solution submitted as OpenAFS RT 
17947 which requested that OpenAFS implement a mechanism for storing 
Vnode Data in compressed form.  The data compression implementation was 
never adopted by OpenAFS but in September 2009 OpenAFS adopted a 
framework which would permit tags and sub-tags to be introduced in the 
future without breaking compatibility with earlier versions of OpenAFS 
that implemented the framework.  OpenAFS 1.4.12 was the first production 
release to implement the framework.

The text of the proposed framework was forwarded to afs3-standardization 
in October 2007.

https://lists.openafs.org/pipermail/afs3-standardization/2007-October/001053.html

OpenAFS adopted the TLV standard for future tags by merging

https://gerrit.openafs.org/#/c/191/4

The framework extends the pre-existing Dump Format as follows:

 1. Valid Dump or "D" Header Tag values range from 0x1 to 0x15
     1. The four tags in use before the adoption of the framework are
        declared "Legacy"
         1. 0x01    D_DUMPHEADER
         2. 0x02    D_VOLUMEHEADER
         3. 0x03    D_VNODE
         4. 0x04    D_DUMPEND
     2. The remaining tags are designated Tag-Length-Value (TLV) tags
     3. Each D Header tag has its own private namespace of sub-tags
 2. Valid Sub-Tags values range from 0x16 to 0x7d
     1. All D_VOLUMEHEADER and D_VNODE Sub-tags in use before the
        adoption of the framework are declared "Legacy"
     2. Non-legacy sub-tags between 0x16 and 0x60 are designated TLV
        sub-tags.
     3. Non-legacy sub-tags between 0x61 and 0x7a are designated
        Standard sub-tags where the sub-tag is followed by a fixed
        32-bit integer value in network-byte-order.
     4. Non-legacy sub-tags between 0x7b and 0x7d are designated
        Dataless sub-tags which are not followed by any data.
 3. Tag 0x0 is reserved and currently invalid
 4. Tag0x7e is dataless and has the special meaning that the Next Tag or
    Sub-tag is CRITICAL.  Failure to recognize the meaning of the Tag or
    Sub-tag must result in a fatal failure.
 5. Tag 0x7f is reserved.

TLV tags are followed by a variable length "Length" field which is then 
followed by the specified number of octets.  The variable length 
"Length" is parsed as follows:

 1. If the first octet is less than or equal to 0x7f, then the Value
    that follows is that number of octets.
 2. If the first octet is 0x80, then the Value that follows is of
    indefinite length.  The value data that follows must be
    self-describing with regards to length.  In other words, a parser
    that understands the data format must be able to identify the end of
    the value from the contents of the data.  If an indefinite length
    tag or sub-tag is unrecognized, it is a fatal error regardless of
    whether or not the CRITICAL tag preceded it.
 3. If the first octet 'L' is between 0x81 and 0x88, the length is value
    of the next (L & 0x0f) octets combined in most significant byte
    (MSB) order.  0x81 indicates one byte, 0x82 two bytes, 0x88 eight bytes.
 4. If the first octet is greater than 0x88, it is an invalid length
    which is a fatal error.

The OpenAFS implementation has the following deficiencies:

 1. It does not handle parsing of unrecognized D Tags.
 2. The MIN_TLV_TAG sub-tag must be 0x16 not 0x5.
 3. Tag values up to and including D_MAX (20) must not be used as sub-tags.
 4. The Legacy tag table is incomplete.


Initial Registry Contents:

I propose that all of the following tags and sub-tags be added to the 
registry.   Sub-tags designated as deprecated are known to have been 
stored in backups and therefore might be present in a restored dump 
stream at some point in the future.   Deprecated sub-tags should be 
avoided but might be reallocated if all other sub-tags are in use.

I propose that new tags be allocated on a first come, first served basis 
with a documentation requirement. Documentation must be provided that is 
sufficient to permit a dump stream to be restored if non-critical 
unrecognized tags are included in the stream.  A stream containing 
unrecognized critical tags will fail to restore but sufficient 
documentation must be present for a site to evaluate the risk associated 
with the criticality.

When designing new tags for registration developers should consider use 
by the entire ecosystem.  For example, in addressing the Year 2038 
problem new tags to transmit 64-bit 100ns timestamps for vnodes and 
volumes should be reusable by many implementations.  The same should be 
true for 64-bit data versions, 96-bit vnode ids, 64-bit volume ids, 
64-bit user ids, longer volume names, 64-bit volume quotas, and 64-bit 
volume space usage.  The Dump Header will require new tags for 64-bit 
100ns timestamps, 64-bit volume ids, and longer volume names,

D Header Tags:

     0x01    D_DUMPHEADER       IBM	Legacy
     0x02    D_VOLUMEHEADER     IBM	Legacy
     0x03    D_VNODE            IBM	Legacy
     0x04    D_DUMPEND          IBM	Legacy

D_DUMPHEADER Sub-Tags

     21      0x15    64-bit Volume ID 	AuriStor	TLV
     22      0x16    100ns Dump Times	AuriStor	TLV
     'n'     0x6e    Volume Name		IBM 		Legacy: C-String
     't'     0x74    32-bit Dump Times	IBM		Legacy
     'v'     0x76    32-bit Volume ID 	IBM		Legacy

D_VOLUMEHEADER Sub-Tags

     21      0x15    64-bit Volume IDs 	      AuriStor	TLV
     22      0x16    Volume Maximum ACL	      AuriStor	TLV
     23      0x17    Volume Sec Levels	      AuriStor	TLV
     24      0x18    64-bit Max Quota 	      AuriStor	TLV
     25      0x19    64-bit Disk Usage	      AuriStor	TLV
     26      0x1a    100ns Timestamps	      AuriStor	TLV
     27      0x1b    64-bit Features	      AuriStor	TLV
     28      0x1c    64-bit Owner	      AuriStor	TLV
     29      0x1d    64-bit Min Quota	      AuriStor	TLV
     30      0x1e    64-bit File Count	      AuriStor  TLV
     'A'     0x41    32-bit Last Access Date   OSD	Legacy
     'B'     0x42    32-bit Backup Date	      IBM	Legacy
     'C'     0x43    32-bit Creation Date      IBM	Legacy
     'D'     0x44    32-bit Day Use Date	      IBM	Legacy
     'E'     0x45    32-bit Expiration Date    IBM	Legacy
     'F'     0x46    OSD Policy		      OSD	Legacy: 32-bit [deprecated - replaced by 'P']
     'M'     0x4d    MOTD		      IBM        Legacy: C-String
     'O'     0x4f    Offline Message	      IBM	Legacy: C-String
     'P'     0x50    OSD Policy		      OSD	Legacy: 32-bit [deprecated - replaced by 'y']
     'U'     0x55    32-bit Update Date        IBM	Legacy
     'V'     0x56    32-bit Update Counter     OpenAFS	Legacy
     'W'     0x57    Week Use Statistics       IBM	Legacy: structure
     'Z'     0x5a    Day Use Statistics	      IBM	Legacy
     'a'     0x61    Account Number	      IBM	Standard [deprecated]
     'b'     0x62    Blessed Flag	      IBM	Legacy: 8-bit
     'c'     0x63    32-bit Clone Id	      IBM	Standard
     'd'     0x64    32-bit Disk Usage	      IBM	Standard
     'f'     0x66    32-bit File Count	      IBM	Standard
     'i'     0x69    32-bit Volume Id	      IBM	Standard
     'm'     0x6d    32-bit Volume Min Quota   IBM	Standard
     'n'     0x6e    Volume Name		      IBM	Legacy: C-String
     'o'     0x6f    32-bit Volume Owner       IBM	Standard
     'p'     0x70    32-bit Parent Volume Id   IBM	Standard
     'q'     0x71    32-bit Volume Max Quota   IBM	Standard
     'r'     0x72    OSD Max number of files   OSD	Standard
     's'     0x73    In-Service Flag	      IBM	Legacy: 8-bit
     't'     0x74    Volume Type		      IBM	Legacy: 8-bit
     'u'     0x75    32-bit Next Uniquifier    IBM	Standard
     'v'     0x76    32-bit Stamp Version      IBM	Standard [deprecated]
     'y'     0x79    OSD Policy		      OSD	Standard

D_VNODE Sub-Tags

     21      0x15    File ACL		      AuriStor	TLV [deprecated]
     22      0x16    100ns Timestamps          AuriStor	TLV
     23      0x17    64-bit A, O, G IDs 	      AuriStor  TLV
     24      0x18    96-bit Vnode# Parent#     AuriStor	TLV
     25      0x19    64-bit Data Version	      AuriStor	TLV
     26      0x1a    64-bit Ext ACL	      AuriStor	TLV
     'A'     0x41    32-bit ACL		      IBM	Legacy: C-String
     'L'     0x4c    OSD vnode length (hi,lo)  OSD	TLV
     'O'     0x4f    OSD metadata string	      OSD	TLV
     'P'     0x50    OSD dir policy index      OSD	Legacy: 32-bit [deprecated - replaced by 'd']
     'a'     0x61    32-bit Author ID	      IBM	Standard
     'b'     0x62    Unix Mode Bits	      IBM	Legacy: 16-bit
     'd'     0x64    OSD dir policy index      OSD	Standard
     'f'     0x66    File Stream - Small	      IBM	Legacy: structured
     'g'     0x67    32-bit Group ID	      IBM	Standard
     'h'     0x68    File Stream - Large	      OpenAFS	Legacy: structured
     'l'     0x6c    Link Count 		      IBM	Legacy: 16-bit
     'm'     0x6d    32-bit UNIX Modify Time   IBM	Standard
     'o'     0x6f    32-bit Owner ID	      IBM	Standard
     'p'     0x70    32-bit Parent Vnode #     IBM	Standard
     's'     0x73    32-bit Server Modify Time IBM	Standard
     't'     0x74    Vnode Type		      IBM	Legacy: 8-bit
     'u'     0x75    32-bit Last Access Time   OSD	Standard [deprecated]
     'v'     0x76    32-bit Data Version	      IBM	Standard
     'x'     0x78    OSD File Online Flag      OSD	Standard
     'y'     0x79    OSD vnode length	      OSD       Legacy: structure [deprecated - replaced by 'L']
     'z'     0x7a    OSD metadata string	      OSD	Legacy: C-String  [deprecated - replaced by 'O']

Guidance for Use of New Tags

 1. If a new sub-tag is allocated to support a larger value, for example
    D_VOLUMEHEADER 0x15 "64-bit IDs" which can replace 0x69 "32-bit
    Volume Id", 0x70 "32-bit Parent Id", and 0x63 "Clone Id", then the
    new tag should only be generated if the data values cannot be
    represented by the older tag.  If the new tag is used, it should be
    preceded by the CRITICAL tag.
 2. If a new sub-tag is allocated to augment existing data, then both
    sub-tags should be generated.  For example, the D_VOLUMEHEADER 0x16
    "100ns timestamps" provides 100ns granularity for accessDate,
    updateDate, creationDate, backupDate and expirationDate.  This
    sub-tag should be generated in addition to the 32-bit second
    granularity equivalents.  The new sub-tag should not be preceded by
    the CRITICAL tag.  This permits the dump stream to be restored both
    to systems that understand 100ns timestamps and those that do not.

Please note that I did not include the RT 17947 Compression D Tags: 
D_COMPRESS_TAG (0x5) and D_FLAGS_TAG (0x6) in the list of registered 
tags.  There are two reasons for this.  First, to the best of my 
knowledge the Compression Proposal was never put into production.  
Second, I believe the allocation of D_COMPRESS_TAG and D_FLAGS_TAG out 
of the Dump Header namespace was a mistake. If anyone revisits the 
Compression proposal I believe that the details of the compression 
should be sent either via D_DUMPHEADER sub-tags or D_VOLUMEHEADER 
sub-tags; and a new D_VNODE sub-tag should be used to store the 
Compressed File Stream.  The choice of D_DUMPHEADER vs D_VOLUMEHEADER 
depends on whether or not all of the components of a "merged dump" 
stream must share the same compression options.  If yes, then 
D_DUMPHEADER is the preferred location.  If not, then D_VOLUMEHEADER 
should be used.

For those that are not familiar with a "merged dump", incremental dump 
streams can be merged into a full dump stream by:

 1. merging the D_DUMPHEADER "from".."to" time range into the
    D_DUMPHEADER at the start of the stream
 2. remove the D_DUMPEND header from the end of the first stream
 3. remove the D_DUMPHEADER from the stream to be appended
 4. concatenate the two streams producing a stream with multiple
    D_VOLUMEHEADER blocks each of which are followed by one or more
    D_VNODE blocks.

A merged stream can be used to restore a volume in a single RPC.

If anyone is familiar with other Tags and Sub-tags that are in use, 
please let me know.

I will provide additional details of the AuriStor sub-tags to the 
Registrar once a Registry has been published.

Sincerely,

Jeffrey Altman