Re: The illusion of simplicity and low cost in data design and computing

Put information about a file inside or outside the file?

Here’s what Eric Raymond [1] says:

“Unix partisans prefer approaches that make file data self-describing so that metadata is stored within the file. The problem with the Unix approach is that every program that writes the file has to know about it. Thus, for example, if we want the file to carry type information inside it, every tool that touches it has to take care to either preserve the type field unaltered or interpret and then rewrite it. While this would be theoretically possible to arrange, in practice it would be far too fragile.”

“On the other hand, supporting file attributes raises awkward questions about which file operations preserve them. It’s clear that a copy of a named file to another name should copy the source file’s attributes as well as its data—but suppose we “cat” the file, redirecting its output to a new one? The question becomes: Which operations preserve the file attributes?”

[Roger] The creators of XML chose the former approach–store metadata within the file. What metadata does XML store? (1) The version of XML, (2) the character encoding. Eric Raymond says that approach has problems: every program that operates on an XML file has to understand how XML stores this metadata.

[1] “The Art of UNIX Programming” by Eric Raymond, page 467.