metadata layout problems and some history

Hi folks,
 The discussion of future changes to the repodata may be a bit sideways,
so let's talk about it from another direction - that of the history of
the rpm-md format:

In 2003 there were many different formats for repositories of pkgs. Yum
had one, Apt had two (apt-deb and apt-rpm+bloat for the filelists),
up2date had rhn's server doing depsolving/listing of pkgs, ximian had
one for red carpet, there were a bunch. The goal of the rpm-metadata
discussions were to have a single format everyone could use so that all
tools could access the other repos. I headed this up and tried to corral
people to talk about it and had some amount of success. Out of it came
roughly what you know today as the format for the xml files in a repo.

There have been some small changes over the years but I believe none
that have broken the parsers of any of tools.

Then what happened was all of the folks using/maintaining the tools
found issues they had to work around and so we did:

- Yum eventually started generating sqlite dbs of the same data that was
in the xml files to speed up access to the data. 

- Zypper uses .solv files to do much the same thing for the satsolver
code. 

- Smart runs an indexer over the xml files to generate the byte-offset
locations of pkgs in the xml file so it can seek to those locations
quickly.


We all worked around the problems but kept the basic compat with the
repo format. So, we achieved the original goal, right? Seems like we
did.



So, with that in mind - I was hoping we could take this opportunity
(since I saw a bunch of people just subscribed to the list) to discuss
the problems we've had to work around and if we can make a new format
that can:

1. cover new changes to rpm packages

2. make our workarounds less necessary/less painful

3. work with new realities of interoperating with MANY MANY repos

4. handle things that were never included in the original specification
we've later found out would be useful.


Think of this as compiling a list of issues to be solved with a Version2
of things.


Ideas?

Thanks,
-sv