Re: metadata layout problems and some history

On Mon, 9 Aug 2010, seth vidal wrote:

> On Mon, 2010-08-09 at 00:00 +0200, Anders F Björklund wrote:
>> seth vidal wrote:
>> 
>> > There have been some small changes over the years but I believe none
>> > that have broken the parsers of any of tools.
>> 
>> The biggest changes for smart was the change to the sha256 checksums
>> and unique-md-filenames which meant old cache files weren't deleted.
>> 
>> > Then what happened was all of the folks using/maintaining the tools
>> > found issues they had to work around and so we did:
>> 
>> > - Smart runs an indexer over the xml files to generate the byte-offset
>> > locations of pkgs in the xml file so it can seek to those locations
>> > quickly.
>> 
>> Except that quick-er doesn't mean that it still wasn't dog slow...
>>
>>     "Well, at least having an index cuts list time from
>>      30 minutes to 30 seconds. On the other hand, that is
>>      still ridiculously slow compared to the usual 3...
>>      So I don't think the cache size/startup time matters ?"
>> 
>> So neither metadata nor sqlite has been merged into the trunk.
>> It's all being stored in the cache, instead of using XML/SQL.
>> 
>
> So help me understand what would make your situation better? What would
> make the rpm-md less problematic for smart?

AFAICT smart, zypp an apt all would just want a raw dump of the relevant 
data in a format that's lightweight to read in once to generate their own 
native data, and at least in case of smart and apt, can be indexed (eg 
jump to file offset X to access the details that aren't included in the 
package cache). Yes, dejavu on yum-metadata-parser, which was the yum 
equivalent of "internal cache creation", only the sqlite format it creates 
got added (as an extension) to repomd.

XML works fine for the raw dump part, except that it's hideously bloated 
format for what it's used for and expensive to parse (dejavu 
yum-metadata-parser again), and isn't good for indexing or searching.
Heck, a dumb tagged plain-text file would be much better suited for apt 
(and I assume smart). I've vague memories of Suse actually having such a 
format at some point before switching to repodata.

 	- Panu -