Re: discussion of createrepo and repodata format future

On Fri, Aug 06, 2010 at 01:02:47PM -0400, seth vidal wrote:
> More or less -but again - using the name/location/label of the file to
> determine language - so you don't have a bunch of duplicate info.
> 
> if the file is named trans.de_DE.xml.gz then we don't really need to
> label summary and description with lang='de' do we?

Right.

> > Zsync works by searching local files for blocks with the same checksum
> > as the target file. As checksum calculation is not a cheap operation,
> > you can't simply do it for every byte offset in the local files. Thus
> > you also need a cheap checksum, and you only verify with the real
> > checksum if the cheap checksum matches.
> 
> Ah - now I recall - the need for the hashing and keeping around older
> revisions of the metadata is what made zsync less palatable.
> 
> Since each pkg is an individual 'chunk' of data that is needed to
> comprise the whole of the repodata, one thought was generating both a
> complete copy of the repodata and a discrete chunk of the metadata
> per-pkg.
> 
> So if I look up the pkglist and see that the changeset from the last
> time I got the metadata was the addition/update of 100 pkgs and the
> removal of 20 and downloading the metadata for those 100 pkgs is smaller
> than downloading the whole thing, then I could just do that and create
> the new metadata on my own.

Yes, that would be possible. It depends on how good each per-pkg chunk
compresses, I guess.
The advantage of the all chunks in one file with delta downloads is that
it doesn't clutter up the repo directory so much.

> > This scheme probably only works with xml (where new packages just
> > get added to the end of the file, at least for our updates) and
> > with a --rsyncable compression method.
> 
> you add the content to the end of the existing metadata? Does that mean
> your metadata grows w/o bound over the duration of a release?

If rpms get deleted they also vanish from the metadata. But we tend
to keep the old rpms for reference, so usually only the number of packages
at the start of the xml data changes and new packages get appended.

Cheers,
  Michael.

-- 
Michael Schroeder                                   [email protected]
SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg
main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}