Re: Rpm-metadata Digest, Vol 75, Issue 2

Matthew Dawkins <[email protected]> Tue, 3 Aug 2010 10:30:00 -0600
Newsgroups gmane.linux.rpm.metadata
Message-ID <[email protected]> 
On Tue, Aug 3, 2010 at 10:02 AM, <[email protected]>wrote:

> Send Rpm-metadata mailing list submissions to
>        [email protected]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>        http://lists.baseurl.org/mailman/listinfo/rpm-metadata
> or, via email, send a message with subject or body 'help' to
>        [email protected]
>
> You can reach the person managing the list at
>        [email protected]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Rpm-metadata digest..."
>
>
> Today's Topics:
>
>   1. Re: adding different compression types to createrepo
>      (Anders F Bj?rklund)
>   2. Re: adding different compression types to createrepo
>      (Duncan Mac-Vicar P.)
>   3. Re: adding different compression types to createrepo (seth vidal)
>   4. Re: adding different compression types to createrepo (seth vidal)
>   5. Re: Rpm-metadata Digest, Vol 75, Issue 1 (Matthew Dawkins)
>   6. Re: adding different compression types to createrepo
>      (James Antill)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 3 Aug 2010 16:55:55 +0200
> From: Anders F Bj?rklund <[email protected]>
> To: "Duncan Mac-Vicar P." <[email protected]>
> Cc: [email protected]
> Subject: Re: [Rpm-metadata] adding different compression types to
>        createrepo
> Message-ID: <[email protected]>
> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
>
> Duncan Mac-Vicar P. wrote:
>
> >> I think the extended opensuse patch should be acceptable, where it
> >> uses "primary" for .xml or .xml.gz and the new "primary_lzma" key
> >> for .xml.lzma or .xml.xz - plus the same added for "filelists_lzma"
> >> and "other_lzma", it causes less breakage than changing "primary"...
> >>
> > /usr/bin/repo2solv.sh which is the tool that parses the repos in
> > openSUSE, already supports lzma (look at the source), we just don't
> > have any tool generating lzma.
>
> And Smart also supports lzma/xz, since it just uses the file suffix.
> I was under the impression that you had patched createrepo for lzma ?
> And it should be simple to support both of .lzma and .xz compressions
> by using PylibLZMA in yum, or liblzma directly for yum-metadata-parser.
>
> > I disagree the type should be changed. It would add more confusing
> > undocumented stuff to the format.
>
> The concern was that if the type wasn't changed, then older versions
> of yum would try to parse the lzma/xz as uncompressed and just get
> checksum errors. So it was more for "compatibility" than anything ?
> But the "primary_lzma" wasn't my idea, since it came from openSUSE.
>
> --anders
>
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 03 Aug 2010 16:45:31 +0200
> From: "Duncan Mac-Vicar P." <[email protected]>
> To: [email protected]
> Subject: Re: [Rpm-metadata] adding different compression types to
>        createrepo
> Message-ID: <[email protected]>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
>  On 08/03/2010 10:46 AM, Anders F Bj?rklund wrote:
> > I think the extended opensuse patch should be acceptable, where it
> > uses "primary" for .xml or .xml.gz and the new "primary_lzma" key
> > for .xml.lzma or .xml.xz - plus the same added for "filelists_lzma"
> > and "other_lzma", it causes less breakage than changing "primary"...
> >
> > --anders
> /usr/bin/repo2solv.sh which is the tool that parses the repos in
> openSUSE, already supports lzma (look at the source), we just don't have
> any tool generating lzma.
>
> I disagree the type should be changed. It would add more confusing
> undocumented stuff to the format.
>
> Duncan
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Tue, 03 Aug 2010 11:07:05 -0400
> From: seth vidal <[email protected]>
> To: [email protected]
> Subject: Re: [Rpm-metadata] adding different compression types to
>        createrepo
> Message-ID: <1280848025.2506.121.camel@oliver>
> Content-Type: text/plain; charset="UTF-8"
>
> On Mon, 2010-08-02 at 17:39 -0600, Matthew Dawkins wrote:
> > Hello, I'm new to the list, but not new to using createrepo and yum
> > for repo creation.
> >
> > I'm looking to find if there are not patches out there to add an
> > option to calling for lzma or xz compression of the metafiles instead
> > of gzip. I have seen  what opensuse uses to compress the primary.xml
> > in conjunction with the gzip compression and I have extended on it to
> > compress the other and filelists files too, but they don't really add
> > clean functionality.
>
>
> there aren't any upstream, yet. I just got a python liblzma pkg into
> fedora to support that compression natively in python. So that's half of
> it. The biggest trick is having to name the datatype in repomd.xml to
> reflect the compression format since older yums (and apts and smarts,
> etc) won't necessarily be able to handle lzma compressed files.
>
> -sv
>
>
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 03 Aug 2010 11:14:28 -0400
> From: seth vidal <[email protected]>
> To: [email protected]
> Subject: Re: [Rpm-metadata] adding different compression types to
>        createrepo
> Message-ID: <1280848468.2506.129.camel@oliver>
> Content-Type: text/plain; charset="UTF-8"
>
> On Tue, 2010-08-03 at 10:46 +0200, Anders F Bj?rklund wrote:
> > Matthew Dawkins wrote:
> > > I'm looking to find if there are not patches out there to add an
> > > option to calling for lzma or xz compression of the metafiles
> > > instead of gzip. I have seen  what opensuse uses to compress the
> > > primary.xml in conjunction with the gzip compression and I have
> > > extended on it to compress the other and filelists files too, but
> > > they don't really add clean functionality.
> >
> > That would be https://features.opensuse.org/309167 I think,
> > but see also http://duncan.mac-vicar.com/blog/archives/537
> > /and http://lists.baseurl.org/pipermail/yum/2009-April/022560.html)
> > and some other discussion about using YAML and/or LZMA for repodata.
>
> YAML? I don't think anyone has discussed YAML and was taken seriously.
>
> More likely than not most of the metadata will be transitioned to
> sqlite-db ONLY and no xml at all. Yum already supports repos with only
> sqlite dbs and no xml (other than repomd.xml).
>
>
> > I think the extended opensuse patch should be acceptable, where it
> > uses "primary" for .xml or .xml.gz and the new "primary_lzma" key
> > for .xml.lzma or .xml.xz - plus the same added for "filelists_lzma"
> > and "other_lzma", it causes less breakage than changing "primary"...
>
> I suspect it would make sense to provide:
>
> [primary|filelists|other][_db]_xz in the repomd.xml, in addition to the
> gzip/bzip2 compressed alternatives and then gradually phase out the
> older ones.
>
>
> My biggest reticence in adding the xz support now is that if we move to
> changing the repodata format to the layout that's been discussed on here
> and on yum-devel list that we'll be able to move to xz support(and some
> future proofing for compression formatting) w/o having to kludge a bunch
> of xz compression crap into createrepo.
>
> What do y'all think?
>
> -sv
>
>
>
>
> ------------------------------
>
> Message: 5
> Date: Tue, 3 Aug 2010 09:41:23 -0600
> From: Matthew Dawkins <[email protected]>
> To: [email protected]
> Subject: Re: [Rpm-metadata] Rpm-metadata Digest, Vol 75, Issue 1
> Message-ID:
>        <[email protected]>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On Tue, Aug 3, 2010 at 6:00 AM, <[email protected]
> >wrote:
>
> > Send Rpm-metadata mailing list submissions to
> >        [email protected]
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >        http://lists.baseurl.org/mailman/listinfo/rpm-metadata
> > or, via email, send a message with subject or body 'help' to
> >        [email protected]
> >
> > You can reach the person managing the list at
> >        [email protected]
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Rpm-metadata digest..."
> >
> >
> > Today's Topics:
> >
> >   1. adding different compression types to createrepo (Matthew Dawkins)
> >   2. Re: adding different compression types to createrepo
> >      (Anders F Bj?rklund)
> >
>
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Mon, 2 Aug 2010 17:39:45 -0600
> > From: Matthew Dawkins <[email protected]>
> > To: [email protected]
> > Subject: [Rpm-metadata] adding different compression types to
> >        createrepo
> > Message-ID:
> >        <[email protected]<AANLkTikD%[email protected]>
> <AANLkTikD%[email protected]<AANLkTikD%[email protected]>
> >
> > >
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > Hello, I'm new to the list, but not new to using createrepo and yum for
> > repo
> > creation.
> >
> > I'm looking to find if there are not patches out there to add an option
> to
> > calling for lzma or xz compression of the metafiles instead of gzip. I
> have
> > seen  what opensuse uses to compress the primary.xml in conjunction with
> > the
> > gzip compression and I have extended on it to compress the other and
> > filelists files too, but they don't really add clean functionality.
> >
> > Regards,
> > Matthew Dawkins
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL: <
> >
> http://lists.baseurl.org/pipermail/rpm-metadata/attachments/20100802/dcf602ee/attachment-0001.html
> > >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Tue, 3 Aug 2010 10:46:07 +0200
> > From: Anders F Bj?rklund <[email protected]>
> > To: [email protected]
> > Subject: Re: [Rpm-metadata] adding different compression types to
> >        createrepo
> > Message-ID: <[email protected]>
> > Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
> >
> > Matthew Dawkins wrote:
> > > I'm looking to find if there are not patches out there to add an
> > > option to calling for lzma or xz compression of the metafiles
> > > instead of gzip. I have seen  what opensuse uses to compress the
> > > primary.xml in conjunction with the gzip compression and I have
> > > extended on it to compress the other and filelists files too, but
> > > they don't really add clean functionality.
> >
> > That would be https://features.opensuse.org/309167 I think,
> > but see also http://duncan.mac-vicar.com/blog/archives/537
> > /and http://lists.baseurl.org/pipermail/yum/2009-April/022560.html)
> > and some other discussion about using YAML and/or LZMA for repodata.
> >
> > I think the extended opensuse patch should be acceptable, where it
> > uses "primary" for .xml or .xml.gz and the new "primary_lzma" key
> > for .xml.lzma or .xml.xz - plus the same added for "filelists_lzma"
> > and "other_lzma", it causes less breakage than changing "primary"...
> >
> > --anders
> >
> >
> >
> > ------------------------------
> >
> > _______________________________________________
> > Rpm-metadata mailing list
> > [email protected]
> > http://lists.baseurl.org/mailman/listinfo/rpm-metadata
> >
> >
> > End of Rpm-metadata Digest, Vol 75, Issue 1
> > *******************************************
> >
>
>
> Well, I'll add the URL to my rediffed patch. It would be nice to drop the
> hybrid from making both compression types files and just use one or the
> other.
>
>
> http://dev.unity-linux.org/projects/unitylinux/repository/entry/packages/createrepo/P/createrepo-0.9.8-add-lzma-option-to-generate-primary.xml.lzma.patch
>
> I tried to do it with little luck. I believe the creation of the checksum
> for other than gzip files present is needed and then not appending the the
> primary,other,filelists_lzma, but rather just that info instead inside
> repomd.xml.
>
> Matt
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.baseurl.org/pipermail/rpm-metadata/attachments/20100803/af6cdff4/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 6
> Date: Tue, 03 Aug 2010 12:02:45 -0400
> From: James Antill <[email protected]>
> To: [email protected]
> Subject: Re: [Rpm-metadata] adding different compression types to
>        createrepo
> Message-ID: <[email protected]>
> Content-Type: text/plain; charset="UTF-8"
>
> On Tue, 2010-08-03 at 11:14 -0400, seth vidal wrote:
>
> > > I think the extended opensuse patch should be acceptable, where it
> > > uses "primary" for .xml or .xml.gz and the new "primary_lzma" key
> > > for .xml.lzma or .xml.xz - plus the same added for "filelists_lzma"
> > > and "other_lzma", it causes less breakage than changing "primary"...
> >
> > I suspect it would make sense to provide:
> >
> > [primary|filelists|other][_db]_xz in the repomd.xml, in addition to the
> > gzip/bzip2 compressed alternatives and then gradually phase out the
> > older ones.
>
>  *shrug*, I'm not sure why we'd want to add support for lzma on the
> obsolete .xml files. Smart and apt both support .sqlite now, and if
> zypper doesn't they can always use modifyrepo to add them for SuSE.
>  Also calling them "primary_db.xz" would fit the convention with
> "groups.gz".
>
> > My biggest reticence in adding the xz support now is that if we move to
> > changing the repodata format to the layout that's been discussed on here
> > and on yum-devel list that we'll be able to move to xz support(and some
> > future proofing for compression formatting) w/o having to kludge a bunch
> > of xz compression crap into createrepo.
>
>  Well it's going to be much easier to just change the compression, but
> it's not the end of the world if you want to put it off a year or so
> (IMO).
>
>
>
> ------------------------------
>
> _______________________________________________
> Rpm-metadata mailing list
> [email protected]
> http://lists.baseurl.org/mailman/listinfo/rpm-metadata
>
>
> End of Rpm-metadata Digest, Vol 75, Issue 2
> *******************************************
>


No time like the present if you ask me, and why force adoption? Shouldn't
this be all optional? If you want to use lzma give the option of --lzma,
same for xz or bz2 and gz. Personally I'd like to see xz as that's what we
use for all our compression. I just said lzma b/c I saw it kinda being
worked on.

The same with forcing the adoption of the sqlitedb, why? Just because
fedora/yum use it??? That's really weak logic. Again, I would say it's all
about options. Let the repo provider decide.

Regards,
Matt

_______________________________________________
Rpm-metadata mailing list
[email protected]
http://lists.baseurl.org/mailman/listinfo/rpm-metadata