Create a catalog from the rpm database

This is a multipart message in MIME format.

------=_NextPart_000_00A3_01CF1B40.D63554F0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

Hello,

I use dar extensively for backups, usually separating system (the linux
itself) and content backups. I store system backups also in my archive in
order to have some history of the changed configurations and such.

The problem is that this is becoming a big organizational nightmare. A
simple update might change a lot of files, bloating backup size and if you
multiply that with a few dozen servers, checking each and every backup is
quite a task. However, most of the stuff is rarely needed as files not
specifically changed are better tracked by rpm itself (having a list of
installed packages + changes would allow you to rebuild the same system). I
have thought of creating a script that would list rpm changed and untracked
files and include only those in the system backup, but there are some
exceptions. Some files might be changed on the disk, but not listed in rpm
verify for instance the rpm database itself:

%verify(not md5 size mtime) %ghost %config(missingok,noreplace)
/var/lib/rpm/*

The files in /var/lib/rpm will not be shown as modified by rpm -V, but will
be included in rpm -ql. Any script working only with the list of changed and
untracked files, would fail to include these in the backup.

At first I thought of somehow interfacing dar directly with the rpm database
but then I decided for a simpler approach. The attached script tries to
create an empty root tree (sparse files) from the rpm database that can be
used to create a dar catalogue (-A +, a snapshot backup) which in turn can
be used as a reference for a system backup. Such a backup would be pretty
lightweight and allow the admin to inspect the changes on the system
compared to the rpm database. I have found similar ideas while searching for
an existing solution (http://tomayko.com/writings/MinimalSystemBackups) but
have not found an already working one :-) The script tries to be careful to
create only the files which are not changed (in comparison to the rpm db)
also taking into account the changes that can occur due to prelinking. Of
course a full dar backup would be more suitable for restoring, since
rebuilding would certainly take more time (plus admin would have to be
careful to install the same version of rpm packages so that the stored
config files would not overwrite a possibly newer config format in a newer
rpm package). But such a diff backup would be easier to inspect and thus
verify the validity of a full backup made at the same time.

A more advanced backup script could use this approach to create a chain of
incremental backups, each storing only the subset of changes compared to the
rpm database and the previous diff backup (which could be made against a
previous state of the rpm database) by merging a new rpm snapshot backup
with the previous diff backup (one note, I have just noticed in the 2.4.10
man page that merging of isolated catalogues is not supported - I have tried
searching for why this has changed, but I only found an explanation that "If
you merge two extracted catalogues you will get an extracted catalogue").

Does this make any sense or have I just wasted a perfectly good Sunday
afternoon?

Regards,
gulikoza

------=_NextPart_000_00A3_01CF1B40.D63554F0
Content-Type: application/octet-stream;
	name="rpm-dbtree.py"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
	filename="rpm-dbtree.py"

#!/usr/bin/python=0A=
#=0A=
# rpm-dbtree.py    Create a (sparse empty) tree from rpm database=0A=
#=0A=
#   License: GPL=0A=
=0A=
import os, sys, subprocess=0A=
import rpm=0A=
=0A=
import stat=0A=
from yum import misc=0A=
=0A=
from pwd import getpwnam=0A=
from grp import getgrnam=0A=
=0A=
RPM_CHECKSUM_TYPES =3D { 1:'md5', 2:'sha1', 8:'sha256', 9:'sha384', 10:'sha=
512',=0A=
                       11:'sha224' } # from RFC 4880=0A=
=0A=
ts =3D rpm.ts()=0A=
mi =3D ts.dbMatch()=0A=
=0A=
# Directory where to extract the tree=0A=
root =3D '/dev/shm/rpm-tree'=0A=
print "Creating RPM root tree in", root=0A=
=0A=
if not os.path.exists(root):=0A=
    os.makedirs(root)=0A=
=0A=
def set_perm(f, u, g, perm):=0A=
    uid =3D getpwnam(u).pw_uid=0A=
    gid =3D getgrnam(g).gr_gid=0A=
=0A=
    os.chown(f, uid, gid)=0A=
    os.chmod(f, perm)=0A=
=0A=
for hdr in mi:=0A=
=0A=
    #print "%s" % (hdr['name'])=0A=
=0A=
    # Check for files in package=0A=
    for fi in hdr.fiFromHeader():=0A=
        # Filename (0), size (1), mode (2), mtime (3), flags (4), rdev? (5)=
, file inode (6), FNlink (7), Fstate (8), vflags (9), user (10), group (11)=
, md5sum (12)=0A=
=0A=
        mode =3D fi[2]=0A=
        perm =3D mode & 07777;=0A=
=0A=
        if stat.S_ISDIR(mode):=0A=
            # It's a directory, recurse into it=0A=
            #print 'Directory:'=0A=
=0A=
            d =3D root + fi[0]=0A=
            if os.path.exists(d):=0A=
                # Not fatal, but don't touch it=0A=
                continue=0A=
=0A=
            os.makedirs(d)=0A=
=0A=
            # Set permissions=0A=
            set_perm(d, fi[10], fi[11], perm)=0A=
=0A=
        elif stat.S_ISREG(mode):=0A=
            # It's a file=0A=
            #print 'File:'=0A=
=0A=
            # Check for %ghost files=0A=
            if fi[4] & rpm.RPMFILE_GHOST:=0A=
                print "Skipping %%ghost file: ", fi[0]=0A=
                continue=0A=
=0A=
            f =3D root + fi[0]=0A=
            if os.path.exists(f):=0A=
                print "Warning, file exists: ", f=0A=
                continue=0A=
=0A=
            d =3D os.path.dirname(f)=0A=
            if not os.path.exists(d):=0A=
                os.makedirs(d)=0A=
=0A=
            # prelink changes size and md5sum=0A=
            size =3D fi[1]=0A=
=0A=
            # check the size and mtime of currently installed file=0A=
            if os.path.exists(fi[0]) and os.path.getmtime(fi[0]) =3D=3D fi[=
3]:=0A=
=0A=
                # stuff from yum packages.py=0A=
=0A=
                # Use prelink_undo_cmd macro?=0A=
                prelink_cmd =3D "/usr/sbin/prelink"=0A=
                size_file =3D os.path.getsize(fi[0])=0A=
=0A=
                # determine what checksum algo to use=0A=
                csum_type =3D 'md5' # default for legacy=0A=
=0A=
                if hasattr(rpm, 'RPMTAG_FILEDIGESTALGO'):=0A=
                    csum_num =3D hdr[rpm.RPMTAG_FILEDIGESTALGO]=0A=
                    if csum_num:=0A=
                        if csum_num in RPM_CHECKSUM_TYPES:=0A=
                            csum_type =3D RPM_CHECKSUM_TYPES[csum_num]=0A=
=0A=
                if size_file !=3D size and os.path.exists(prelink_cmd):=0A=
=0A=
                    #  This is how rpm -V works, try and if that fails try=
=0A=
                    # again with prelink.=0A=
                    p =3D subprocess.Popen([prelink_cmd, "-y", fi[0]],=0A=
                        bufsize=3D-1, stdin=3Dsubprocess.PIPE,=0A=
                        stdout=3Dsubprocess.PIPE, stderr=3Dsubprocess.PIPE,=
 close_fds=3DTrue)=0A=
                    (ig, fp, er) =3D (p.stdin, p.stdout, p.stderr)=0A=
=0A=
                    md5file =3D misc.checksum(csum_type, fp)=0A=
=0A=
                    #print "File checksum: ", md5file=0A=
                    #print "RPM checksum: ", fi[12]=0A=
=0A=
                    # if original (not prelinked) checksum matches rpm data=
base,=0A=
                    # increase size to (post-)prelink size so that it match=
es current filesystem size=0A=
                    if md5file =3D=3D fi[12]:=0A=
                        size =3D size_file=0A=
                        #print "Prelink '", fi[0], "' ,using size: ", size=
=0A=
                    else:=0A=
                        print "RPM checksum does not match, using RPM size:=
 ", fi[0]=0A=
=0A=
                elif size_file =3D=3D size:=0A=
=0A=
                    # Compare checksum=0A=
                    md5file =3D misc.checksum(csum_type, fi[0])=0A=
=0A=
                    if md5file !=3D fi[12]:=0A=
                        print "WARNING RPM checksum does not match, skippin=
g file: ", fi[0]=0A=
                        continue=0A=
=0A=
            with open(f, "wb") as out:=0A=
                out.truncate(size)=0A=
=0A=
            # Set permissions=0A=
            set_perm(f, fi[10], fi[11], perm)=0A=
=0A=
            # Set mtime=0A=
            os.utime(f, (fi[3], fi[3]))=0A=
=0A=
        elif stat.S_ISLNK(mode):=0A=
            # It's a (symbolic) link=0A=
            #print 'Link:'=0A=
            pass=0A=
        else:=0A=
            # Unknown file type, print a message=0A=
            print 'Unknown file type, skipping: ', fi[0]=0A=
            continue=0A=
=0A=
        #print fi=0A=
        #print "f:  ", fi[0], "s:  ", fi[1], "m:  ", fi[2], "t:  ", fi[3], =
"f:  ", fi[4], "5:  ", fi[12]=0A=
        #print fi[0], "perm: ", oct(perm)=0A=

------=_NextPart_000_00A3_01CF1B40.D63554F0
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today. 
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
------=_NextPart_000_00A3_01CF1B40.D63554F0
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Dar-discussions mailing list
Dar-discussions-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/dar-discussions

------=_NextPart_000_00A3_01CF1B40.D63554F0--