Re: CVS access problem
Hans-Peter Jansen <[email protected]>
| Newsgroups | gmane.linux.rpm.metadata |
|---|---|
| Message-ID | <[email protected]> |
Hi Paul, Am Mittwoch, 19. Juli 2006 14:35 schrieb Paul Nasrat: > On Wed, 2006-06-28 at 19:03 +0200, Hans-Peter Jansen wrote: > > Am Mittwoch, 28. Juni 2006 17:37 schrieb Jeff Johnson: > > > Hint: > > > > Seth, for your convenience, it's also attached.. Without this, e.g. > > VMwareWorkstation rpms will get the None cache file chksum tag, > > which defeats its purpose. On the downside, it requires to rebuild > > the whole repocache. > > Hans is this the only outstanding patch of yours? Yep, no other serious issues from my side (other than it could do much faster, but unfortunately no spare time to tackle this [again]). [I wrote this three hours ago] But since you asked so nice, I just took the opportunity to implement timestamp checking again. From createrepo.8: -C --checkts Don't generate repo metadata, if their timestamps are newer than its rpms. This option decreases the processing time drastically again, if you happen to run it on an unmodified repo, but it is (currently) mutual exclusive with the --split option. I did this already a year ago, but took the wrong timestamp, which made it rather unreliable. In the meantime, I've learned from another problem, that there is no simple way to manipulate file ctimes other than setting a different system time. I think, this implementation is quite robust, if the repo is located on a decent file system (other the vfat and the like). It's running since about an hour here. For my usage pattern, it's a great achievement, since I locally mirror a bunch of repos with rsync a few times per day, and trigger createrepo on each, which is quite noticeable when using the cachedir option only. Here are some numbers (admittedly a pathologic case): # du -sh inst-source 13G inst-source # find inst-source -name \*.rpm | wc -l 7619 First run, no cache, no repo: # time createrepo -q inst-source real 4m1.900s user 3m31.414s sys 0m20.946s Second run: # time createrepo -qc repocache inst-source real 1m42.746s user 1m30.029s sys 0m2.385s Now with timestamp check: # time createrepo -qCc repocache inst-source real 0m0.699s user 0m0.456s sys 0m0.208s Not too bad, isn't it ;-) Let me know, what you think. > If so I'll commit > it and release later today - else send me the patch set and then I'll > do the same. Great, thanks for caring. Pete _______________________________________________ Rpm-metadata mailing list [email protected] https://lists.dulug.duke.edu/mailman/listinfo/rpm-metadata
createrepo-checktsoption-take3.diff
(text/x-diff, 5.1 KB)
--- genpkgmetadata.py.orig 2006-07-19 19:39:07.042852436 +0200
+++ genpkgmetadata.py 2006-07-19 21:29:53.893998466 +0200
@@ -55,6 +55,8 @@ def usage(retval=1):
(<filename> relative to directory-of-packages)
-v, --verbose = run verbosely
-c, --cachedir <dir> = specify which dir to use for the checksum cache
+ -C, --checkts = don't generate repo metadata, if their ctimes are newer
+ than the rpm ctimes.
-h, --help = show this help
-V, --version = output version
-p, --pretty = output xml files in pretty format.
@@ -95,6 +97,17 @@ class MetaDataGenerator:
return filelist
+ def checkTimeStamps(self, directory):
+ if self.cmds['checkts']:
+ files = self.getFileList(self.cmds['basedir'], directory, '.rpm', [])
+ files = self.trimRpms(files)
+ for f in files:
+ fn = os.path.join(self.cmds['basedir'], directory, f)
+ if not os.path.exists(fn):
+ errorprint(_('cannot get to file: %s') % fn)
+ if os.path.getctime(fn) > self.cmds['mdtimestamp']:
+ return False
+ return True
def trimRpms(self, files):
badrpms = []
@@ -348,17 +361,19 @@ def parseArgs(args):
cmds['cachedir'] = None
cmds['basedir'] = os.getcwd()
cmds['cache'] = False
+ cmds['checkts'] = False
+ cmds['mdtimestamp'] = 0
cmds['split'] = False
cmds['outputdir'] = ""
cmds['file-pattern-match'] = ['.*bin\/.*', '^\/etc\/.*', '^\/usr\/lib\/sendmail$']
cmds['dir-pattern-match'] = ['.*bin\/.*', '^\/etc\/.*']
try:
- gopts, argsleft = getopt.getopt(args, 'phqVvng:s:x:u:c:o:', ['help', 'exclude=',
+ gopts, argsleft = getopt.getopt(args, 'phqVvng:s:x:u:c:o:C', ['help', 'exclude=',
'quiet', 'verbose', 'cachedir=', 'basedir=',
'baseurl=', 'groupfile=', 'checksum=',
'version', 'pretty', 'split', 'outputdir=',
- 'noepoch'])
+ 'noepoch', 'checkts'])
except getopt.error, e:
errorprint(_('Options Error: %s.') % e)
usage()
@@ -416,6 +431,8 @@ def parseArgs(args):
elif arg in ['-c', '--cachedir']:
cmds['cache'] = True
cmds['cachedir'] = a
+ elif arg in ['-C', '--checkts']:
+ cmds['checkts'] = True
elif arg == '--basedir':
cmds['basedir'] = a
elif arg in ['-o','--outputdir']:
@@ -427,6 +444,10 @@ def parseArgs(args):
errorprint(_('Options Error: %s') % e)
usage()
+ if cmds['split'] and cmds['checkts']:
+ errorprint(_('--split and --checkts options are mutually exclusive'))
+ sys.exit(1)
+
directory = directories[0]
# Fix paths
directory = os.path.normpath(directory)
@@ -507,13 +528,21 @@ def main(args):
if not os.access(filepath, os.W_OK):
errorprint(_('error in must be able to write to metadata files:\n -> %s') % filepath)
usage()
-
+ if cmds['checkts']:
+ ts = os.path.getctime(filepath)
+ if ts > cmds['mdtimestamp']:
+ cmds['mdtimestamp'] = ts
+
if cmds['split']:
cmds['basedir'] = oldbase
mdgen = SplitMetaDataGenerator(cmds)
mdgen.doPkgMetadata(directories)
else:
mdgen = MetaDataGenerator(cmds)
+ if cmds['checkts'] and mdgen.checkTimeStamps(directory):
+ if cmds['verbose']:
+ print _('repo is up to date')
+ sys.exit(0)
mdgen.doPkgMetadata(directory)
mdgen.doRepoMetadata()
--- ChangeLog.orig 2006-07-19 21:12:39.546793958 +0200
+++ ChangeLog 2006-07-19 21:29:48.365194764 +0200
@@ -1,3 +1,10 @@
+2006-07-19 21:03 [email protected]
+
+ * genpkgmetadata.py, docs/createrepo.8:
+ -C, --checkts option added to avoid metadata generation, if ctime
+ filestamps are up to date. It's currently mutually exclusive with
+ the --split option.
+
2006-06-28 17:03 [email protected]
* dumpMetadata.py:
--- docs/createrepo.8.orig 2006-07-19 21:12:54.095275300 +0200
+++ docs/createrepo.8 2006-07-19 21:29:45.396300173 +0200
@@ -32,6 +32,12 @@ cache of checksums of packages in the re
createrepo over the same repository of files that do not have a complete
change out of all packages this decreases the processing time dramatically.
.br
+.IP "\fB\-C --checkts\fP"
+Don't generate repo metadata, if their timestamps are newer than its rpms.
+This option decreases the processing time drastically again, if you happen
+to run it on an unmodified repo, but it is (currently) mutual exclusive
+with the --split option.
+.br
.IP "\fB\--split\fP"
Run in split media mode. Rather than pass a single directory, take a set of
directories corresponding to different volumes in a media set.