Re: [PATCH] switch --update to use the sqlite dbs instead of the xml files. Should massively impact memory footprint and hopefully only marginally impact performance.

On Wed, 2011-09-07 at 13:53 -0400, Dennis Gregorovic wrote:
> On Wed, 2011-09-07 at 12:58 -0400, seth vidal wrote:
> > On Wed, 2011-09-07 at 12:18 -0400, Dennis Gregorovic wrote:
> > > On Fri, 2011-07-15 at 17:53 -0400, Seth Vidal wrote:
> > > > -    def getNodes(self, relpath):
> > > > -        """Return base, filelist, and other nodes for file, if they exist
> > > >  
> > > > -        Returns a tuple of nodes, or None if not found
> > > > +    def getNodes(self, relpath):
> > > > +        """return a package object based on relative path of pkg
> > > >          """
> > > > -        bnode = self.basenodes.get(relpath,None)
> > > > -        if bnode is None:
> > > > -            return None
> > > > -        pkgid = self.pkg_ids.get(relpath,None)
> > > > -        if pkgid is None:
> > > > -            print _("No pkgid found for: %s") % relpath
> > > > -            return None
> > > > -        fnode = self.filesnodes.get(pkgid,None)
> > > > -        if fnode is None:
> > > > -            return None
> > > > -        onode = self.othernodes.get(pkgid,None)
> > > > -        if onode is None:
> > > > -            return None
> > > > -        return bnode, fnode, onode
> > > > -
> > > > -    def freeNodes(self,relpath):
> > > > -        #causing problems
> > > > -        """Free up nodes corresponding to file, if possible"""
> > > > -        bnode = self.basenodes.get(relpath,None)
> > > > -        if bnode is None:
> > > > -            print "Missing node for %s" % relpath
> > > > -            return
> > > > -        bnode.unlinkNode()
> > > > -        bnode.freeNode()
> > > > -        del self.basenodes[relpath]
> > > > -        pkgid = self.pkg_ids.get(relpath,None)
> > > > -        if pkgid is None:
> > > > -            print _("No pkgid found for: %s") % relpath
> > > > +        if relpath in self.pkg_tups_by_path:
> > > > +            pkgtup = self.pkg_tups_by_path[relpath]
> > > > +            return self._repo.sack.searchPkgTuple(pkgtup)[0]
> > > > +        else:
> > > > +            print _("No pkg found for: %s") % relpath
> > > >              return None
> > > 
> > > Seth,
> > > 
> > > Thank you very much for this patch.  I've locally backported it to RHEL
> > > 6 and the memory savings are tremendous.  However, I just noticed one
> > > small issue.  Before, the getNodes call would only print "No pkg found
> > > for: %" when a package was found in the basenodes cache but not the
> > > pkg_ids cache.  That was an edge condition that never actually happened
> > > as far as I can tell, so the net effect was that "No pkg found" did not
> > > get printed.  After the code change, "No pkg found" is now printed every
> > > time getNodes is called with a package that isn't cached.  
> > 
> > Yah - I think someone else noticed that. Dgilmore, I think.
> > 
> > I was a bit torn about it - since it is quasi-useful to know which ones
> > went walk-about and which ones did not. However, it isn't helpful for
> > the general case of 'yes, I know I removed pkgs from the repo, just
> > rebuild it'.
> > 
> > I suspect it is really only useful when debugging something ridiculous.
> > 
> > Anyone object to me just pulling it?
> > -sv
> 
> Maybe only print out the message if run with --verbose?
> 


I just yanked it - the only person who will probably need this debug
info is me and it is easy to do it manually (ls -lR on the dir and a
sqlite dump of pkg filenames)

thanks
-sv