Re: createrepo: huge memory consumption?

On Mon, 24 Aug 2009, Ryan Lynch wrote:

> My python is decent, and I have already been poking around in the
> code. I will see if I can come up with something and put some patches
> together.
>
> In the meantime, I suppose I can go without the '--update' option. Or,
> maybe I could split my repos up into smaller chunks, since the peak
> memory usage seems to be ~ proportional to the package count.
>

it is directly related to the size of the xml files - which is related to 
the number of pkgs in the repo.


> BTW: I have a couple of patches, now, that I wrote while trying to
> log, parse, and handle 'cobbler reposync' with large numbers
> directories. One affects output formatting, to make logging and
> parsing easier, and the other helps to handle an annoying error:
>
> - adding a '--log-friendly' option: eliminates the '\r'
> terminal-control magic chars; prints the individual package names, one
> per line, instead;
>
> - adding a '--remove-on-error': if RPM cannot parse a package file
> (which normally raises an exception), delete the offending file before
> barfing, so subsequent 'reposync' runs will re-download it;
>
> I think they could be helpful in automated situations, like 'cobbler'.
> I had much frustration: My sync cronjobs were constantly breaking
> down, and I wan't getting useful logs, so I made these changes.
>
> Any interest in either of these functions?

possibly, but send them over on yum-devel list - since they aren't related 
to createrepo.

thanks,
-sv