Re: createrepo: huge memory consumption?

On Sun, 23 Aug 2009, Ryan Lynch wrote:

> I've noticed that running 'createrepo' against large repositories
> takes up an enormous amount of memory, on my machines, at least.
>
> I'm currently using the following options:
>
> * --cachedir cache
> * --database
> * --update
> * --unique-md-filenames
>
> I see createrepo's 'genpkgmetadata.py' script eating up ~1.5 GB of
> memory, or 37% physical memory (runtime ~1 min).  I wouldn't have
> noticed this, except that it usually triggers the OOM killer and
> randomly whacks firefox, eclipse, etc.
>
> So, is this normal/expected?  Are other users seeing this happen with
> large repositories?  I've noticed that dropping the '--update' option
> takes a lot longer to run (~8-10 min), but with a much lower peak
> memory utilization.  Is it just a case of weighing memory usage vs.
> runtime, and making my choice?
>
> Thanks,
> Ryan

--update reads in ALL of the old metadata into memory so it can add/delete 
from it and dump it back out quickly.

the speed up you get from reading that in is at the cost of memory.

code to improve that is welcome.

-sv