Re: createrepo: external locking?

On Wed, 2010-08-25 at 16:54 +0200, Oliver Hookins wrote:
> > First:
> > Could you subscribe to the list from the address you're posting from? I
> > don't enjoy approving emails for delivery.
> 
> I was pretty sure I did subscribe already, before I posted initially. I just
> tried to subscribe again and it sent me the standard privacy alert...
> 


You're subscribed now.


> > 
> > you have N build processes - at the end of the process they spit out an
> > rpm or a set of rpms into your repo dir - and they run createrepo - but
> > they could be finishing at roughly the same time and therefore will
> > tread on one another?
> 
> That's pretty much it. We did all of the builds initially dropping their RPMs
> into a single repository and had frequent collisions between createrepo runs.

Then it sounds to me like you need a proper build system with dependent
build relationships. Something like koji.

A few of other options:

1. have each process generate their repodata in another outputdir and
then copy the repodata over top of the path you want atomically.

 you still run the risk of there being races  there, however.

2. have a repo per-builder and use yum to merge the repos at
install/build time. Then each builder has their own repo they are
controlling uniquely.

Then when the builds are finished one process can merge all of those.

3. put a wrapper with a simple lockfile around createrepo so the other
processes wait until it is complete before they start.

I don't see putting this code into createrepo - if only b/c it is a bit
out of scope for createrepo, I think.

-sv