Re: createrepo: external locking?

On Wed, Aug 25, 2010 at 04:46:46PM +0200, ext seth vidal wrote:
> On Wed, 2010-08-25 at 15:48 +0200, Oliver Hookins wrote:
> > > I don't see how createrepo locking the dir would help unless all other
> > > apps honored the locks quasi-sanely.
> > > 
> > > Is there a reason you cannot just make the program(s) which need to
> > > access the metadata run in sequence and not in parallel?
> > 
> 
> First:
> Could you subscribe to the list from the address you're posting from? I
> don't enjoy approving emails for delivery.

I was pretty sure I did subscribe already, before I posted initially. I just
tried to subscribe again and it sent me the standard privacy alert...

> 
> 
> > Not easily, and it's not desirable. The build process is often very long but the
> > RPM generation and subsequent metadata generation is relatively short. I don't
> > think we can put locks in our build system for just this part of the build,
> > sadly.
> 
> So then I'm confused again
> 
> 
> you have N build processes - at the end of the process they spit out an
> rpm or a set of rpms into your repo dir - and they run createrepo - but
> they could be finishing at roughly the same time and therefore will
> tread on one another?

That's pretty much it. We did all of the builds initially dropping their RPMs
into a single repository and had frequent collisions between createrepo runs.

> 
> Do they all need to update the metadata immediately following their
> builds?

Yes, as we have some continuous integration jobs which are triggered from the
finish of the build jobs that install the latest RPM from the repositories on
other machines.

> Can you have them notify the createrepo runner that their builds are
> completed?

Currently, we just run createrepo at the end of the build job, but as I
mentioned, running a single instance from cron isn't possible since we need the
metadata to be generated synchronously. If there is even a possibility that we
have to wait 59 seconds for the metadata to be next created, the triggered
integration job won't be able to see the new RPM in the repository when it runs.