Re: using cdb for 'update' without race condition?

[email protected] wrote:

> ari edelkind <[email protected]> wrote:
> > set -o noclobber
> >
> > while ! (exec >"$REMASTER_FILE") 2>&-
> > do
> > 	sleep 1
> > done
> 
> Ah, so you're talking about O_EXCL, not locking.  Yes, that will work,
> but again, you're busy-waiting instead of letting the kernel wake you
> up when it's your turn.

Yes, it's a form of lock, and it's what i used for illustration in this
example.  This method combines a clarification of what you presumed
"won't work" with a method that you conceded _would_ work, so it's a
fitting demonstration.  You can, frankly, use whatever form of locking
you desire; you still approach it in the same manner.  Rewriting cdb
files isn't a procedure that would be appropriate for a high-traffic
update environment anyway, so "busy-waiting" isn't necessarily a poor
tactic.

I should add that there aren't many options for shell scripts that can
_avoid_ "busy-waiting", but perhaps it would have been clearer if i'd
written:

------------------------------
REMASTER_FILE=$REMASTER_FILE perl -e 'use Fcntl ":flock";
    open (LOCK, ">>$ENV{REMASTER_FILE}") or die;
    die unless flock(LOCK, LOCK_EX); print "Y\n"; close(STDOUT);
    <STDIN>; # block until co-process input (or close)
' 2>/dev/null |&

read -p islocked; [[ "$islocked" = "Y" ]] || exit 1
...
------------------------------


You can make your own judgments.

To recap, when i say, "you still approach it in the same manner," the
"manner" to which i've been referring since my response to the original
post is:

    lock -> do whatever -> commit -> unlock

Depending on what you want to do, you can translate this into:

    set -o noclobber && (exec >target) && do_stuff_to_target \
        && commit; rm -f target

or

    set -o noclobber && (echo $$ >target.lock) && do_stuff_to_target \
        && commit; rm -f target.lock

or

    ln -s target.$myhost.$$ target.lock && do_stuff_to_target \
        && commit; rm -f target.lock
        # better over nfs, thanks to symlink(2)

or

    lockfile target.lock && do_stuff_to_target \
        && commit; rm -f target.lock
        # also better over nfs, thanks to link(2)
        # lets `lockfile` "busy-wait" for you
        # not all systems come with `lockfile`

or

    pseudocode:
    fd=open(target) && lock(fd) && do_stuff_to_target \
        && commit; unlock(fd)


Once again, i've intentionally left logic for handling errors and stale
lock files out of my examples.

As i've also stated, the "manner" can be mildly adjusted to:

    lock -> do whatever -> commit_and_unlock

If the lock file is the file to be committed itself, you can still
commit the file by renaming it into place (e.g. `mv file.tmp file.cdb`),
which will effectively unlock 'file.tmp' directly after it is committed.

That is, even if you still hold a kernel-level lock on the committed
open file descriptor, new processes attempting to create and acquire a
lock on a new 'file.tmp' can.  And this is what you want, because the
file has already been committed.


> Also, if the script gets killed in the middle
> of making an update, you'll have a stale remaster file blocking all
> further updates until you intervene manually.  With a permanent,
> separate file used just for locking, you have no busy-waiting and (if
> you're not on NFS) no chance of stale locks.

If you _are_ using nfs, stale locks will not be the greatest of your
locking problems using kernel-supported file locks (nor is a simple
open(...,O_CREAT|O_EXCL) sufficient in such a case, as i noted in a
prior e-mail).  Stale remaster files needn't require manual
intervention, either, if one takes appropriate measures in one's code.

I've taken a neutral stance on locking tactics from the start; i
couldn't care less which method you use.  You've made it clear that you,
personally, wouldn't use open(...,O_CREAT|O_EXCL) locking, and that's
fine.  Indeed, depending on your environment, perhaps only the
prjsuperduperlockprocessfilesystemnetworkthegoogleandtheinternets method
would be suitable.  Feel free to use it and be happy.

ari