[PATCH] Race condition in svnsync can wedge a mirror repository

Jon Foster <[email protected]>
Newsgroups gmane.comp.version-control.subversion.devel,gmane.comp.version-control.subversion.rapidsvn.devel
Message-ID <[email protected]>
Hi,

I think I've found a bug in svnsync's locking code, that can cause
a stale "svn:sync-lock" revprop to be left in the repository.  This
will prevent svnsync from being run again, until an administrator
manually deletes that property.

This is triggered by another process having the lock, and releasing
it between 8 and 9 seconds after svnsync is started.  (Any earlier
and svnsync will successfully take the lock.  Any later and svnsync
will give up and not try to take the lock).

This bug was found by code inspection.  I have confirmed it can be
reproduced with the test script attached, using Subversion 1.6.6 on
Linux.  After the script is run, the "svn:sync-lock" revprop should
not exist, but it does.  (It is a timing-related problem, so you may
need to try a few times.  With a hot cache, I can reproduce it 100%
of the time using this script).

The issue is with the get_lock() function in
subversion/svnsync/main.c.  The algorithm used is:
>  for (i = 0; i < 10; ++i)
>    {
>      [...]
>      SVN_ERR(svn_ra_rev_prop(session, 0, "svn:sync-lock",
>                              &reposlocktoken, subpool));
>
>      if (reposlocktoken)
>        [...]
>      else
>        {
>          SVN_ERR(svn_ra_change_rev_prop(session, 0,
>                                         "svn:sync-lock",
>                                         mylocktoken, subpool));
>          // BUG: if i == 9 then we're about to return failure.
>          //      But we just took the lock!
>        }
>    }
>
>  return svn_error_createf(APR_EINVAL, NULL,
>                           "Couldn't get lock on destination "
>                           "repos after %d attempts\n", i);

If the "else" branch is entered on the last iteration of the loop,
then the "svn:sync-lock" revprop is set but the function returns
failure.  This wedges the repository.

A patch is attached.  The patch is against 1.6.6.  I've tried to
keep the changes minimal, because I think this might be a candidate
for a Subversion 1.6.7 patch release.

[[[
Fix svnserve bug that could leave repository locked.

* subversion/svnsync/main.c
  (get_lock): Move loop exit point so we cannot drop out of the
   loop (and return failure) immediately after successfully
   locking the mirror.
]]]

Kind regards,

Jon Foster




**********************************************************************
This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Cabot Communications Ltd.

If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone.

Cabot Communications Limited
Verona House, Filwood Road, Bristol BS16 3RY, UK
+44 (0) 1179584232

Co. Registered in England number 02817269

Please contact the sender if you believe you have received this email in error.

**********************************************************************


______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2424345

Please start new threads on the <[email protected]> mailing list.
To subscribe to the new list, send an empty e-mail to <[email protected]>.
svnsync-8-9-bug-demo.sh (application/octet-stream, 1.3 KB)
#! /bin/bash

# Create two repositories, $source_url and $target_url,
# with revprop changes enabled in $target_url.
#
# The "rm -rf" lines are commented out for safety; feel free to uncomment them
# if you want to run this script repeatedly
scratch_dir=`pwd`
#rm -rf $scratch_dir/source
#rm -rf $scratch_dir/target
svnadmin create $scratch_dir/source
svnadmin create $scratch_dir/target
echo '#! /bin/bash' >$scratch_dir/target/hooks/pre-revprop-change
chmod +x $scratch_dir/target/hooks/pre-revprop-change
source_url=file://$scratch_dir/source
target_url=file://$scratch_dir/target

# Set up svnsync between the two repositories
svnsync init $target_url $source_url

# Lock the target repo.  (We're pretending to be another instance of svnsync).
svn propset svn:sync-lock --revprop -r0 'test:3bd51d9e-7d3f-4e90-af25-78de900e46b5' $target_url

# Schedule an unlock for 8.5 seconds in the future.
(sleep 8.5 && svn propdel svn:sync-lock --revprop -r0 $target_url) &

# Run svnsync
svnsync sync $target_url

# Wait for the unlock process to complete.
for job in `jobs -p`; do
    wait $job
done

# Print results.
# NOTE: The lock should not exist.  svnsync should either take and release
# the lock, or never take it at all.
echo Checking if lock property exists...
svn propget svn:sync-lock --revprop -r0 $target_url
svnsync-8-9-bug-fix.patch (application/octet-stream, 2.7 KB)
diff -urp orig/subversion-1.6.6/subversion/svnsync/main.c subversion-1.6.6/subversion/svnsync/main.c
--- orig/subversion-1.6.6/subversion/svnsync/main.c	2009-08-17 19:41:38.000000000 +0100
+++ subversion-1.6.6/subversion/svnsync/main.c	2009-11-25 17:02:50.000000000 +0000
@@ -235,7 +235,7 @@ get_lock(svn_ra_session_t *session, apr_
   svn_string_t *mylocktoken, *reposlocktoken;
   apr_status_t apr_err;
   apr_pool_t *subpool;
-  int i;
+  int i = 0;
 
   apr_err = apr_gethostname(hostname_str, sizeof(hostname_str), pool);
   if (apr_err)
@@ -246,7 +246,7 @@ get_lock(svn_ra_session_t *session, apr_
 
   subpool = svn_pool_create(pool);
 
-  for (i = 0; i < 10; ++i)
+  for (;;)
     {
       svn_pool_clear(subpool);
       SVN_ERR(check_cancel(NULL));
@@ -259,18 +259,39 @@ get_lock(svn_ra_session_t *session, apr_
           /* Did we get it?   If so, we're done, otherwise we sleep. */
           if (strcmp(reposlocktoken->data, mylocktoken->data) == 0)
             return SVN_NO_ERROR;
-          else
-            {
-              SVN_ERR(svn_cmdline_printf
-                      (pool, _("Failed to get lock on destination "
-                               "repos, currently held by '%s'\n"),
-                       reposlocktoken->data));
 
-              apr_sleep(apr_time_from_sec(1));
-            }
+          SVN_ERR(svn_cmdline_printf
+                  (pool, _("Failed to get lock on destination "
+                           "repos, currently held by '%s'\n"),
+                   reposlocktoken->data));
+
+          /* We check loop condition before we sleep, to
+           * avoid doing a pointless sleep at the end.
+           *
+           * XXX: If ++i==9 here then we cannot possibly take the lock.
+           * Even if it's released next time we look, we're going to
+           * bail out of the loop just before grabbing the lock.
+           * Is there any point in sleeping and going arond the loop
+           * again?  (Note that a fix for this should maybe wait for
+           * 1.7, it's probably not worth worrying about in 1.6.x).
+           */
+          ++i;
+          if (i >= 10)
+            break;
+
+          apr_sleep(apr_time_from_sec(1));
         }
       else
         {
+          /* We check loop condition before we try to take the lock,
+           * to avoid the bug where we take the lock then report
+           * a failure - that would permanently wedge the repository,
+           * requiring administrator intervention to fix.
+           */
+          ++i;
+          if (i >= 10)
+            break;
+
           SVN_ERR(svn_ra_change_rev_prop(session, 0, SVNSYNC_PROP_LOCK,
                                          mylocktoken, subpool));
         }
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.