Re: [jgroups-users] Locks get locked forever after a cluster members leaves

Questions/problems related to using JGroups <[email protected]> Mon, 23 Oct 2017 16:16:54 +0200
Newsgroups gmane.comp.java.javagroups.general
Message-ID <mailman.16765.1508768229.12770.javagroups-users@lists.sourceforge.net>
I'm afraid I don't have time to look into your code, but if you write a 
unit test which reproduces the issue *with JGroups code only*, I'll take 
a look.

E.g. create 5 channels in the same JVM (each with a LockService), run 
tryLock() / unlock() cycles, make the coord leave and rejoin etc.

 From reading your previous emails, this should be able to reproduce the 
issue.

On 23/10/17 16:10, Questions/problems related to using JGroups via 
javagroups-users wrote:
> I wrote a whole reply about how the trylock thing works and I saw that I 
> don't wrap the Condition returned by newCondition. Can this explain it? 
> Conditions are used. My reply:
> 
> The only reference to 
> org.jgroups.blocks.locking.LockService#getLock(name) on my part is in 
> JGroupsLockManager (actually, its the only class that has any reference 
> to the JGroups LockService).
> 
> The JGroupsLockManager is as I described: it wraps the JGroups lock in a 
> LoggingLock that logs all method calls so we can debug it. It then wraps 
> the LoggingLock in a TrylockWrappingLock that ensures that a call to 
> lock() is done as a tryLock(...) throwing an exception if the lock could 
> not be acquired within 20 seconds. I do this to prevent the application 
> locking up when something bad (like a deadlock) happens. Both the 
> LoggingLock and the TryLockWrappingLock implement 
> java.util.concurrent.locks.Lock:
> 
>      final Lock actualLock = lockService.getLock(name); // <-- 
> lockService is a org.jgroups.blocks.locking.LockService instance (the 
> only instance used)
>      final LoggingLock loggingLock = new LoggingLock(name, actualLock);
>      return new TrylockWrappingLock(loggingLock, timeout, 
> TimeUnit.MILLISECONDS, this::printLockError);
> 
> The lock() and lockInterruptibly() implementation of the 
> TrylockWrappingLock looks like this:
> 
>      try {
>          if (!tryLock(maxTrylockTimeout, maxTrylockTimeUnit)) {
>              throw new IllegalStateException(String.format("Regular lock 
> request timed out on thread '[%d][%s]' with lock '%s'!", 
> Thread.currentThread().getId(), Thread.currentThread().getName(), 
> actualLock));
>          }
>      } catch (IllegalStateException e) {
>          log.error(e.getMessage(), e);
>          if (null != onLockTimeout) {
>              try {
>                  onLockTimeout.accept(actualLock);
>              } catch (RuntimeException e2) {
>                  log.warn(e2.getMessage(), e2);
>              }
>          }
>          throw e;
>      }
> 
> The tryLock implementation looks like this:
> 
>      boolean result = false;
>      try {
>          result = actualLock.tryLock(time, unit);
>      } finally {
>          if (!result) {
>              silentUnlock();
>          }
>      }
>      return result;
> 
> And silentUnlock looks like this:
> 
>      try {
>          unlock();
>      } catch (RuntimeException e) {
>          /* ignore */
>      }
> 
> The tryLock implementation of the LoggingLock looks like this:
> 
>      public boolean tryLock() {
>          preAquireLock(track);
> 
>          boolean result = actualLock.tryLock();
> 
>          postAquireLock(track, result);
>          return result;
>      }
> 
>      public boolean tryLock(long time, TimeUnit unit) throws 
> InterruptedException {
>          preAquireLock(track);
> 
>          boolean result = actualLock.tryLock(time, unit);
> 
>          postAquireLock(track, result);
>          return result;
>      }
> 
> I see what you mean with the fact that the TL/L output is from a Request 
> (and not a client or server lock).
> 
> Met vriendelijke groet / Kind regards,
> Bram Klein Gunnewiek | Shock Media B.V.
> 
> Tel: +31 (0)546 - 714360
> Fax: +31 (0)546 - 714361
> Web:https://www.shockmedia.nl/
> 
> On 10/23/2017 03:31 PM, Questions/problems related to using JGroups wrote:
>>
>>
>> On 23/10/17 15:27, Questions/problems related to using JGroups via 
>> javagroups-users wrote:
>>>
>>> Met vriendelijke groet / Kind regards,
>>> Bram Klein Gunnewiek | Shock Media B.V.
>>>
>>> Tel: +31 (0)546 - 714360
>>> Fax: +31 (0)546 - 714361
>>> Web:https://www.shockmedia.nl/
>>>
>>> On 10/23/2017 03:15 PM, Questions/problems related to using JGroups 
>>> wrote:
>>>>
>>>>
>>>> On 23/10/17 15:05, Questions/problems related to using JGroups via 
>>>> javagroups-users wrote:
>>>>
>>>>>> What do you mean? You must have called Lock.lock(), right? Below 
>>>>>> you mention that all lock() calls are changed into corresponding 
>>>>>> tryLock() calls, but the output you showed clearly indicate that 
>>>>>> the queue has a regular lock() request by 373 pending, followed by 
>>>>>> a tryLock() request...
>>>>>
>>>>> I need some more time to reply to your first response but this one 
>>>>> is easy. I *never* use a regular lock() call on the JGroups lock (I 
>>>>> wrap them, see example), the only calls done on the JGroups lock 
>>>>> are tryLock(..) calls.
>>>>
>>>> Well, why am I seeing a regular lock() request in the queue then 
>>>> (see the first queue element below)?
>>>>
>>>>   queue:
>>>>                 L(writelock/regular/instance_2_1/,40388::373)
>>>> TL(writelock/regular/instance_2_1/,45507::413,20000)
>>>>
>>>> The  first queued lock request is clearly a Request for a lock() , 
>>>> *not* a tryLock()!
>>>
>>> I'll check this again. What my assumption whas that something had 
>>> reset the is_trylock flag while the lock was still in the queue but I 
>>> could be wrong.
>>
>> That's impossible; we're talking about a Request, and 
>> Request.is_try_lock is (pretty much) immutable!
>>
>>
>>>> -
>>>> Bela Ban | http://www.jgroups.org
>>>>
>>>>
>>>> ------------------------------------------------------------------------------ 
>>>>
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>> _______________________________________________
>>>> javagroups-users mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------ 
>>>
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>
>>>
>>>
>>> _______________________________________________
>>> javagroups-users mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>
>>
> 
> 
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> 
> 
> 
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
> 

-- 
Bela Ban | http://www.jgroups.org


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users