Re: [jgroups-users] Locks get locked forever after a cluster members leaves

It took a while but we finally have a test that reproduces our problem. See https://issues.jboss.org/browse/JGRP-2234 and https://issues.jboss.org/browse/JGRP-2235

Met vriendelijke groet / Kind regards,
Bram Klein Gunnewiek | Shock Media B.V.

Tel: +31 (0)546 - 714360
Fax: +31 (0)546 - 714361
Web: https://www.shockmedia.nl/

On 10/23/2017 04:16 PM, Questions/problems related to using JGroups wrote:
> I'm afraid I don't have time to look into your code, but if you write a unit test which reproduces the issue *with JGroups code only*, I'll take a look.
>
> E.g. create 5 channels in the same JVM (each with a LockService), run tryLock() / unlock() cycles, make the coord leave and rejoin etc.
>
> From reading your previous emails, this should be able to reproduce the issue.
>
> On 23/10/17 16:10, Questions/problems related to using JGroups via javagroups-users wrote:
>> I wrote a whole reply about how the trylock thing works and I saw that I don't wrap the Condition returned by newCondition. Can this explain it? Conditions are used. My reply:
>>
>> The only reference to org.jgroups.blocks.locking.LockService#getLock(name) on my part is in JGroupsLockManager (actually, its the only class that has any reference to the JGroups LockService).
>>
>> The JGroupsLockManager is as I described: it wraps the JGroups lock in a LoggingLock that logs all method calls so we can debug it. It then wraps the LoggingLock in a TrylockWrappingLock that ensures that a call to lock() is done as a tryLock(...) throwing an exception if the lock could not be acquired within 20 seconds. I do this to prevent the application locking up when something bad (like a deadlock) happens. Both the LoggingLock and the TryLockWrappingLock implement java.util.concurrent.locks.Lock:
>>
>>      final Lock actualLock = lockService.getLock(name); // <-- lockService is a org.jgroups.blocks.locking.LockService instance (the only instance used)
>>      final LoggingLock loggingLock = new LoggingLock(name, actualLock);
>>      return new TrylockWrappingLock(loggingLock, timeout, TimeUnit.MILLISECONDS, this::printLockError);
>>
>> The lock() and lockInterruptibly() implementation of the TrylockWrappingLock looks like this:
>>
>>      try {
>>          if (!tryLock(maxTrylockTimeout, maxTrylockTimeUnit)) {
>>              throw new IllegalStateException(String.format("Regular lock request timed out on thread '[%d][%s]' with lock '%s'!", Thread.currentThread().getId(), Thread.currentThread().getName(), actualLock));
>>          }
>>      } catch (IllegalStateException e) {
>>          log.error(e.getMessage(), e);
>>          if (null != onLockTimeout) {
>>              try {
>>                  onLockTimeout.accept(actualLock);
>>              } catch (RuntimeException e2) {
>>                  log.warn(e2.getMessage(), e2);
>>              }
>>          }
>>          throw e;
>>      }
>>
>> The tryLock implementation looks like this:
>>
>>      boolean result = false;
>>      try {
>>          result = actualLock.tryLock(time, unit);
>>      } finally {
>>          if (!result) {
>>              silentUnlock();
>>          }
>>      }
>>      return result;
>>
>> And silentUnlock looks like this:
>>
>>      try {
>>          unlock();
>>      } catch (RuntimeException e) {
>>          /* ignore */
>>      }
>>
>> The tryLock implementation of the LoggingLock looks like this:
>>
>>      public boolean tryLock() {
>>          preAquireLock(track);
>>
>>          boolean result = actualLock.tryLock();
>>
>>          postAquireLock(track, result);
>>          return result;
>>      }
>>
>>      public boolean tryLock(long time, TimeUnit unit) throws InterruptedException {
>>          preAquireLock(track);
>>
>>          boolean result = actualLock.tryLock(time, unit);
>>
>>          postAquireLock(track, result);
>>          return result;
>>      }
>>
>> I see what you mean with the fact that the TL/L output is from a Request (and not a client or server lock).
>>
>> Met vriendelijke groet / Kind regards,
>> Bram Klein Gunnewiek | Shock Media B.V.
>>
>> Tel: +31 (0)546 - 714360
>> Fax: +31 (0)546 - 714361
>> Web:https://www.shockmedia.nl/
>>
>> On 10/23/2017 03:31 PM, Questions/problems related to using JGroups wrote:
>>>
>>>
>>> On 23/10/17 15:27, Questions/problems related to using JGroups via javagroups-users wrote:
>>>>
>>>> Met vriendelijke groet / Kind regards,
>>>> Bram Klein Gunnewiek | Shock Media B.V.
>>>>
>>>> Tel: +31 (0)546 - 714360
>>>> Fax: +31 (0)546 - 714361
>>>> Web:https://www.shockmedia.nl/
>>>>
>>>> On 10/23/2017 03:15 PM, Questions/problems related to using JGroups wrote:
>>>>>
>>>>>
>>>>> On 23/10/17 15:05, Questions/problems related to using JGroups via javagroups-users wrote:
>>>>>
>>>>>>> What do you mean? You must have called Lock.lock(), right? Below you mention that all lock() calls are changed into corresponding tryLock() calls, but the output you showed clearly indicate that the queue has a regular lock() request by 373 pending, followed by a tryLock() request...
>>>>>>
>>>>>> I need some more time to reply to your first response but this one is easy. I *never* use a regular lock() call on the JGroups lock (I wrap them, see example), the only calls done on the JGroups lock are tryLock(..) calls.
>>>>>
>>>>> Well, why am I seeing a regular lock() request in the queue then (see the first queue element below)?
>>>>>
>>>>>   queue:
>>>>> L(writelock/regular/instance_2_1/,40388::373)
>>>>> TL(writelock/regular/instance_2_1/,45507::413,20000)
>>>>>
>>>>> The  first queued lock request is clearly a Request for a lock() , *not* a tryLock()!
>>>>
>>>> I'll check this again. What my assumption whas that something had reset the is_trylock flag while the lock was still in the queue but I could be wrong.
>>>
>>> That's impossible; we're talking about a Request, and Request.is_try_lock is (pretty much) immutable!
>>>
>>>
>>>>> -
>>>>> Bela Ban | http://www.jgroups.org
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Check out the vibrant tech community on one of the world's most
>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>> _______________________________________________
>>>>> javagroups-users mailing list
>>>>> [email protected]
>>>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> javagroups-users mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>>
>>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>
>>
>>
>> _______________________________________________
>> javagroups-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users