Re: [lttng-dev] [PATCH v2 05/12] urcu/uatomic: Add CMM memory model
"Paul E. McKenney via lttng-dev" <[email protected]>
| Newsgroups | org.lttng.lists.lttng-dev |
|---|---|
| Message-ID | <b669606b-74bf-4a07-8094-babbd829d9e8@paulmck-laptop> |
On Wed, Jun 07, 2023 at 02:53:52PM -0400, Olivier Dion wrote: > Introducing the CMM memory model with the following new primitives: > > - uatomic_load(addr, memory_order) > > - uatomic_store(addr, value, memory_order) > - uatomic_and_mo(addr, mask, memory_order) > - uatomic_or_mo(addr, mask, memory_order) > - uatomic_add_mo(addr, value, memory_order) > - uatomic_sub_mo(addr, value, memory_order) > - uatomic_inc_mo(addr, memory_order) > - uatomic_dec_mo(addr, memory_order) > > - uatomic_add_return_mo(addr, value, memory_order) > - uatomic_sub_return_mo(addr, value, memory_order) > > - uatomic_xchg_mo(addr, value, memory_order) > > - uatomic_cmpxchg_mo(addr, old, new, > memory_order_success, > memory_order_failure) > > The CMM memory model reflects the C11 memory model with an additional > CMM_SEQ_CST_FENCE memory order. The memory order can be selected through > the enum cmm_memorder. > > * With Atomic Builtins > > If configured with atomic builtins, the correspondence between the CMM > memory model and the C11 memory model is a one to one at the exception > of the CMM_SEQ_CST_FENCE memory order which implies the memory order > CMM_SEQ_CST and a thread fence after the operation. > > * Without Atomic Builtins > > However, if not configured with atomic builtins, the following stipulate > the memory model. > > For load operations with uatomic_load(), the memory orders CMM_RELAXED, > CMM_CONSUME, CMM_ACQUIRE, CMM_SEQ_CST and CMM_SEQ_CST_FENCE are > allowed. A barrier may be inserted before and after the load from memory > depending on the memory order: > > - CMM_RELAXED: No barrier > - CMM_CONSUME: Memory barrier after read > - CMM_ACQUIRE: Memory barrier after read > - CMM_SEQ_CST: Memory barriers before and after read > - CMM_SEQ_CST_FENCE: Memory barriers before and after read > > For store operations with uatomic_store(), the memory orders > CMM_RELAXED, CMM_RELEASE, CMM_SEQ_CST and CMM_SEQ_CST_FENCE are > allowed. A barrier may be inserted before and after the store to memory > depending on the memory order: > > - CMM_RELAXED: No barrier > - CMM_RELEASE: Memory barrier before operation > - CMM_SEQ_CST: Memory barriers before and after operation > - CMM_SEQ_CST_FENCE: Memory barriers before and after operation > > For load/store operations with uatomic_and_mo(), uatomic_or_mo(), > uatomic_add_mo(), uatomic_sub_mo(), uatomic_inc_mo(), uatomic_dec_mo(), > uatomic_add_return_mo() and uatomic_sub_return_mo(), all memory orders > are allowed. A barrier may be inserted before and after the operation > depending on the memory order: > > - CMM_RELAXED: No barrier > - CMM_ACQUIRE: Memory barrier after operation > - CMM_CONSUME: Memory barrier after operation > - CMM_RELEASE: Memory barrier before operation > - CMM_ACQ_REL: Memory barriers before and after operation > - CMM_SEQ_CST: Memory barriers before and after operation > - CMM_SEQ_CST_FENCE: Memory barriers before and after operation > > For the exchange operation uatomic_xchg_mo(), any memory order is > valid. A barrier may be inserted before and after the exchange to memory > depending on the memory order: > > - CMM_RELAXED: No barrier > - CMM_ACQUIRE: Memory barrier after operation > - CMM_CONSUME: Memory barrier after operation > - CMM_RELEASE: Memory barrier before operation > - CMM_ACQ_REL: Memory barriers before and after operation > - CMM_SEQ_CST: Memory barriers before and after operation > - CMM_SEQ_CST_FENCE: Memory barriers before and after operation > > For the compare exchange operation uatomic_cmpxchg_mo(), the success > memory order can be anything while the failure memory order cannot be > CMM_RELEASE nor CMM_ACQ_REL and cannot be stronger than the success > memory order. A barrier may be inserted before and after the store to > memory depending on the memory orders: > > Success memory order: > > - CMM_RELAXED: No barrier > - CMM_ACQUIRE: Memory barrier after operation > - CMM_CONSUME: Memory barrier after operation > - CMM_RELEASE: Memory barrier before operation > - CMM_ACQ_REL: Memory barriers before and after operation > - CMM_SEQ_CST: Memory barriers before and after operation > - CMM_SEQ_CST_FENCE: Memory barriers before and after operation > > Barriers after the operations are only emitted if the compare exchange > succeed. > > Failure memory order: > - CMM_RELAXED: No barrier > - CMM_ACQUIRE: Memory barrier after operation > - CMM_CONSUME: Memory barrier after operation > - CMM_SEQ_CST: Memory barriers before and after operation > - CMM_SEQ_CST_FENCE: Memory barriers before and after operation > > Barriers after the operations are only emitted if the compare exchange > failed. Barriers before the operation are never emitted by this > memory order. > > Change-Id: I213ba19c84e82a63083f00143a3142ffbdab1d52 > Co-authored-by: Mathieu Desnoyers <[email protected]> > Signed-off-by: Olivier Dion <[email protected]> > --- > doc/uatomic-api.md | 3 +- > include/Makefile.am | 2 + > include/urcu/static/pointer.h | 40 ++-- > include/urcu/uatomic.h | 63 ++++++- > include/urcu/uatomic/builtins-generic.h | 170 +++++++++++++++++ > include/urcu/uatomic/builtins.h | 79 ++++++++ > include/urcu/uatomic/generic.h | 234 ++++++++++++++++++++++++ > src/urcu-pointer.c | 9 +- > 8 files changed, 565 insertions(+), 35 deletions(-) > create mode 100644 include/urcu/uatomic/builtins-generic.h > create mode 100644 include/urcu/uatomic/builtins.h > > diff --git a/doc/uatomic-api.md b/doc/uatomic-api.md > index 0962399..7341ee8 100644 > --- a/doc/uatomic-api.md > +++ b/doc/uatomic-api.md > @@ -52,7 +52,8 @@ An atomic read-modify-write operation that performs this > sequence of operations atomically: check if `addr` contains `old`. > If true, then replace the content of `addr` by `new`. Return the > value previously contained by `addr`. This function implies a full > -memory barrier before and after the atomic operation. > +memory barrier before and after the atomic operation. The second memory > +barrier is only emitted if the operation succeeded. > > > ```c > diff --git a/include/Makefile.am b/include/Makefile.am > index ba1fe60..b20e56d 100644 > --- a/include/Makefile.am > +++ b/include/Makefile.am > @@ -63,6 +63,8 @@ nobase_include_HEADERS = \ > urcu/uatomic/alpha.h \ > urcu/uatomic_arch.h \ > urcu/uatomic/arm.h \ > + urcu/uatomic/builtins.h \ > + urcu/uatomic/builtins-generic.h \ > urcu/uatomic/gcc.h \ > urcu/uatomic/generic.h \ > urcu/uatomic.h \ > diff --git a/include/urcu/static/pointer.h b/include/urcu/static/pointer.h > index 9e46a57..9da8657 100644 > --- a/include/urcu/static/pointer.h > +++ b/include/urcu/static/pointer.h > @@ -96,23 +96,8 @@ extern "C" { > * -Wincompatible-pointer-types errors. Using the statement expression > * makes it an rvalue and gets rid of the const-ness. > */ > -#ifdef __URCU_DEREFERENCE_USE_ATOMIC_CONSUME > -# define _rcu_dereference(p) __extension__ ({ \ > - __typeof__(__extension__ ({ \ > - __typeof__(p) __attribute__((unused)) _________p0 = { 0 }; \ > - _________p0; \ > - })) _________p1; \ > - __atomic_load(&(p), &_________p1, __ATOMIC_CONSUME); \ There is talk of getting rid of memory_order_consume. But for the moment, it is what there is. Another alternative is to use a volatile load, similar to old-style CMM_LOAD_SHARED() or in-kernel READ_ONCE(). Thanx, Paul > - (_________p1); \ > - }) > -#else > -# define _rcu_dereference(p) __extension__ ({ \ > - __typeof__(p) _________p1 = CMM_LOAD_SHARED(p); \ > - cmm_smp_read_barrier_depends(); \ > - (_________p1); \ > - }) > -#endif > - > +# define _rcu_dereference(p) \ > + uatomic_load(&(p), CMM_CONSUME) > /** > * _rcu_cmpxchg_pointer - same as rcu_assign_pointer, but tests if the pointer > * is as expected by "old". If succeeds, returns the previous pointer to the > @@ -131,8 +116,9 @@ extern "C" { > ({ \ > __typeof__(*p) _________pold = (old); \ > __typeof__(*p) _________pnew = (_new); \ > - uatomic_cmpxchg(p, _________pold, _________pnew); \ > - }) > + uatomic_cmpxchg_mo(p, _________pold, _________pnew, \ > + CMM_SEQ_CST, CMM_SEQ_CST); \ > + }); > > /** > * _rcu_xchg_pointer - same as rcu_assign_pointer, but returns the previous > @@ -149,17 +135,17 @@ extern "C" { > __extension__ \ > ({ \ > __typeof__(*p) _________pv = (v); \ > - uatomic_xchg(p, _________pv); \ > + uatomic_xchg_mo(p, _________pv, \ > + CMM_SEQ_CST); \ > }) > > > -#define _rcu_set_pointer(p, v) \ > - do { \ > - __typeof__(*p) _________pv = (v); \ > - if (!__builtin_constant_p(v) || \ > - ((v) != NULL)) \ > - cmm_wmb(); \ > - uatomic_set(p, _________pv); \ > +#define _rcu_set_pointer(p, v) \ > + do { \ > + __typeof__(*p) _________pv = (v); \ > + uatomic_store(p, _________pv, \ > + __builtin_constant_p(v) && (v) == NULL ? \ > + CMM_RELAXED : CMM_RELEASE); \ > } while (0) > > /** > diff --git a/include/urcu/uatomic.h b/include/urcu/uatomic.h > index 2fb5fd4..be857e1 100644 > --- a/include/urcu/uatomic.h > +++ b/include/urcu/uatomic.h > @@ -21,9 +21,70 @@ > #ifndef _URCU_UATOMIC_H > #define _URCU_UATOMIC_H > > +#include <assert.h> > + > #include <urcu/arch.h> > +#include <urcu/config.h> > > -#if defined(URCU_ARCH_X86) > +enum cmm_memorder { > + CMM_RELAXED = 0, > + CMM_CONSUME = 1, > + CMM_ACQUIRE = 2, > + CMM_RELEASE = 3, > + CMM_ACQ_REL = 4, > + CMM_SEQ_CST = 5, > + CMM_SEQ_CST_FENCE = 6, > +}; > + > +#ifdef CONFIG_RCU_USE_ATOMIC_BUILTINS > + > +/* > + * Make sure that CMM_SEQ_CST_FENCE is not equivalent to other memory orders. > + */ > +# ifdef static_assert > +static_assert(CMM_RELAXED == __ATOMIC_RELAXED, ""); > +static_assert(CMM_CONSUME == __ATOMIC_CONSUME, ""); > +static_assert(CMM_ACQUIRE == __ATOMIC_ACQUIRE, ""); > +static_assert(CMM_RELEASE == __ATOMIC_RELEASE, ""); > +static_assert(CMM_ACQ_REL == __ATOMIC_ACQ_REL, ""); > +static_assert(CMM_SEQ_CST == __ATOMIC_SEQ_CST, ""); > +# endif > + > +/* > + * This is not part of the public API. It it used internally to implement the > + * CMM_SEQ_CST_FENCE memory order. > + * > + * NOTE: Using switch here instead of if statement to avoid -Wduplicated-cond > + * warning when memory order is conditionally determined. > + */ > +static inline void cmm_seq_cst_fence_after_atomic(enum cmm_memorder mo) > +{ > + switch (mo) { > + case CMM_SEQ_CST_FENCE: > + cmm_smp_mb(); > + break; > + default: > + break; > + } > +} > + > +#endif > + > +/* > + * This is not part of the public API. It is used internally to convert from the > + * CMM memory model to the C11 memory model. > + */ > +static inline int cmm_to_c11(int mo) > +{ > + if (mo == CMM_SEQ_CST_FENCE) { > + return CMM_SEQ_CST; > + } > + return mo; > +} > + > +#if defined(CONFIG_RCU_USE_ATOMIC_BUILTINS) > +#include <urcu/uatomic/builtins.h> > +#elif defined(URCU_ARCH_X86) > #include <urcu/uatomic/x86.h> > #elif defined(URCU_ARCH_PPC) > #include <urcu/uatomic/ppc.h> > diff --git a/include/urcu/uatomic/builtins-generic.h b/include/urcu/uatomic/builtins-generic.h > new file mode 100644 > index 0000000..673e888 > --- /dev/null > +++ b/include/urcu/uatomic/builtins-generic.h > @@ -0,0 +1,170 @@ > +/* > + * urcu/uatomic/builtins-generic.h > + * > + * Copyright (c) 2023 Olivier Dion <[email protected]> > + * > + * This library is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * This library is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with this library; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#ifndef _URCU_UATOMIC_BUILTINS_GENERIC_H > +#define _URCU_UATOMIC_BUILTINS_GENERIC_H > + > +#include <urcu/system.h> > + > +#define uatomic_store(addr, v, mo) \ > + __extension__ \ > + ({ \ > + __atomic_store_n(addr, v, cmm_to_c11(mo)); \ > + cmm_seq_cst_fence_after_atomic(mo); \ > + }) > + > +#define uatomic_set(addr, v) \ > + uatomic_store(addr, v, CMM_RELAXED) > + > +#define uatomic_load(addr, mo) \ > + __extension__ \ > + ({ \ > + __typeof__(*(addr)) _value = __atomic_load_n(addr, \ > + cmm_to_c11(mo)); \ > + cmm_seq_cst_fence_after_atomic(mo); \ > + \ > + _value; \ > + }) > + > +#define uatomic_read(addr) \ > + uatomic_load(addr, CMM_RELAXED) > + > +#define uatomic_cmpxchg_mo(addr, old, new, mos, mof) \ > + __extension__ \ > + ({ \ > + __typeof__(*(addr)) _old = (__typeof__(*(addr)))old; \ > + \ > + if (__atomic_compare_exchange_n(addr, &_old, new, 0, \ > + cmm_to_c11(mos), \ > + cmm_to_c11(mof))) { \ > + cmm_seq_cst_fence_after_atomic(mos); \ > + } else { \ > + cmm_seq_cst_fence_after_atomic(mof); \ > + } \ > + _old; \ > + }) > + > +#define uatomic_cmpxchg(addr, old, new) \ > + uatomic_cmpxchg_mo(addr, old, new, CMM_SEQ_CST_FENCE, CMM_RELAXED) > + > +#define uatomic_xchg_mo(addr, v, mo) \ > + __extension__ \ > + ({ \ > + __typeof__((*addr)) _old = __atomic_exchange_n(addr, v, \ > + cmm_to_c11(mo)); \ > + cmm_seq_cst_fence_after_atomic(mo); \ > + _old; \ > + }) > + > +#define uatomic_xchg(addr, v) \ > + uatomic_xchg_mo(addr, v, CMM_SEQ_CST_FENCE) > + > +#define uatomic_add_return_mo(addr, v, mo) \ > + __extension__ \ > + ({ \ > + __typeof__(*(addr)) _old = __atomic_add_fetch(addr, v, \ > + cmm_to_c11(mo)); \ > + cmm_seq_cst_fence_after_atomic(mo); \ > + _old; \ > + }) > + > +#define uatomic_add_return(addr, v) \ > + uatomic_add_return_mo(addr, v, CMM_SEQ_CST_FENCE) > + > +#define uatomic_sub_return_mo(addr, v, mo) \ > + __extension__ \ > + ({ \ > + __typeof__(*(addr)) _old = __atomic_sub_fetch(addr, v, \ > + cmm_to_c11(mo)); \ > + cmm_seq_cst_fence_after_atomic(mo); \ > + _old; \ > + }) > + > +#define uatomic_sub_return(addr, v) \ > + uatomic_sub_return_mo(addr, v, CMM_SEQ_CST_FENCE) > + > +#define uatomic_and_mo(addr, mask, mo) \ > + __extension__ \ > + ({ \ > + __typeof__(*(addr)) _old = __atomic_and_fetch(addr, mask, \ > + cmm_to_c11(mo)); \ > + cmm_seq_cst_fence_after_atomic(mo); \ > + _old; \ > + }) > + > +#define uatomic_and(addr, mask) \ > + (void) uatomic_and_mo(addr, mask, CMM_SEQ_CST) > + > +#define uatomic_or_mo(addr, mask, mo) \ > + __extension__ \ > + ({ \ > + __typeof__(*(addr)) _old = __atomic_or_fetch(addr, mask, \ > + cmm_to_c11(mo)); \ > + cmm_seq_cst_fence_after_atomic(mo); \ > + _old; \ > + }) > + > + > +#define uatomic_or(addr, mask) \ > + (void) uatomic_or_mo(addr, mask, CMM_RELAXED) > + > +#define uatomic_add_mo(addr, v, mo) \ > + (void) uatomic_add_return_mo(addr, v, mo) > + > +#define uatomic_add(addr, v) \ > + (void) uatomic_add_mo(addr, v, CMM_RELAXED) > + > +#define uatomic_sub_mo(addr, v, mo) \ > + (void) uatomic_sub_return_mo(addr, v, mo) > + > +#define uatomic_sub(addr, v) \ > + (void) uatomic_sub_mo(addr, v, CMM_RELAXED) > + > +#define uatomic_inc_mo(addr, mo) \ > + (void) uatomic_add_mo(addr, 1, mo) > + > +#define uatomic_inc(addr) \ > + (void) uatomic_inc_mo(addr, CMM_RELAXED) > + > +#define uatomic_dec_mo(addr, mo) \ > + (void) uatomic_sub_mo(addr, 1, mo) > + > +#define uatomic_dec(addr) \ > + (void) uatomic_dec_mo(addr, CMM_RELAXED) > + > +#define cmm_smp_mb__before_uatomic_and() cmm_smp_mb() > +#define cmm_smp_mb__after_uatomic_and() cmm_smp_mb() > + > +#define cmm_smp_mb__before_uatomic_or() cmm_smp_mb() > +#define cmm_smp_mb__after_uatomic_or() cmm_smp_mb() > + > +#define cmm_smp_mb__before_uatomic_add() cmm_smp_mb() > +#define cmm_smp_mb__after_uatomic_add() cmm_smp_mb() > + > +#define cmm_smp_mb__before_uatomic_sub() cmm_smp_mb() > +#define cmm_smp_mb__after_uatomic_sub() cmm_smp_mb() > + > +#define cmm_smp_mb__before_uatomic_inc() cmm_smp_mb() > +#define cmm_smp_mb__after_uatomic_inc() cmm_smp_mb() > + > +#define cmm_smp_mb__before_uatomic_dec() cmm_smp_mb() > +#define cmm_smp_mb__after_uatomic_dec() cmm_smp_mb() > + > +#endif /* _URCU_UATOMIC_BUILTINS_X86_H */ > diff --git a/include/urcu/uatomic/builtins.h b/include/urcu/uatomic/builtins.h > new file mode 100644 > index 0000000..82e98f8 > --- /dev/null > +++ b/include/urcu/uatomic/builtins.h > @@ -0,0 +1,79 @@ > +/* > + * urcu/uatomic/builtins.h > + * > + * Copyright (c) 2023 Olivier Dion <[email protected]> > + * > + * This library is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * This library is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with this library; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#ifndef _URCU_UATOMIC_BUILTINS_H > +#define _URCU_UATOMIC_BUILTINS_H > + > +#include <urcu/arch.h> > + > +#if defined(__has_builtin) > +# if !__has_builtin(__atomic_store_n) > +# error "Toolchain does not support __atomic_store_n." > +# endif > +# if !__has_builtin(__atomic_load_n) > +# error "Toolchain does not support __atomic_load_n." > +# endif > +# if !__has_builtin(__atomic_exchange_n) > +# error "Toolchain does not support __atomic_exchange_n." > +# endif > +# if !__has_builtin(__atomic_compare_exchange_n) > +# error "Toolchain does not support __atomic_compare_exchange_n." > +# endif > +# if !__has_builtin(__atomic_add_fetch) > +# error "Toolchain does not support __atomic_add_fetch." > +# endif > +# if !__has_builtin(__atomic_sub_fetch) > +# error "Toolchain does not support __atomic_sub_fetch." > +# endif > +# if !__has_builtin(__atomic_or_fetch) > +# error "Toolchain does not support __atomic_or_fetch." > +# endif > +# if !__has_builtin(__atomic_thread_fence) > +# error "Toolchain does not support __atomic_thread_fence." > +# endif > +# if !__has_builtin(__atomic_signal_fence) > +# error "Toolchain does not support __atomic_signal_fence." > +# endif > +#elif defined(__GNUC__) > +# define GCC_VERSION (__GNUC__ * 10000 + \ > + __GNUC_MINOR__ * 100 + \ > + __GNUC_PATCHLEVEL__) > +# if GCC_VERSION < 40700 > +# error "GCC version is too old. Version must be 4.7 or greater" > +# endif > +# undef GCC_VERSION > +#else > +# error "Toolchain is not supported." > +#endif > + > +#if defined(__GNUC__) > +# define UATOMIC_HAS_ATOMIC_BYTE __GCC_ATOMIC_CHAR_LOCK_FREE > +# define UATOMIC_HAS_ATOMIC_SHORT __GCC_ATOMIC_SHORT_LOCK_FREE > +#elif defined(__clang__) > +# define UATOMIC_HAS_ATOMIC_BYTE __CLANG_ATOMIC_CHAR_LOCK_FREE > +# define UATOMIC_HAS_ATOMIC_SHORT __CLANG_ATOMIC_SHORT_LOCK_FREE > +#else > +/* # define UATOMIC_HAS_ATOMIC_BYTE */ > +/* # define UATOMIC_HAS_ATOMIC_SHORT */ > +#endif > + > +#include <urcu/uatomic/builtins-generic.h> > + > +#endif /* _URCU_UATOMIC_BUILTINS_H */ > diff --git a/include/urcu/uatomic/generic.h b/include/urcu/uatomic/generic.h > index e31a19b..6b9c153 100644 > --- a/include/urcu/uatomic/generic.h > +++ b/include/urcu/uatomic/generic.h > @@ -33,10 +33,244 @@ extern "C" { > #define uatomic_set(addr, v) ((void) CMM_STORE_SHARED(*(addr), (v))) > #endif > > +extern void abort(void); > + > +#define uatomic_load_store_return_op(op, addr, v, mo) \ > + __extension__ \ > + ({ \ > + \ > + switch (mo) { \ > + case CMM_ACQUIRE: \ > + case CMM_CONSUME: \ > + case CMM_RELAXED: \ > + break; \ > + case CMM_RELEASE: \ > + case CMM_ACQ_REL: \ > + case CMM_SEQ_CST: \ > + case CMM_SEQ_CST_FENCE: \ > + cmm_smp_mb(); \ > + break; \ > + default: \ > + abort(); \ > + } \ > + \ > + __typeof__((*addr)) _value = op(addr, v); \ > + \ > + switch (mo) { \ > + case CMM_CONSUME: \ > + cmm_smp_read_barrier_depends(); \ > + break; \ > + case CMM_ACQUIRE: \ > + case CMM_ACQ_REL: \ > + case CMM_SEQ_CST: \ > + case CMM_SEQ_CST_FENCE: \ > + cmm_smp_mb(); \ > + break; \ > + case CMM_RELAXED: \ > + case CMM_RELEASE: \ > + break; \ > + default: \ > + abort(); \ > + } \ > + _value; \ > + }) > + > +#define uatomic_load_store_op(op, addr, v, mo) \ > + ({ \ > + switch (mo) { \ > + case CMM_ACQUIRE: \ > + case CMM_CONSUME: \ > + case CMM_RELAXED: \ > + break; \ > + case CMM_RELEASE: \ > + case CMM_ACQ_REL: \ > + case CMM_SEQ_CST: \ > + case CMM_SEQ_CST_FENCE: \ > + cmm_smp_mb(); \ > + break; \ > + default: \ > + abort(); \ > + } \ > + \ > + op(addr, v); \ > + \ > + switch (mo) { \ > + case CMM_CONSUME: \ > + cmm_smp_read_barrier_depends(); \ > + break; \ > + case CMM_ACQUIRE: \ > + case CMM_ACQ_REL: \ > + case CMM_SEQ_CST: \ > + case CMM_SEQ_CST_FENCE: \ > + cmm_smp_mb(); \ > + break; \ > + case CMM_RELAXED: \ > + case CMM_RELEASE: \ > + break; \ > + default: \ > + abort(); \ > + } \ > + }) > + > +#define uatomic_store(addr, v, mo) \ > + ({ \ > + switch (mo) { \ > + case CMM_RELAXED: \ > + break; \ > + case CMM_RELEASE: \ > + case CMM_SEQ_CST: \ > + case CMM_SEQ_CST_FENCE: \ > + cmm_smp_mb(); \ > + break; \ > + default: \ > + abort(); \ > + } \ > + \ > + uatomic_set(addr, v); \ > + \ > + switch (mo) { \ > + case CMM_RELAXED: \ > + case CMM_RELEASE: \ > + break; \ > + case CMM_SEQ_CST: \ > + case CMM_SEQ_CST_FENCE: \ > + cmm_smp_mb(); \ > + break; \ > + default: \ > + abort(); \ > + } \ > + }) > + > +#define uatomic_and_mo(addr, v, mo) \ > + uatomic_load_store_op(uatomic_and, addr, v, mo) > + > +#define uatomic_or_mo(addr, v, mo) \ > + uatomic_load_store_op(uatomic_or, addr, v, mo) > + > +#define uatomic_add_mo(addr, v, mo) \ > + uatomic_load_store_op(uatomic_add, addr, v, mo) > + > +#define uatomic_sub_mo(addr, v, mo) \ > + uatomic_load_store_op(uatomic_sub, addr, v, mo) > + > +#define uatomic_inc_mo(addr, mo) \ > + uatomic_load_store_op(uatomic_add, addr, 1, mo) > + > +#define uatomic_dec_mo(addr, mo) \ > + uatomic_load_store_op(uatomic_add, addr, -1, mo) > +/* > + * NOTE: We can not just do switch (_value == (old) ? mos : mof) otherwise the > + * compiler emit a -Wduplicated-cond warning. > + */ > +#define uatomic_cmpxchg_mo(addr, old, new, mos, mof) \ > + __extension__ \ > + ({ \ > + switch (mos) { \ > + case CMM_ACQUIRE: \ > + case CMM_CONSUME: \ > + case CMM_RELAXED: \ > + break; \ > + case CMM_RELEASE: \ > + case CMM_ACQ_REL: \ > + case CMM_SEQ_CST: \ > + case CMM_SEQ_CST_FENCE: \ > + cmm_smp_mb(); \ > + break; \ > + default: \ > + abort(); \ > + } \ > + \ > + __typeof__(*(addr)) _value = uatomic_cmpxchg(addr, old, \ > + new); \ > + \ > + if (_value == (old)) { \ > + switch (mos) { \ > + case CMM_CONSUME: \ > + cmm_smp_read_barrier_depends(); \ > + break; \ > + case CMM_ACQUIRE: \ > + case CMM_ACQ_REL: \ > + case CMM_SEQ_CST: \ > + case CMM_SEQ_CST_FENCE: \ > + cmm_smp_mb(); \ > + break; \ > + case CMM_RELAXED: \ > + case CMM_RELEASE: \ > + break; \ > + default: \ > + abort(); \ > + } \ > + } else { \ > + switch (mof) { \ > + case CMM_CONSUME: \ > + cmm_smp_read_barrier_depends(); \ > + break; \ > + case CMM_ACQUIRE: \ > + case CMM_ACQ_REL: \ > + case CMM_SEQ_CST: \ > + case CMM_SEQ_CST_FENCE: \ > + cmm_smp_mb(); \ > + break; \ > + case CMM_RELAXED: \ > + case CMM_RELEASE: \ > + break; \ > + default: \ > + abort(); \ > + } \ > + } \ > + _value; \ > + }) > + > +#define uatomic_xchg_mo(addr, v, mo) \ > + uatomic_load_store_return_op(uatomic_xchg, addr, v, mo) > + > +#define uatomic_add_return_mo(addr, v, mo) \ > + uatomic_load_store_return_op(uatomic_add_return, addr, v) > + > +#define uatomic_sub_return_mo(addr, v, mo) \ > + uatomic_load_store_return_op(uatomic_sub_return, addr, v) > + > + > #ifndef uatomic_read > #define uatomic_read(addr) CMM_LOAD_SHARED(*(addr)) > #endif > > +#define uatomic_load(addr, mo) \ > + __extension__ \ > + ({ \ > + switch (mo) { \ > + case CMM_ACQUIRE: \ > + case CMM_CONSUME: \ > + case CMM_RELAXED: \ > + break; \ > + case CMM_SEQ_CST: \ > + case CMM_SEQ_CST_FENCE: \ > + cmm_smp_mb(); \ > + break; \ > + default: \ > + abort(); \ > + } \ > + \ > + __typeof__(*(addr)) _rcu_value = uatomic_read(addr); \ > + \ > + switch (mo) { \ > + case CMM_RELAXED: \ > + break; \ > + case CMM_CONSUME: \ > + cmm_smp_read_barrier_depends(); \ > + break; \ > + case CMM_ACQUIRE: \ > + case CMM_SEQ_CST: \ > + case CMM_SEQ_CST_FENCE: \ > + cmm_smp_mb(); \ > + break; \ > + default: \ > + abort(); \ > + } \ > + \ > + _rcu_value; \ > + }) > + > #if !defined __OPTIMIZE__ || defined UATOMIC_NO_LINK_ERROR > #ifdef ILLEGAL_INSTR > static inline __attribute__((always_inline)) > diff --git a/src/urcu-pointer.c b/src/urcu-pointer.c > index d0854ac..cea8aeb 100644 > --- a/src/urcu-pointer.c > +++ b/src/urcu-pointer.c > @@ -39,19 +39,16 @@ void *rcu_dereference_sym(void *p) > > void *rcu_set_pointer_sym(void **p, void *v) > { > - cmm_wmb(); > - uatomic_set(p, v); > + uatomic_store(p, v, CMM_RELEASE); > return v; > } > > void *rcu_xchg_pointer_sym(void **p, void *v) > { > - cmm_wmb(); > - return uatomic_xchg(p, v); > + return uatomic_xchg_mo(p, v, CMM_SEQ_CST); > } > > void *rcu_cmpxchg_pointer_sym(void **p, void *old, void *_new) > { > - cmm_wmb(); > - return uatomic_cmpxchg(p, old, _new); > + return uatomic_cmpxchg_mo(p, old, _new, CMM_SEQ_CST, CMM_SEQ_CST); > } > -- > 2.40.1 > _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev