[PATCH] libgomp: New barriers in new linux/futex_waitv target

From: Matthew Malcomson <[email protected]>

All changes discussed in the previous libgomp barrier patches.
(I.e. w.r.t. this patch is the entire most recent proposed patch for
barrier improvements in libgomp -- there are "overhead" improvements for
parallel regions left but nothing else w.r.t. barriers).
I believe this patch addresses everything brought up in the last review
(now have tree-based barrier chosen depending on environment):
https://gcc.gnu.org/pipermail/gcc-patches/2026-July/724801.html

As with recent patches sent upstream, not all testing completed yet
(bootstrapped & ran testsuite and checked micro-benchmark, but not ran
everything yet).

Working on PR119588.  From that PR:
We've seen on some internal workloads (NVPL BLAS running GEMM routine
on a small matrix) that the overhead of a `#pragma omp parallel`
statement when running with a high number of cores (72 or 144) is much
higher in libgomp when compared to LLVM's libomp.

In a program which has both some work that can be handled with high
parallelism (so OMP is running with many threads) and a large number
of small pieces of work that need to be performed with low overhead,
this has been seen to cause a significant overhead when accumulated.

------------------------------
Here I combine all improvements changes requested into one patch --
hoping that's an easier review -- I can easily separate them if that
helps.

N.b. preliminary performance results are showing not noticable impact of
the if condition on choosing a tree or flat barrier.

Changes from the previous 2-patch combination:
1) Put new barrier implementation into new linux/futex_waitv target.
   This target is chosen when the futex_waitv syscall is available.  It
   can be explicitly disabled via a configure argument.
2) Remove the futex_waitv fallback mechanism.  New barrier is only used
   when futex_waitv is available.
3) Merge `thread_lock_data` members `gen` and `cgen` into a single `gen`
   member.  These were separate in order to make the RMW "cancellation"
   increment atomic.  This is only necessary if the futex_waitv fallback
   is being used, since that was the only way more than one thread could
   modify the secondary thread-specific generation numbers.  Without
   that possibility the read and modification can be non-atomic within
   the secondary thread with the atomic write being necessary for
   acquire/release semantics.
4) Allocate thread-local generation numbers as part of the team
   allocation.  This can be done since the "simple" barrier that the
   thread pool uses does not use this structure (i.e. all barriers that
   use these thread-local generation numbers are part of a team).
5) Using the linux/futex.h header when configure has identified that the
   syscall is available.
6) Implement a "tree" based barrier (actually hypercube, but I believe
   tree is a term that conveys the majority of the meaning to more
   people while hypercube is not something that tells most people what
   is happening).
7) Choose between "tree" based or "flat" barrier depending on the
   environment variables.
8) Refactoring since the number of variations was getting confusing.

---------------------- 8< --------------- >8 --------------------------
This patch introduces two barriers that perform better on the
microbenchmark provided in PR119588.  I have ran various higher-level
benchmarks and have not seen anything I can claim as outside noise.

We introduce it in a new target directory of linux/futex_waitv since it
benefits from the futex_waitv syscall introduced in Linux 5.16.

The two barriers are a "flat" barrier, which usually performs better
when OMP_WAIT_POLICY=active, and a "tree" based barrier (to be precise
is a hypercube following LLVM's implementation, but I believe "tree"
conveys the most meaning to the most people) which usually performs
better when OMP_WAIT_POLICY=passive.  We choose between them based on
the `spincount` (the underlying variable determined by wait policy) and
if the user specifically requests a given barrier type with the new
environment variable GOMP_BARRIER_TYPE={flat,tree,auto}.

The outline of the flat barrier is:
1) Each thread associated with a barrier has its own ID (currently the
   team_id of the team that the current thread is a part of).
2) When a thread arrives at the barrier it marks its "arrived flag" as
   having arrived.
3) ID 0 (which is always the thread which does the bookkeeping for
   this team, at the top-level it's the primary thread) waits on each
   of the other threads "arrived flags".  It executes tasks while
   waiting on these threads to arrive.
4) Once ID 0 sees all other threads have arrived it signals to all
   secondary threads that the barrier is complete via incrementing the
   global generation (as usual).

While the outline of the tree-based barrier is:
1) Each thread associated with a barrier has its own ID (currently the
   team_id of the team that the current thread is a part of).
2) When a thread arrives at the barrier it first waits on any children
   it might have to arrive, then it indicates its arrival to its parent
   by incrementing its "arrived flag".
   They then wait on the global generation for the indication that all
   threads have arrived.
3) The primary thread has ID 0 and all other threads are either its
   direct children or children of its children.
4) When all its children have indicated their arrival to the primary
   thread, the primary thread then increments the global generation
   number to indicate to all secondary threads that the barrier has
   completed.

Fundamental difference that affects performance being that we have
thread-local "arrived" flags instead of a single counter.  That means
there is less contention and appears to speed up the barrier
significantly when threads are hitting the barrier very hard.

Another interesting difference is that the "coordinating" thread is
pre-determined rather than "whichever thread hits the barrier last".
- This means that the behaviour of the pre-determined "coordinating"
  thread must be more complicated while it's waiting on other threads to
  arrive.  See the task handling in
  `gomp_team_barrier_wait_on_two_flags`.
- Because we assign the "coordinating" thread to be the primary thread
  this does mean that in-between points (3) and (4) above the primary
  thread can see the results of the operations of all secondary threads.
  This is important for the second optimisation in this patch where we
  only go through one barrier per iteration in the main execution loop
  of `gomp_thread_start` for top-level threads.

This second optimisation is as follows:
- In the current libgomp implementation each non-nested thread in the
  thread pool works in a loop along the lines of:
  1:
     perform work
     wait for all threads to have finished  (gather all threads)
     free some data structures
     wait for "go" signal                   (wait on release)
     goto 1
- The "wait for all threads to have finished" and "wait for go signal"
  actions are both currently implemented with full barriers.
- The implementation of each barrier has both of those steps -- "wait
  for all threads to have finished" and then all threads except the last
  "wait for go signal".
- LLVM only has one barrier between executing parallel regions in the
  main loop of a cached thread as can be observed by running in the
  debugger or by observing the performance numbers in the
  micro-benchmark I added to PR119588
- We can naturally "pause" the new barrier in a way similar to LLVM so
  that threads only go through a barrier once per parallel region.

Changes made for this optimisation:
1) Introduce the target hook `gomp_barrier_can_hold` to let generic code
   know when the "final" barrier holds all secondary threads until the
   primary thread calls `gomp_team_barrier_done_final`.
   When `gomp_barrier_can_hold` returns false then the semantics of the
   "final" barrier are essentially the same as that of the "team"
   barrier in that when the primary thread is past it all secondary
   threads are known to have been released.
2) When there are threads "held" waiting for new work, these threads are
   now waiting on the barrier of their previous team rather than waiting
   on the barrier in the "thread_pool".
   Hence we introduce a `prev_barrier` field on pool.h that records the
   team barrier of the last team -- this is the barrier that needs to be
   "released".
   - This is recorded on the thread_pool in `gomp_team_end` and used to
     release the secondary threads in `gomp_team_start` (plus
     `gomp_free_thread` and `gomp_pause_host`).
3) Change the "simple" barrier used by the thread_pool to one using the
   "centralized" implementation.  Since we no longer go through the
   "simple" barrier except in the case of a new thread or an exiting
   thread, this barrier is no longer a performance bottleneck.  That
   means we no longer require it to be of the faster implementation
   introduced in the first patch mentioned above.
   - This means adding back a field of `awaited` for this barrier only
     into the `gomp_barrier_t` data structure.
   - This also means that the "simple" barrier no longer needs a thread
     identification number.  So we can remove the argument of
     `thr->ts.team_id` to each of these functions, which is nice because
     logically these functions are not tied to a given team.
4) Using a "centralized" implementation means that the extra allocation
   and complexity around the new barrier does not affect the "reinit"
   functions used in team.c.
   - We do however need to add some extra complexity around calculating
     the number of threads exiting and the number of threads starting
     for the transition from the old team to the new one.
5) The "final" barrier is adjusted so that once all threads have arrived
   and all tasks have been performed the primary thread continues while
   all secondary threads wait.
   - An interface to releasing all secondary threads is introduced and
     that is called on the `prev_barrier` stored on the thread pool as
     mentioned above.
   - This state of "secondaries held but primary to continue" is indicated
     by a new flag in the generation number.

This ends up with a new split on implementation between the four kinds
of barrier interface presented to the generic code:
- "simple" barrier is now a centralized barrier.
- "team"   barrier is a more scalable, tasking barrier.
  This barrier is never called directly after a cancellable barrier.
  Hence it does not have to handle the BAR_CANCELLED flag in the
  generation.
- "team final" barrier is a scalable tasking barrier that can break
  half-way through in order to provide different memory semantics.
- "cancel" barrier is a scalable tasking barrier that can be cancelled.

--------------
There are a few subtleties in this change to be aware of:
1) The memory ordering between secondary and primary threads has some
   subtle correctness reasoning.
   - In the existing behaviour there is an acquire-release ordering
     from all threads to each other after having gone through the
     first barrier (when the primary thread is executing code in between
     parallel regions).
     In the new change there is an acquire-release ordering from all
     secondary threads to the primary thread, but not one in the other
     direction until after the primary thread has released all other
     threads in the team.
   - I believe this is fine -- the only difference in memory ordering
     is visible in threads that are not running any code (secondary
     threads just waiting on the primary one to release them).  As far
     as the primary thread is concerned it knows that any stores added
     in other threads must be visible in its own one.
   - The memory model semantics at barrier regions (and specifically at
     the entry to a parallel region) are discussed in the OpenMP
     standard here:
     https://www.openmp.org/spec-html/5.0/openmpsu96.html#x127-4920002.17.8
     It explicitly calls out that on entry to a parallel region "the
     behavior is as if the release flush performed by the master thread
     on entry to the parallel region synchronizes with the acquire flush
     performed on entry to each implicit task that is assigned to a
     different thread" which is precisely the second "release" half of
     the barrier.  The exit from a parallel region is an implicit
     barrier and so the documentation could be interpreted as requiring
     an implicit release/acquire flush both ways -- but it's hard to
     decide what the semantics of a release flush from the primary
     syncing with an acquire flush in the secondary means in the time
     when the secondary is not performing any code (as soon as the
     secondaries do start to perform code there is indeed a flush from
     the primary to them).
2) In gomp_barrier_handle_tasks the existing race condition between a
   thread leaving a tasking barrier and the primary thread changing the
   `team` underneath it becomes more of a problem.
   - TSAN pointed this out to me (surprisingly seems to work pretty well
     if building libgomp with TSAN).
   - Race condition is:
     - Secondary thread enters `gomp_barrier_handle_tasks` then gets
       interrupted.
     - Some other thread executes the last tasks, clears the flags,
       increments the generation.
     - Primary thread continues and eventually calls `gomp_team_start`
       with a new team.  It stores this new team on `nthr->ts.team`
       while in `gomp_team_start`.
     - Then secondary thread restarts and reads `thr->ts.team`.  This is
       now the *next* team.
   - Before this was mostly benign as the new team is initialised to all
     zeros so no tasks get run.  With this patch we have to account for
     this data race as we have removed the secondary synchronisation
     point of the "simple barrier" in each iteration around the
     gomp_thread_start loop.  This means that other threads could race
     ahead and schedule new tasks before the secondary thread wakes up.
   - In order to account for this I pass the "current" barrier that
     we're waiting on into `gomp_barrier_handle_tasks`.
3) We adjust when we save the "last_team" in `gomp_team_end`.  Had to
   adjust something in this function in order to save the "prev_barrier"
   (i.e. the barrier that threads in the non-nested gomp_thread_start
   main loop are waiting on).
   - Needed to ensure that if `last_team` was used we still had access
     to `prev_barrier`.
   There was one existing case where the team of non-nested threads
   was not getting saved -- when there was a 1 thread artificial team in
   the outermost scope (that is one higher than "not nested").
   - Chose to save teams in this case as well.
4) We still need to have a second barrier in the primary thread to
   synchronise threads that are starting (and hence were not held in the
   previous team barrier) and to synchronise threads that are exiting.
   The synchronisation of threads that are exiting is to ensure all
   threads have exited the "holding" team barrier before that team gets
   freed.  This can happen in the below case:
   1) Secondary thread A is in team barrier.
   2) All tasks are done, primary thread continues.
   3) Primary thread gets a new team ready that does not include A.
      (Previous team still saved in `pool->last_team`).
   4) Primary thread releases all threads from team barrier (but A does
      not move due to scheduling).
   5) Primary thread and all other threads in this new team perform
      their work.
   6) Primary thread ends this new team.  This replaces the previous
      team on `pool->last_team` and calls `free` on it.
   7) Now secondary thread A gets scheduled, and attempts to use the
      previously freed team barrier.

------------------------------
Since we have two implementations of barriers (one tree-based and one
flat) we need some way to choose between the two.  We allow the user to
specify directly with an environment variable of
GOMP_BARRIER_TYPE={flat,tree,auto}, and by default adjust based on how
long each thread will spin on a flag.  These options are mostly set at
initialization time and kept constant throughout execution, however
there is the chance that the amount of spinning changes due to the
number of threads started.  When `gomp_managed_threads` becomes greater
than `gomp_available_cpus` we switch to using
`gomp_throttled_spin_count_var`.  Hence in order to choose between a
tree and flat barrier according to the amount of spinning done on a
control word we need to account for this.
This decision can change at runtime, we require the decision to stay
constant for an entire generation of a barrier (one thread performs a
tree barrier while another performs the flat barrier would not perform
either properly).  The safe place to make this choice is when all
threads have entered the barrier but not exited it -- we then make the
decision for the next generation.  Since we make the decision at this
point, other threads have all synchronised with the one making the
decision and hence we can perform the write non-atomically and on the
thread that is incrementing the main generation.

------------------------------
I've had to adjust the interface to the other barriers.  Most of the
cases are simply to add an extra unused parameter.  There are also
some dummied out functions to add.

Testing done:
- Bootstrap & regtest on aarch64 and x86_64.
  - With & without _LIBGOMP_CHECKING_.
  - Testsuite with & without OMP_WAIT_POLICY=passive for both
    GOMP_BARRIER_TYPE=flat and GOMP_BARRIER_TYPE=tree.
  - With and without configure `--enable-linux-futex=no` for posix
    target.
- Cross compilation & regtest on arm.
- Built gcn, rtems, nvptx targets (just building -- not ran testsuite).
- TSAN done on this as part of all my upstream patches.

Performance testing done:
- SPEC 2026 speed (those benchmarks using OpenMP)
- SPEC OMP 2012
- SPEC HPC 2021 (restricted to one socket)
- EPCC OpenMP Microbenchmarks.
- Microbenchmark mentioned in PR119588.

libgomp/ChangeLog:

	* barrier.c (GOMP_barrier): Pass barrier ID argument.
	(GOMP_barrier_cancel): Likewise.
	* config/gcn/bar.c (gomp_barrier_wait_end): Add unused barrier ID
	argument.
	(gomp_barrier_wait): Take and pass barrier ID argument.
	(gomp_barrier_wait_last): Likewise.
	(gomp_team_barrier_wait_end): Likewise.  Pass barrier and increment
	arguments to gomp_barrier_handle_tasks.
	(gomp_team_barrier_wait): Take and pass barrier ID argument.
	(gomp_team_barrier_wait_final): Likewise.
	(gomp_team_barrier_wait_cancel_end): Likewise.  Pass barrier and
	increment arguments to gomp_barrier_handle_tasks.
	(gomp_team_barrier_wait_cancel): Take and pass barrier ID argument.
	* config/gcn/bar.h (gomp_barrier_init): Add unused extra allocation
	argument.
	(gomp_barrier_extra_alloc): New function.
	(gomp_barrier_wait): Add barrier ID argument.
	(gomp_barrier_wait_last): Likewise.
	(gomp_barrier_wait_end): Likewise.
	(gomp_team_barrier_wait): Likewise.
	(gomp_team_barrier_wait_final): Likewise.
	(gomp_team_barrier_wait_end): Likewise.
	(gomp_team_barrier_wait_cancel): Likewise.
	(gomp_team_barrier_wait_cancel_end): Likewise.
	(gomp_barrier_wait_start): Likewise.
	(gomp_barrier_wait_cancel_start): Likewise.
	(gomp_barrier_wait_final_start): Likewise.
	(gomp_team_barrier_done): Add unused increment argument.
	(gomp_barrier_has_completed): Take barrier and increment arguments.
	(gomp_barrier_can_hold): New function.
	(gomp_team_barrier_done_final): New function.
	* config/gcn/team.c (gomp_thread_start): Pass barrier ID argument to
	gomp_team_barrier_wait_final.
	* config/linux/bar.c (gomp_barrier_wait_end): Add unused barrier ID
	argument.
	(gomp_barrier_wait): Take and pass barrier ID argument.
	(gomp_barrier_wait_last): Likewise.
	(gomp_team_barrier_wait_end): Likewise.  Pass barrier and increment
	arguments to gomp_barrier_handle_tasks.
	(gomp_team_barrier_wait): Take and pass barrier ID argument.
	(gomp_team_barrier_wait_final): Likewise.
	(gomp_team_barrier_wait_cancel_end): Likewise.  Pass barrier and
	increment arguments to gomp_barrier_handle_tasks.
	(gomp_team_barrier_wait_cancel): Take and pass barrier ID argument.
	* config/linux/bar.h (gomp_barrier_init): Add unused extra
	allocation argument.
	(gomp_barrier_wait): Add barrier ID argument.
	(gomp_barrier_wait_last): Likewise.
	(gomp_barrier_wait_end): Likewise.
	(gomp_team_barrier_wait): Likewise.
	(gomp_team_barrier_wait_final): Likewise.
	(gomp_team_barrier_wait_end): Likewise.
	(gomp_team_barrier_wait_cancel): Likewise.
	(gomp_team_barrier_wait_cancel_end): Likewise.
	(gomp_barrier_wait_start): Likewise.
	(gomp_barrier_wait_cancel_start): Likewise.
	(gomp_barrier_wait_final_start): Likewise.
	(gomp_team_barrier_done): Add unused increment argument.
	(gomp_barrier_has_completed): Take barrier and increment arguments.
	(gomp_barrier_can_hold): New function.
	(gomp_team_barrier_done_final): New function.
	(gomp_barrier_extra_alloc): New function.
	* config/linux/futex_waitv/bar.c: New file.
	* config/linux/futex_waitv/bar.h: New file.
	* config/linux/futex_waitv/futex_waitv.h: New file.
	* config/linux/futex_waitv/simple-bar.h: New file.
	* config/linux/wait.h (do_spin): Use spin_count.
	* config/nvptx/bar.c (gomp_barrier_wait_end): Add unused barrier ID
	argument.
	(gomp_barrier_wait): Take and pass barrier ID argument.
	(gomp_barrier_wait_last): Add unused barrier ID argument.
	(gomp_team_barrier_wait_end): Take barrier ID argument.  Pass barrier
	and increment arguments to gomp_barrier_handle_tasks.
	(gomp_team_barrier_wait): Take and pass barrier ID argument.
	(gomp_team_barrier_wait_final): Likewise.
	(gomp_team_barrier_wait_cancel_end): Take barrier ID argument.  Pass
	barrier and increment arguments to gomp_barrier_handle_tasks.
	(gomp_team_barrier_wait_cancel): Take and pass barrier ID argument.
	* config/nvptx/bar.h (gomp_barrier_init): Add unused extra
	allocation argument.
	(gomp_barrier_extra_alloc): New function.
	(gomp_barrier_wait): Add barrier ID argument.
	(gomp_barrier_wait_last): Likewise.
	(gomp_barrier_wait_end): Likewise.
	(gomp_team_barrier_wait): Likewise.
	(gomp_team_barrier_wait_final): Likewise.
	(gomp_team_barrier_wait_end): Likewise.
	(gomp_team_barrier_wait_cancel): Likewise.
	(gomp_team_barrier_wait_cancel_end): Likewise.
	(gomp_barrier_wait_start): Likewise.
	(gomp_barrier_wait_cancel_start): Likewise.
	(gomp_barrier_wait_final_start): Likewise.
	(gomp_team_barrier_done): Add unused increment argument.
	(gomp_barrier_has_completed): Take barrier and increment arguments.
	(gomp_barrier_can_hold): New function.
	(gomp_team_barrier_done_final): New function.
	* config/nvptx/team.c (gomp_thread_start): Pass barrier ID argument
	to gomp_team_barrier_wait_final.
	* config/posix/bar.c (gomp_barrier_init): Add unused extra
	allocation argument.
	(gomp_barrier_wait): Take and pass barrier ID argument.
	(gomp_team_barrier_wait_end): Likewise.  Pass barrier and increment
	arguments to gomp_barrier_handle_tasks.
	(gomp_team_barrier_wait_cancel_end): Likewise.
	(gomp_team_barrier_wait): Take and pass barrier ID argument.
	(gomp_team_barrier_wait_cancel): Likewise.
	* config/posix/bar.h (gomp_barrier_init): Add extra allocation
	argument.
	(gomp_barrier_extra_alloc): New function.
	(gomp_barrier_wait): Add barrier ID argument.
	(gomp_team_barrier_wait): Likewise.
	(gomp_team_barrier_wait_end): Likewise.
	(gomp_team_barrier_wait_cancel): Likewise.
	(gomp_team_barrier_wait_cancel_end): Likewise.
	(gomp_barrier_wait_start): Likewise.
	(gomp_barrier_wait_cancel_start): Likewise.
	(gomp_team_barrier_wait_final): Likewise.
	(gomp_barrier_wait_last): Likewise.
	(gomp_team_barrier_done): Add unused increment argument.
	(gomp_barrier_has_completed): Take barrier and increment arguments.
	(gomp_barrier_can_hold): New function.
	(gomp_team_barrier_done_final): New function.
	* config/posix/pool.h (gomp_get_thread_pool): Initialise
	prev_barrier.
	* config/posix/simple-bar.h (gomp_simple_barrier_init): Pass extra
	allocation argument.
	(gomp_simple_barrier_reinit): Rename local argument and pass through.
	(gomp_simple_barrier_wait): Pass dummy barrier ID argument.
	(gomp_simple_barrier_wait_last): Likewise.
	* config/rtems/bar.h (gomp_barrier_init): Add unused extra
	allocation argument.
	(gomp_barrier_extra_alloc): New function.
	(gomp_barrier_wait): Add barrier ID argument.
	(gomp_barrier_wait_last): Likewise.
	(gomp_barrier_wait_end): Likewise.
	(gomp_team_barrier_wait): Likewise.
	(gomp_team_barrier_wait_final): Likewise.
	(gomp_team_barrier_wait_end): Likewise.
	(gomp_team_barrier_wait_cancel): Likewise.
	(gomp_team_barrier_wait_cancel_end): Likewise.
	(gomp_barrier_wait_start): Likewise.
	(gomp_barrier_wait_cancel_start): Likewise.
	(gomp_barrier_wait_final_start): Likewise.
	(gomp_team_barrier_done): Add unused increment argument.
	(gomp_barrier_has_completed): Take barrier and increment arguments.
	(gomp_barrier_can_hold): New function.
	(gomp_team_barrier_done_final): New function.
	* configure: Regenerate.
	* configure.ac: Force SYS_futex_waitv definition.  Add
	--enable-linux-futex-waitv handling and futex_waitv link check.  Force
	enable_linux_futex_waitv.
	* configure.tgt: Add linux_path and use linux/futex_waitv in config_path
	when futex_waitv is enabled.
	* env.c (gomp_barrier_type): New variable.
	(parse_barrier_choice): New function.
	(omp_display_env): Display GOMP_BARRIER_TYPE.
	(initialize_env): Parse GOMP_BARRIER_TYPE.
	* libgomp.h: Include limits.h.
	(gomp_assert): New macro.
	(struct gomp_barrier_extra_alloc_needs): New structure.
	(enum gomp_barrier_t): New enum.
	(gomp_barrier_type): New declaration.
	(spin_count): New function.
	(struct gomp_thread_pool): Add prev_barrier member.
	(gomp_barrier_handle_tasks): Add barrier and increment arguments.
	* single.c (GOMP_single_copy_start): Pass barrier ID argument.
	(GOMP_single_copy_end): Likewise.
	* task.c (gomp_barrier_handle_tasks): Take barrier and increment
	arguments.  Load thr->ts.team atomically and return early when it no
	longer matches the barrier being handled.  Pass increment argument to
	barrier completion helpers.
	(GOMP_taskgroup_end): Pass barrier ID argument.
	(GOMP_workshare_task_reduction_unregister): Likewise.
	* team.c (gomp_release_held_threads): New function.
	(gomp_thread_start): Pass barrier ID arguments.  Avoid the second pool
	barrier when the team barrier can hold secondary threads.
	(get_last_team): Add assertions for cached team barrier state.
	(gomp_new_team): Allocate barrier extra storage as part of the team
	allocation and pass it to gomp_barrier_init.
	(gomp_free_thread): Release held threads and adjust pool barrier use.
	(gomp_barrier_calc_wait): New function.
	(gomp_team_start): Track new and exiting threads, use pool barrier to
        handle these if there are any, avoid pool barrier if none and
        `gomp_barrier_can_hold` is true.
	(gomp_team_end): Pass barrier ID arguments, release nested-team
	threads explicitly, cache held top-level team barriers, and add
	assertions before freeing or caching teams.
	(gomp_pause_host): Release held threads and adjust pool barrier use.
	* testsuite/lib/libgomp.exp
	(check_effective_target_linux_futex_waitv): New effective target.
	* testsuite/libgomp-site-extra.exp.in (libgomp_config_path): New
	variable.
	* testsuite/libgomp.c++/task-reduction-20.C: New test.
	* testsuite/libgomp.c++/task-reduction-21.C: New test.
	* testsuite/libgomp.c-c++-common/gomp-barrier-type-env-auto.c: New
	test.
	* testsuite/libgomp.c-c++-common/gomp-barrier-type-env-flat.c: New
	test.
	* testsuite/libgomp.c-c++-common/gomp-barrier-type-env-unset.c: New
	test.
	* testsuite/libgomp.c-c++-common/gomp-barrier-type-env.c: New test.
	* testsuite/libgomp.c/barrier_generation_overflow.c: New test.
	* testsuite/libgomp.c/barrier-switch-1.c: New test.
	* testsuite/libgomp.c/cancel_in_implicit_barrier.c: New test.
	* testsuite/libgomp.c/cancel_in_implicit_barrier_while_primary_waiting.c:
	New test.
	* testsuite/libgomp.c/cancel_incr_wraparound.c: New test.
	* testsuite/libgomp.c/futex_waitv_overflow.c: New test.
	* testsuite/libgomp.c/primary-thread-tasking.c: New test.
	* work.c (gomp_work_share_end): Pass barrier ID argument.
	(gomp_work_share_end_cancel): Pass barrier ID argument.

Signed-off-by: Matthew Malcomson <[email protected]>
---
 libgomp/barrier.c                             |   4 +-
 libgomp/config/gcn/bar.c                      |  42 +-
 libgomp/config/gcn/bar.h                      |  59 +-
 libgomp/config/gcn/team.c                     |   2 +-
 libgomp/config/linux/bar.c                    |  46 +-
 libgomp/config/linux/bar.h                    |  60 +-
 libgomp/config/linux/futex_waitv/bar.c        | 869 ++++++++++++++++++
 libgomp/config/linux/futex_waitv/bar.h        | 406 ++++++++
 .../config/linux/futex_waitv/futex_waitv.h    |  67 ++
 libgomp/config/linux/futex_waitv/simple-bar.h |  66 ++
 libgomp/config/linux/wait.h                   |  11 +-
 libgomp/config/nvptx/bar.c                    |  36 +-
 libgomp/config/nvptx/bar.h                    |  56 +-
 libgomp/config/nvptx/team.c                   |   2 +-
 libgomp/config/posix/bar.c                    |  33 +-
 libgomp/config/posix/bar.h                    |  53 +-
 libgomp/config/posix/pool.h                   |   1 +
 libgomp/config/posix/simple-bar.h             |  12 +-
 libgomp/config/rtems/bar.h                    |  58 +-
 libgomp/configure                             |  82 +-
 libgomp/configure.ac                          |  39 +
 libgomp/configure.tgt                         |  32 +-
 libgomp/env.c                                 |  36 +
 libgomp/libgomp.h                             |  63 +-
 libgomp/single.c                              |   4 +-
 libgomp/task.c                                |  43 +-
 libgomp/team.c                                | 301 +++++-
 libgomp/testsuite/lib/libgomp.exp             |   8 +-
 libgomp/testsuite/libgomp-site-extra.exp.in   |   1 +
 .../testsuite/libgomp.c++/task-reduction-20.C | 136 +++
 .../testsuite/libgomp.c++/task-reduction-21.C | 140 +++
 .../gomp-barrier-type-env-auto.c              |  11 +
 .../gomp-barrier-type-env-flat.c              |  11 +
 .../gomp-barrier-type-env-unset.c             |  10 +
 .../gomp-barrier-type-env.c                   |  11 +
 .../testsuite/libgomp.c/barrier-switch-1.c    | 135 +++
 .../libgomp.c/barrier_generation_overflow.c   |  88 ++
 .../libgomp.c/cancel_in_implicit_barrier.c    | 115 +++
 ...n_implicit_barrier_while_primary_waiting.c |  70 ++
 .../libgomp.c/cancel_incr_wraparound.c        |  94 ++
 .../libgomp.c/futex_waitv_overflow.c          |  36 +
 .../libgomp.c/primary-thread-tasking.c        |  80 ++
 libgomp/work.c                                |   9 +-
 43 files changed, 3201 insertions(+), 237 deletions(-)
 create mode 100644 libgomp/config/linux/futex_waitv/bar.c
 create mode 100644 libgomp/config/linux/futex_waitv/bar.h
 create mode 100644 libgomp/config/linux/futex_waitv/futex_waitv.h
 create mode 100644 libgomp/config/linux/futex_waitv/simple-bar.h
 create mode 100644 libgomp/testsuite/libgomp.c++/task-reduction-20.C
 create mode 100644 libgomp/testsuite/libgomp.c++/task-reduction-21.C
 create mode 100644 libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env-auto.c
 create mode 100644 libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env-flat.c
 create mode 100644 libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env-unset.c
 create mode 100644 libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env.c
 create mode 100644 libgomp/testsuite/libgomp.c/barrier-switch-1.c
 create mode 100644 libgomp/testsuite/libgomp.c/barrier_generation_overflow.c
 create mode 100644 libgomp/testsuite/libgomp.c/cancel_in_implicit_barrier.c
 create mode 100644 libgomp/testsuite/libgomp.c/cancel_in_implicit_barrier_while_primary_waiting.c
 create mode 100644 libgomp/testsuite/libgomp.c/cancel_incr_wraparound.c
 create mode 100644 libgomp/testsuite/libgomp.c/futex_waitv_overflow.c
 create mode 100644 libgomp/testsuite/libgomp.c/primary-thread-tasking.c

diff --git a/libgomp/barrier.c b/libgomp/barrier.c
index 3a6037e4355..89193022cd2 100644
--- a/libgomp/barrier.c
+++ b/libgomp/barrier.c
@@ -38,7 +38,7 @@ GOMP_barrier (void)
   if (team == NULL)
     return;
 
-  gomp_team_barrier_wait (&team->barrier);
+  gomp_team_barrier_wait (&team->barrier, thr->ts.team_id);
 }
 
 bool
@@ -50,5 +50,5 @@ GOMP_barrier_cancel (void)
   /* The compiler transforms to barrier_cancel when it sees that the
      barrier is within a construct that can cancel.  Thus we should
      never have an orphaned cancellable barrier.  */
-  return gomp_team_barrier_wait_cancel (&team->barrier);
+  return gomp_team_barrier_wait_cancel (&team->barrier, thr->ts.team_id);
 }
diff --git a/libgomp/config/gcn/bar.c b/libgomp/config/gcn/bar.c
index b4ea8e68a92..d3e4464347b 100644
--- a/libgomp/config/gcn/bar.c
+++ b/libgomp/config/gcn/bar.c
@@ -32,9 +32,9 @@
 #include <limits.h>
 #include "libgomp.h"
 
-
 void
-gomp_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
+gomp_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state,
+		       unsigned id __attribute__ ((unused)))
 {
   if (__builtin_expect (state & BAR_WAS_LAST, 0))
     {
@@ -48,9 +48,9 @@ gomp_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
 }
 
 void
-gomp_barrier_wait (gomp_barrier_t *bar)
+gomp_barrier_wait (gomp_barrier_t *bar, unsigned id)
 {
-  gomp_barrier_wait_end (bar, gomp_barrier_wait_start (bar));
+  gomp_barrier_wait_end (bar, gomp_barrier_wait_start (bar, id), id);
 }
 
 /* Like gomp_barrier_wait, except that if the encountering thread
@@ -61,12 +61,12 @@ gomp_barrier_wait (gomp_barrier_t *bar)
    the barrier can be safely destroyed.  */
 
 void
-gomp_barrier_wait_last (gomp_barrier_t *bar)
+gomp_barrier_wait_last (gomp_barrier_t *bar, unsigned id)
 {
   /* Deferring to gomp_barrier_wait does not use the optimization opportunity
      allowed by the interface contract for all-but-last participants.  The
      original implementation in config/linux/bar.c handles this better.  */
-  gomp_barrier_wait (bar);
+  gomp_barrier_wait (bar, id);
 }
 
 void
@@ -77,7 +77,8 @@ gomp_team_barrier_wake (gomp_barrier_t *bar, int count)
 }
 
 void
-gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
+gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state,
+			    unsigned id __attribute__ ((unused)))
 {
   unsigned int gen;
 
@@ -93,7 +94,7 @@ gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
 	= __atomic_load_n (&team->task_count, MEMMODEL_ACQUIRE);
       if (__builtin_expect (task_count, 0))
 	{
-	  gomp_barrier_handle_tasks (state);
+	  gomp_barrier_handle_tasks (state, bar, 0);
 	  state &= ~BAR_WAS_LAST;
 	}
       else
@@ -126,7 +127,7 @@ gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
       gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
       if (__builtin_expect (gen & BAR_TASK_PENDING, 0))
 	{
-	  gomp_barrier_handle_tasks (state);
+	  gomp_barrier_handle_tasks (state, bar, 0);
 	  gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
 	}
     }
@@ -134,23 +135,24 @@ gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
 }
 
 void
-gomp_team_barrier_wait (gomp_barrier_t *bar)
+gomp_team_barrier_wait (gomp_barrier_t *bar, unsigned id)
 {
-  gomp_team_barrier_wait_end (bar, gomp_barrier_wait_start (bar));
+  gomp_team_barrier_wait_end (bar, gomp_barrier_wait_start (bar, id), id);
 }
 
 void
-gomp_team_barrier_wait_final (gomp_barrier_t *bar)
+gomp_team_barrier_wait_final (gomp_barrier_t *bar, unsigned id)
 {
-  gomp_barrier_state_t state = gomp_barrier_wait_final_start (bar);
+  gomp_barrier_state_t state = gomp_barrier_wait_final_start (bar, id);
   if (__builtin_expect (state & BAR_WAS_LAST, 0))
     bar->awaited_final = bar->total;
-  gomp_team_barrier_wait_end (bar, state);
+  gomp_team_barrier_wait_end (bar, state, id);
 }
 
 bool
 gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
-				   gomp_barrier_state_t state)
+				   gomp_barrier_state_t state,
+				   unsigned id __attribute__ ((unused)))
 {
   unsigned int gen;
 
@@ -170,7 +172,7 @@ gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
 	= __atomic_load_n (&team->task_count, MEMMODEL_ACQUIRE);
       if (__builtin_expect (task_count, 0))
 	{
-	  gomp_barrier_handle_tasks (state);
+	  gomp_barrier_handle_tasks (state, bar, 0);
 	  state &= ~BAR_WAS_LAST;
 	}
       else
@@ -207,7 +209,7 @@ gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
 	return true;
       if (__builtin_expect (gen & BAR_TASK_PENDING, 0))
 	{
-	  gomp_barrier_handle_tasks (state);
+	  gomp_barrier_handle_tasks (state, bar, 0);
 	  gen = __atomic_load_n (&bar->generation, MEMMODEL_RELAXED);
 	}
     }
@@ -217,9 +219,11 @@ gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
 }
 
 bool
-gomp_team_barrier_wait_cancel (gomp_barrier_t *bar)
+gomp_team_barrier_wait_cancel (gomp_barrier_t *bar, unsigned id)
 {
-  return gomp_team_barrier_wait_cancel_end (bar, gomp_barrier_wait_start (bar));
+  return gomp_team_barrier_wait_cancel_end (bar,
+					    gomp_barrier_wait_start (bar, id),
+					    id);
 }
 
 void
diff --git a/libgomp/config/gcn/bar.h b/libgomp/config/gcn/bar.h
index 943bafc3551..812e946ce94 100644
--- a/libgomp/config/gcn/bar.h
+++ b/libgomp/config/gcn/bar.h
@@ -53,7 +53,9 @@ typedef unsigned int gomp_barrier_state_t;
 #define BAR_CANCELLED		4
 #define BAR_INCR		8
 
-static inline void gomp_barrier_init (gomp_barrier_t *bar, unsigned count)
+static inline void
+gomp_barrier_init (gomp_barrier_t *bar, unsigned count,
+		   void *ptr __attribute__ ((unused)))
 {
   bar->total = count;
   bar->awaited = count;
@@ -71,22 +73,32 @@ static inline void gomp_barrier_destroy (gomp_barrier_t *bar)
 {
 }
 
-extern void gomp_barrier_wait (gomp_barrier_t *);
-extern void gomp_barrier_wait_last (gomp_barrier_t *);
-extern void gomp_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t);
-extern void gomp_team_barrier_wait (gomp_barrier_t *);
-extern void gomp_team_barrier_wait_final (gomp_barrier_t *);
-extern void gomp_team_barrier_wait_end (gomp_barrier_t *,
-					gomp_barrier_state_t);
-extern bool gomp_team_barrier_wait_cancel (gomp_barrier_t *);
+static inline struct gomp_barrier_extra_alloc_needs
+gomp_barrier_extra_alloc (unsigned count __attribute__ ((unused)))
+{
+  return (struct gomp_barrier_extra_alloc_needs){0, 1};
+}
+
+extern void gomp_barrier_wait (gomp_barrier_t *, unsigned);
+extern void gomp_barrier_wait_last (gomp_barrier_t *, unsigned);
+extern void gomp_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t,
+				   unsigned);
+
+extern void gomp_team_barrier_wait (gomp_barrier_t *, unsigned);
+extern void gomp_team_barrier_wait_final (gomp_barrier_t *, unsigned);
+extern void gomp_team_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t,
+					unsigned);
+
+extern bool gomp_team_barrier_wait_cancel (gomp_barrier_t *, unsigned);
 extern bool gomp_team_barrier_wait_cancel_end (gomp_barrier_t *,
-					       gomp_barrier_state_t);
+					       gomp_barrier_state_t, unsigned);
 extern void gomp_team_barrier_wake (gomp_barrier_t *, int);
 struct gomp_team;
 extern void gomp_team_barrier_cancel (struct gomp_team *);
 
 static inline gomp_barrier_state_t
-gomp_barrier_wait_start (gomp_barrier_t *bar)
+gomp_barrier_wait_start (gomp_barrier_t *bar,
+			 unsigned id __attribute__ ((unused)))
 {
   unsigned int ret = __atomic_load_n (&bar->generation, MEMMODEL_RELAXED);
   ret &= -BAR_INCR | BAR_CANCELLED;
@@ -101,16 +113,17 @@ gomp_barrier_wait_start (gomp_barrier_t *bar)
 }
 
 static inline gomp_barrier_state_t
-gomp_barrier_wait_cancel_start (gomp_barrier_t *bar)
+gomp_barrier_wait_cancel_start (gomp_barrier_t *bar, unsigned id)
 {
-  return gomp_barrier_wait_start (bar);
+  return gomp_barrier_wait_start (bar, id);
 }
 
 /* This is like gomp_barrier_wait_start, except it decrements
    bar->awaited_final rather than bar->awaited and should be used
    for the gomp_team_end barrier only.  */
 static inline gomp_barrier_state_t
-gomp_barrier_wait_final_start (gomp_barrier_t *bar)
+gomp_barrier_wait_final_start (gomp_barrier_t *bar,
+			       unsigned id __attribute__ ((unused)))
 {
   unsigned int ret = __atomic_load_n (&bar->generation, MEMMODEL_RELAXED);
   ret &= -BAR_INCR | BAR_CANCELLED;
@@ -160,7 +173,8 @@ gomp_team_barrier_cancelled (gomp_barrier_t *bar)
 }
 
 static inline void
-gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state)
+gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state,
+			unsigned increment __attribute__ ((unused)))
 {
   /* Need the atomic store for acquire-release synchronisation with the
      load in `gomp_team_barrier_wait_{cancel_,}end`.  See PR112356  */
@@ -169,10 +183,23 @@ gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state)
 }
 
 static inline bool
-gomp_barrier_has_completed (gomp_barrier_state_t state, gomp_barrier_t *bar)
+gomp_barrier_has_completed (gomp_barrier_state_t state, gomp_barrier_t *bar,
+			    unsigned increment __attribute__ ((unused)))
 {
   unsigned int gen = bar->generation;
   return (gen & -BAR_INCR) == (state & -BAR_INCR) + BAR_INCR;
 }
 
+static inline bool
+gomp_barrier_can_hold (gomp_barrier_t *bar)
+{
+  return false;
+}
+
+/* Functions dummied out for this implementation.  */
+static inline void
+gomp_team_barrier_done_final (gomp_barrier_t *bar,
+			      unsigned id __attribute__ ((unused)))
+{}
+
 #endif /* GOMP_BARRIER_H */
diff --git a/libgomp/config/gcn/team.c b/libgomp/config/gcn/team.c
index c9c2f3c2419..97fa56716cb 100644
--- a/libgomp/config/gcn/team.c
+++ b/libgomp/config/gcn/team.c
@@ -166,7 +166,7 @@ gomp_thread_start (struct gomp_thread_pool *pool)
       thr->fn = NULL;
 
       struct gomp_task *task = thr->task;
-      gomp_team_barrier_wait_final (&thr->ts.team->barrier);
+      gomp_team_barrier_wait_final (&thr->ts.team->barrier, thr->ts.team_id);
       gomp_finish_task (task);
     }
   while (1);
diff --git a/libgomp/config/linux/bar.c b/libgomp/config/linux/bar.c
index 330b37a9692..1ec1baa4bc1 100644
--- a/libgomp/config/linux/bar.c
+++ b/libgomp/config/linux/bar.c
@@ -24,15 +24,15 @@
    <http://www.gnu.org/licenses/>.  */
 
 /* This is a Linux specific implementation of a barrier synchronization
-   mechanism for libgomp.  This type is private to the library.  This 
+   mechanism for libgomp.  This type is private to the library.  This
    implementation uses atomic instructions and the futex syscall.  */
 
 #include <limits.h>
 #include "wait.h"
 
-
 void
-gomp_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
+gomp_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state,
+		       unsigned id __attribute__ ((unused)))
 {
   if (__builtin_expect (state & BAR_WAS_LAST, 0))
     {
@@ -51,9 +51,9 @@ gomp_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
 }
 
 void
-gomp_barrier_wait (gomp_barrier_t *bar)
+gomp_barrier_wait (gomp_barrier_t *bar, unsigned id)
 {
-  gomp_barrier_wait_end (bar, gomp_barrier_wait_start (bar));
+  gomp_barrier_wait_end (bar, gomp_barrier_wait_start (bar, id), id);
 }
 
 /* Like gomp_barrier_wait, except that if the encountering thread
@@ -64,11 +64,11 @@ gomp_barrier_wait (gomp_barrier_t *bar)
    the barrier can be safely destroyed.  */
 
 void
-gomp_barrier_wait_last (gomp_barrier_t *bar)
+gomp_barrier_wait_last (gomp_barrier_t *bar, unsigned id)
 {
-  gomp_barrier_state_t state = gomp_barrier_wait_start (bar);
+  gomp_barrier_state_t state = gomp_barrier_wait_start (bar, id);
   if (state & BAR_WAS_LAST)
-    gomp_barrier_wait_end (bar, state);
+    gomp_barrier_wait_end (bar, state, id);
 }
 
 void
@@ -78,7 +78,8 @@ gomp_team_barrier_wake (gomp_barrier_t *bar, int count)
 }
 
 void
-gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
+gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state,
+			    unsigned id __attribute__ ((unused)))
 {
   unsigned int generation, gen;
 
@@ -94,7 +95,7 @@ gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
 	= __atomic_load_n (&team->task_count, MEMMODEL_ACQUIRE);
       if (__builtin_expect (task_count, 0))
 	{
-	  gomp_barrier_handle_tasks (state);
+	  gomp_barrier_handle_tasks (state, bar, BAR_INCR);
 	  state &= ~BAR_WAS_LAST;
 	}
       else
@@ -115,7 +116,7 @@ gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
       gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
       if (__builtin_expect (gen & BAR_TASK_PENDING, 0))
 	{
-	  gomp_barrier_handle_tasks (state);
+	  gomp_barrier_handle_tasks (state, bar, BAR_INCR);
 	  gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
 	}
       generation |= gen & BAR_WAITING_FOR_TASK;
@@ -124,23 +125,24 @@ gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
 }
 
 void
-gomp_team_barrier_wait (gomp_barrier_t *bar)
+gomp_team_barrier_wait (gomp_barrier_t *bar, unsigned id)
 {
-  gomp_team_barrier_wait_end (bar, gomp_barrier_wait_start (bar));
+  gomp_team_barrier_wait_end (bar, gomp_barrier_wait_start (bar, id), id);
 }
 
 void
-gomp_team_barrier_wait_final (gomp_barrier_t *bar)
+gomp_team_barrier_wait_final (gomp_barrier_t *bar, unsigned id)
 {
-  gomp_barrier_state_t state = gomp_barrier_wait_final_start (bar);
+  gomp_barrier_state_t state = gomp_barrier_wait_final_start (bar, id);
   if (__builtin_expect (state & BAR_WAS_LAST, 0))
     bar->awaited_final = bar->total;
-  gomp_team_barrier_wait_end (bar, state);
+  gomp_team_barrier_wait_end (bar, state, id);
 }
 
 bool
 gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
-				   gomp_barrier_state_t state)
+				   gomp_barrier_state_t state,
+				   unsigned id __attribute__ ((unused)))
 {
   unsigned int generation, gen;
 
@@ -160,7 +162,7 @@ gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
 	= __atomic_load_n (&team->task_count, MEMMODEL_ACQUIRE);
       if (__builtin_expect (task_count, 0))
 	{
-	  gomp_barrier_handle_tasks (state);
+	  gomp_barrier_handle_tasks (state, bar, BAR_INCR);
 	  state &= ~BAR_WAS_LAST;
 	}
       else
@@ -184,7 +186,7 @@ gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
 	return true;
       if (__builtin_expect (gen & BAR_TASK_PENDING, 0))
 	{
-	  gomp_barrier_handle_tasks (state);
+	  gomp_barrier_handle_tasks (state, bar, BAR_INCR);
 	  gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
 	}
       generation |= gen & BAR_WAITING_FOR_TASK;
@@ -195,9 +197,11 @@ gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
 }
 
 bool
-gomp_team_barrier_wait_cancel (gomp_barrier_t *bar)
+gomp_team_barrier_wait_cancel (gomp_barrier_t *bar, unsigned id)
 {
-  return gomp_team_barrier_wait_cancel_end (bar, gomp_barrier_wait_start (bar));
+  return gomp_team_barrier_wait_cancel_end (bar,
+					    gomp_barrier_wait_start (bar, id),
+					    id);
 }
 
 void
diff --git a/libgomp/config/linux/bar.h b/libgomp/config/linux/bar.h
index b7399f88dc0..71565caa409 100644
--- a/libgomp/config/linux/bar.h
+++ b/libgomp/config/linux/bar.h
@@ -24,7 +24,7 @@
    <http://www.gnu.org/licenses/>.  */
 
 /* This is a Linux specific implementation of a barrier synchronization
-   mechanism for libgomp.  This type is private to the library.  This 
+   mechanism for libgomp.  This type is private to the library.  This
    implementation uses atomic instructions and the futex syscall.  */
 
 #ifndef GOMP_BARRIER_H
@@ -53,7 +53,9 @@ typedef unsigned int gomp_barrier_state_t;
 #define BAR_CANCELLED		4
 #define BAR_INCR		8
 
-static inline void gomp_barrier_init (gomp_barrier_t *bar, unsigned count)
+static inline void
+gomp_barrier_init (gomp_barrier_t *bar, unsigned count,
+		   void *ptr __attribute__ ((unused)))
 {
   bar->total = count;
   bar->awaited = count;
@@ -71,22 +73,25 @@ static inline void gomp_barrier_destroy (gomp_barrier_t *bar)
 {
 }
 
-extern void gomp_barrier_wait (gomp_barrier_t *);
-extern void gomp_barrier_wait_last (gomp_barrier_t *);
-extern void gomp_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t);
-extern void gomp_team_barrier_wait (gomp_barrier_t *);
-extern void gomp_team_barrier_wait_final (gomp_barrier_t *);
-extern void gomp_team_barrier_wait_end (gomp_barrier_t *,
-					gomp_barrier_state_t);
-extern bool gomp_team_barrier_wait_cancel (gomp_barrier_t *);
+extern void gomp_barrier_wait (gomp_barrier_t *, unsigned);
+extern void gomp_barrier_wait_last (gomp_barrier_t *, unsigned);
+extern void gomp_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t,
+				   unsigned);
+
+extern void gomp_team_barrier_wait (gomp_barrier_t *, unsigned);
+extern void gomp_team_barrier_wait_final (gomp_barrier_t *, unsigned);
+extern void gomp_team_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t,
+					unsigned);
+extern bool gomp_team_barrier_wait_cancel (gomp_barrier_t *, unsigned);
 extern bool gomp_team_barrier_wait_cancel_end (gomp_barrier_t *,
-					       gomp_barrier_state_t);
+					       gomp_barrier_state_t, unsigned);
 extern void gomp_team_barrier_wake (gomp_barrier_t *, int);
 struct gomp_team;
 extern void gomp_team_barrier_cancel (struct gomp_team *);
 
 static inline gomp_barrier_state_t
-gomp_barrier_wait_start (gomp_barrier_t *bar)
+gomp_barrier_wait_start (gomp_barrier_t *bar,
+			 unsigned id __attribute__ ((unused)))
 {
   unsigned int ret = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
   ret &= -BAR_INCR | BAR_CANCELLED;
@@ -101,16 +106,17 @@ gomp_barrier_wait_start (gomp_barrier_t *bar)
 }
 
 static inline gomp_barrier_state_t
-gomp_barrier_wait_cancel_start (gomp_barrier_t *bar)
+gomp_barrier_wait_cancel_start (gomp_barrier_t *bar, unsigned id)
 {
-  return gomp_barrier_wait_start (bar);
+  return gomp_barrier_wait_start (bar, id);
 }
 
 /* This is like gomp_barrier_wait_start, except it decrements
    bar->awaited_final rather than bar->awaited and should be used
    for the gomp_team_end barrier only.  */
 static inline gomp_barrier_state_t
-gomp_barrier_wait_final_start (gomp_barrier_t *bar)
+gomp_barrier_wait_final_start (gomp_barrier_t *bar,
+			       unsigned id __attribute__ ((unused)))
 {
   unsigned int ret = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
   ret &= -BAR_INCR | BAR_CANCELLED;
@@ -160,7 +166,8 @@ gomp_team_barrier_cancelled (gomp_barrier_t *bar)
 }
 
 static inline void
-gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state)
+gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state,
+			unsigned increment __attribute__ ((unused)))
 {
   /* Need the atomic store for acquire-release synchronisation with the
      load in `gomp_team_barrier_wait_{cancel_,}end`.  See PR112356  */
@@ -169,10 +176,29 @@ gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state)
 }
 
 static inline bool
-gomp_barrier_has_completed (gomp_barrier_state_t state, gomp_barrier_t *bar)
+gomp_barrier_has_completed (gomp_barrier_state_t state, gomp_barrier_t *bar,
+			    unsigned increment __attribute__ ((unused)))
 {
   unsigned int gen = bar->generation;
   return (gen & -BAR_INCR) == (state & -BAR_INCR) + BAR_INCR;
 }
 
+static inline bool
+gomp_barrier_can_hold (gomp_barrier_t *bar)
+{
+  return false;
+}
+
+/* Functions dummied out for this implementation.  */
+static inline void
+gomp_team_barrier_done_final (gomp_barrier_t *bar,
+			      unsigned id __attribute__ ((unused)))
+{}
+
+static inline struct gomp_barrier_extra_alloc_needs
+gomp_barrier_extra_alloc (unsigned count __attribute__ ((unused)))
+{
+  return (struct gomp_barrier_extra_alloc_needs){0, 1};
+}
+
 #endif /* GOMP_BARRIER_H */
diff --git a/libgomp/config/linux/futex_waitv/bar.c b/libgomp/config/linux/futex_waitv/bar.c
new file mode 100644
index 00000000000..1ae3f9df9e7
--- /dev/null
+++ b/libgomp/config/linux/futex_waitv/bar.c
@@ -0,0 +1,869 @@
+/* Copyright The GNU Toolchain Authors.
+
+   This file is part of the GNU Offloading and Multi Processing Library
+   (libgomp).
+
+   Libgomp is free software; you can redistribute it and/or modify it
+   under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3, or (at your option)
+   any later version.
+
+   Libgomp is distributed in the hope that it will be useful, but WITHOUT ANY
+   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+   FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+   more details.
+
+   Under Section 7 of GPL version 3, you are granted additional
+   permissions described in the GCC Runtime Library Exception, version
+   3.1, as published by the Free Software Foundation.
+
+   You should have received a copy of the GNU General Public License and
+   a copy of the GCC Runtime Library Exception along with this program;
+   see the files COPYING3 and COPYING.RUNTIME respectively.  If not, see
+   <http://www.gnu.org/licenses/>.  */
+
+/* This is an implementation of a barrier synchronization mechanism for libgomp
+   written for Linux versions > 5.16 (that have the `futex_waitv` syscall).
+   This type is private to the library.  This implementation uses atomic
+   instructions and the futex and futex_waitv syscalls.  */
+
+#include <limits.h>
+#include "wait.h"
+#include "futex_waitv.h"
+
+#ifndef GOMP_USE_ALIGNED_WORK_SHARES
+#error "Assume GOMP_USE_ALIGNED_WORK_SHARES when allocating barrier in team.c"
+#endif
+
+static inline gomp_barrier_state_t
+gomp_centralized_barrier_wait_start (gomp_barrier_t *bar)
+{
+  unsigned ret = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
+  ret &= -BAR_INCR | BAR_CANCELLED;
+  /* A memory barrier is needed before exiting from the various forms
+     of gomp_barrier_wait, to satisfy OpenMP API version 3.1 section
+     2.8.6 flush Construct, which says there is an implicit flush during
+     a barrier region.  This is a convenient place to add the barrier,
+     so we use MEMMODEL_ACQ_REL here rather than MEMMODEL_ACQUIRE.  */
+  if (__atomic_add_fetch (&bar->awaited, -1, MEMMODEL_ACQ_REL) == 0)
+    ret |= BAR_WAS_LAST;
+  return ret;
+}
+
+static inline void
+gomp_centralized_barrier_wait_end (gomp_barrier_t *bar,
+				   gomp_barrier_state_t state)
+{
+  gomp_assert (!(state & BAR_CANCELLED),
+	       "BAR_CANCELLED set when using plain barrier: %u", state);
+  if (__builtin_expect (state & BAR_WAS_LAST, 0))
+    {
+      /* Next time we'll be awaiting TOTAL threads again.  */
+      bar->awaited = bar->total;
+      /* No need to atomically load bar->generation here since we know we are
+	 last and since there's no tasking no-one else is writing to this
+	 location.  */
+      __atomic_store_n (&bar->generation, bar->generation + BAR_INCR,
+			MEMMODEL_RELEASE);
+      futex_wake ((int *) &bar->generation, INT_MAX);
+    }
+  else
+    {
+      do
+	do_wait ((int *) &bar->generation, state);
+      while (__atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE) == state);
+    }
+}
+
+void
+gomp_centralized_barrier_wait (gomp_barrier_t *bar)
+{
+  gomp_barrier_state_t state = gomp_centralized_barrier_wait_start (bar);
+  gomp_centralized_barrier_wait_end (bar, state);
+}
+
+/* Like gomp_centralized_barrier_wait, except that if the encountering thread
+   is not the last one to hit the barrier, it returns immediately.  The
+   intended usage is that a thread which intends to gomp_barrier_destroy this
+   barrier calls gomp_centralized_barrier_wait, while all other threads call
+   gomp_centralized_barrier_wait_last.  When gomp_centralized_barrier_wait
+   returns, the barrier can be safely destroyed.  */
+
+void
+gomp_centralized_barrier_wait_last (gomp_barrier_t *bar)
+{
+  gomp_barrier_state_t state = gomp_centralized_barrier_wait_start (bar);
+  if (state & BAR_WAS_LAST)
+    gomp_centralized_barrier_wait_end (bar, state);
+}
+
+static inline void
+gomp_assert_seenflags (gomp_barrier_t *bar, bool cancellable)
+{
+#if _LIBGOMP_CHECKING_
+  unsigned gen = __atomic_load_n (&bar->generation, MEMMODEL_RELAXED);
+  struct thread_lock_data *arr = bar->threadgens;
+  unsigned cancel_incr = cancellable ? BAR_CANCEL_INCR : 0;
+  unsigned incr = cancellable ? 0 : BAR_INCR;
+  /* Assert that all threads have been seen.  */
+  for (unsigned i = 0; i < bar->total; i++)
+    {
+      unsigned g = arr[i].gen;
+      gomp_assert ((g & BAR_GEN_MASK) == (gen & BAR_GEN_MASK) + incr,
+		   "Index %u generation is %u (global is %u)\n", i, g, gen);
+      gomp_assert ((g & BAR_CANCEL_GEN_MASK)
+		     == ((gen + cancel_incr) & BAR_CANCEL_GEN_MASK),
+		   "Index %u cancel generation out of sync.  Gen is %u (global "
+		   "is %u)\n",
+		   i, g, gen);
+    }
+
+  /* Assert that generation numbers not corresponding to any thread are
+     cleared.  This helps us test code-paths.  */
+  for (unsigned i = bar->total; i < bar->allocated; i++)
+    {
+      unsigned g = arr[i].gen;
+      gomp_assert (g == 0,
+		   "Index %u gen should be 0.  Is %u (global gen is %u)\n", i,
+		   g, gen);
+    }
+#endif
+}
+
+static inline void
+gomp_assert_and_store_flag (unsigned *addr, unsigned val, unsigned orig,
+			    unsigned gens, unsigned id)
+{
+  __atomic_store_n (addr, val, MEMMODEL_RELEASE);
+  futex_wake ((int *) addr, INT_MAX);
+  gomp_assert (orig == (gens & BAR_BOTH_GENS_MASK),
+	       "Id %u:  Original flag %u != generation of %u\n", id, orig,
+	       gens);
+  gomp_assert ((orig & BAR_FLAGS_MASK) == 0,
+	       "Id %u:  Original flag %u has flags set\n", id, orig);
+}
+
+static inline void
+gomp_assert_and_increment_flag (gomp_barrier_t *bar, unsigned id, unsigned gens)
+{
+  /* Atomically update our thread-local generation for the primary thread to
+     observe.  We are the only thread writing to this value, so we only need an
+     atomic store for publishing and do not need __atomic_fetch_add.  */
+  unsigned orig = bar->threadgens[id].gen;
+  unsigned new = orig + BAR_INCR;
+  gomp_assert_and_store_flag (&bar->threadgens[id].gen, new, orig, gens, id);
+}
+
+static inline gomp_barrier_state_t
+gomp_flat_barrier_arrive (gomp_barrier_t *bar, unsigned id)
+{
+  /* TODO I don't believe this MEMMODEL_ACQUIRE is needed.
+     Look into it later.  Point being that this should only ever read a value
+     from last barrier or from tasks/cancellation/etc.  There was already an
+     acquire-release ordering at exit of the last barrier, all setting of
+     tasks/cancellation etc are done with RELAXED memory model => using ACQUIRE
+     doesn't help.
+
+     See corresponding comment in `gomp_team_barrier_cancel` when thinking
+     about this.  */
+  unsigned ret = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
+  ret &= (BAR_BOTH_GENS_MASK | BAR_CANCELLED);
+#if !_LIBGOMP_CHECKING_
+  if (id != 0)
+#endif
+    /* Increment local flag.  For thread id 0 this doesn't communicate
+       anything to *other* threads, but it is useful for debugging purposes.  */
+    gomp_assert_and_increment_flag (bar, id, ret);
+
+  if (id == 0)
+    ret |= BAR_WAS_LAST;
+  return ret;
+}
+
+static void
+gomp_wait_on_flag (gomp_barrier_t *bar, unsigned child_tid, unsigned tstate)
+{
+  unsigned tmp;
+  struct thread_lock_data *arr = bar->threadgens;
+  do
+    {
+      do_wait ((int *) &arr[child_tid].gen, tstate);
+      tmp = __atomic_load_n (&arr[child_tid].gen, MEMMODEL_ACQUIRE);
+      gomp_assert (tmp == tstate || tmp == (tstate + BAR_INCR),
+		   "Thread-local state seen to be %u"
+		   " when global gens are %u",
+		   tmp, tstate);
+    }
+  while (tmp != (tstate + BAR_INCR));
+}
+
+gomp_barrier_state_t
+gomp_flat_barrier_wait_start (gomp_barrier_t *bar, unsigned id)
+{
+  gomp_barrier_state_t state = gomp_flat_barrier_arrive (bar, id);
+  gomp_assert (!(state & BAR_CANCELLED),
+	       "BAR_CANCELLED set when using plain barrier: %u", state);
+  if (state & BAR_WAS_LAST)
+    {
+      gomp_assert (id == 0, "Secondary thread %u believes is last", id);
+      unsigned tstate = state & BAR_BOTH_GENS_MASK;
+      for (unsigned i = 1; i < bar->total; i++)
+	gomp_wait_on_flag (bar, i, tstate);
+      gomp_assert_seenflags (bar, false);
+    }
+  return state;
+}
+
+#define hyper_iteration(ID, BAR, FINISHED, WAIT_ON_CHILD)                      \
+  do                                                                           \
+    {                                                                          \
+      uint32_t num_threads = (BAR)->total;                                     \
+      uint32_t offset;                                                         \
+      uint32_t level;                                                          \
+      uint32_t branch_bits = 2;                                                \
+      uint32_t branch_factor = 1 << branch_bits;                               \
+      for (level = 0, offset = 1; offset < num_threads;                        \
+	   level += branch_bits, offset <<= branch_bits)                       \
+	{                                                                      \
+	  uint32_t child;                                                      \
+	  uint32_t child_tid;                                                  \
+	  if ((((ID) >> level) & (branch_factor - 1)) != 0)                    \
+	    {                                                                  \
+	      gomp_assert ((ID) != 0,                                          \
+			   "Primary thread incrementing thread local gen\n");  \
+	      FINISHED;                                                        \
+	      break;                                                           \
+	    }                                                                  \
+	  for (child = 1, child_tid = (ID) + (1 << level);                     \
+	       child < branch_factor && child_tid < num_threads;               \
+	       child++, child_tid += (1 << level))                             \
+	    WAIT_ON_CHILD (child_tid);                                         \
+	}                                                                      \
+    }                                                                          \
+  while (0)
+
+gomp_barrier_state_t
+gomp_tree_barrier_wait_start (gomp_barrier_t *bar, unsigned id)
+{
+  unsigned gstate = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
+  gomp_assert (!(gstate & BAR_CANCELLED),
+	       "BAR_CANCELLED set when using plain barrier: %u", gstate);
+  gstate &= BAR_BOTH_GENS_MASK;
+  unsigned tstate = gstate & BAR_BOTH_GENS_MASK;
+#define wait_on_child(child_tid) gomp_wait_on_flag (bar, child_tid, tstate)
+  hyper_iteration (id, bar, gomp_assert_and_increment_flag (bar, id, gstate),
+		   wait_on_child);
+#undef wait_on_child
+  /* At this point children threads have marked their arrival to their parent
+     thread but don't know that their parent thread has seen that mark.
+     Parent threads know that all their children have arrived here (and that
+     we've seen it).  */
+  if (id == 0)
+  {
+#if _LIBGOMP_CHECKING_
+    gomp_assert_and_increment_flag (bar, id, gstate);
+#endif
+    gstate |= BAR_WAS_LAST;
+    gomp_assert_seenflags (bar, false);
+  }
+  return gstate;
+}
+
+void
+gomp_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state,
+		       unsigned id)
+{
+  gomp_assert (!(state & BAR_CANCELLED),
+	       "BAR_CANCELLED set when using plain barrier: %u", state);
+  if (__builtin_expect (state & BAR_WAS_LAST, 0))
+    {
+      gomp_assert (id == 0, "Id %u believes it is last\n", id);
+      /* Shouldn't have anything modifying bar->generation at this point since
+	 we have waited for all other threads to enter this function and
+	 they're all in the `do_wait` loop below.  */
+      gomp_assert ((bar->generation & ~BAR_BOTH_GENS_MASK) == 0,
+		   "flags set in gomp_barrier_wait_end: %u",
+		   bar->generation & ~BAR_BOTH_GENS_MASK);
+      bar->use_tree = gomp_barrier_choose_tree ();
+      __atomic_store_n (&bar->generation, bar->generation + BAR_INCR,
+			MEMMODEL_RELEASE);
+      futex_wake ((int *) &bar->generation, INT_MAX);
+    }
+  else
+    {
+      do
+	do_wait ((int *) &bar->generation, state);
+      while (__atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE) == state);
+    }
+}
+
+/* We need to have some "ensure we're last, but perform useful work
+   in the meantime" behaviour.  This allows the primary thread to perform
+   useful work while it is waiting on all secondary threads.
+   - Related to the fundamental difference between this barrier and the
+     "centralized" barrier where the thread doing the bookkeeping is
+     pre-determined as some "primary" thread and that is not necessarily the
+     last thread to enter the barrier.
+
+   To do that we loop through checking each of the other threads flags.  If any
+   are not set then we take that opportunity to check the global generation
+   number, if there's some task to handle then do so before going back to
+   checking the remaining thread-local flags.
+
+   This approach means that the cancellable barriers work reasonably naturally.
+   If we're checking the global generation flag then we can see when it is
+   cancelled.  Hence `gomp_team_barrier_cancel_wait_on_two_flags` below does
+   something very similar to this function here.
+
+   When there is nothing for a thread to do we want to wait on one of two
+   addresses being changed (either the secondary threads control flag or the
+   barrier main generation).  The futex_waitv syscall provides exactly that
+   functionality.  */
+static inline void
+gomp_team_barrier_wait_on_two_flags (gomp_barrier_t *bar, unsigned child_tid,
+				     unsigned long long spincount,
+				     unsigned gstate, unsigned tstate)
+{
+  /* Thought process here:
+     - Until we are the last thread, we are "some thread waiting on the
+       barrier".  Hence we should doing only a variant of what is done in
+       `gomp_team_barrier_wait_end` for the other threads.
+     - The loop done in that function is:
+       1) Wait on `generation` having some flag.
+       2) When it changes, enforce the acquire-release semantics and
+	  re-load.
+       3) If added `BAR_TASK_PENDING`, then handle some tasks.
+       4) Ignore `BAR_WAITING_FOR_TASK` flag.
+     - Hence the loop done in this function is:
+       1) Wait on `child_tid` generation number.
+	  - When this is not incremented, then:
+	    1. wait on `generation` having some flag.
+	    2. When it does have such a flag, enforce the acquire-release
+	       semantics and load.
+	    3. If added `BAR_TASK_PENDING` then handle some tasks.
+	    4. Ignore `BAR_WAITING_FOR_TASK` not necessary because it's us that
+	       would set that flag.
+       Differences otherwise are that we put pausing in different positions.  */
+  unsigned long long j;
+  unsigned threadgen_orig;
+  struct thread_lock_data *arr = bar->threadgens;
+restart_loop:
+  /* Use `j <= spincount` here in case `gomp_spin_count_var == 0` (which can
+     happen with OMP_WAIT_POLICY=passive).  We need to go around this loop at
+     least once to check and handle any changes.  */
+  for (j = 0; j <= spincount; j++)
+    {
+      /* Thought about using MEMMODEL_ACQUIRE or MEMMODEL_RELAXED until we
+	 see a difference and *then* MEMMODEL_ACQUIRE for the
+	 acquire-release semantics.  Idea was that the more relaxed memory
+	 model might provide a performance boost.  Did not see any
+	 improvement on a micro-benchmark and decided not to do that.
+
+	 TODO Question whether to run one loop checking both variables, or
+	      two loops each checking one variable multiple times.  Suspect
+	      that one loop checking both variables is going to be more
+	      responsive and more understandable in terms of performance
+	      characteristics when specifying GOMP_SPINCOUNT.
+	      However there's the chance that checking each variable
+	      something like 10000 times and going around this loop
+	      count/10000 times could give better throughput?  */
+      threadgen_orig = __atomic_load_n (&arr[child_tid].gen, MEMMODEL_ACQUIRE);
+      unsigned threadgen = threadgen_orig & BAR_GEN_MASK;
+      gomp_assert ((threadgen_orig & BAR_FLAGS_MASK) == 0,
+		   "Id %u: thread local generation has non-zero flags: "
+		   "%u\n",
+		   child_tid, threadgen_orig);
+      if (__builtin_expect (threadgen != tstate, 0))
+	{
+	  gomp_assert (threadgen == (tstate + BAR_INCR),
+		       "Thread-local state seen to be %u"
+		       " when global state was %u.\n",
+		       threadgen, tstate);
+	  /* This is the only place we return from this function.  */
+	  return;
+	}
+      unsigned gen = __atomic_load_n (&bar->generation, MEMMODEL_RELAXED);
+      if (__builtin_expect (gen != gstate, 0))
+	{
+	  if (gen & BAR_TASK_PENDING)
+	    gomp_barrier_handle_tasks (gstate, bar, false);
+	  /* When running as a logical "final" barrier the `CANCELLED` flag
+	     can get set on the barrier if we were in a cancellable region
+	     whose only barrier was the implicit "final" barrier at the end
+	     of the region.  Ensure we update `gstate` in this case to
+	     avoid any unnecessary busy-waiting when nothing is actually
+	     changing.  */
+	  gomp_assert (!(gstate & BAR_CANCELLED) || (gen & BAR_CANCELLED),
+		       "BAR_CANCELLED state cleared by non-primary thread"
+		       " bar->generation: %u   state: %u",
+		       gen, gstate);
+	  gstate |= gen & BAR_CANCELLED;
+
+	  gomp_assert (!(gen & BAR_WAITING_FOR_TASK),
+		       "BAR_WAITING_FOR_TASK set by non-primary thread "
+		       "gen=%d",
+		       gen);
+	  gomp_assert (!gomp_barrier_state_is_incremented (gen, gstate,
+							   BAR_INCR),
+		       "Global state incremented by non-primary thread "
+		       "gen=%d",
+		       gen);
+	  goto restart_loop;
+	}
+    }
+  futex_waitv (&arr[child_tid].gen, threadgen_orig, &bar->generation, gstate);
+  /* One of the above values has changed, restart this loop and we can find out
+     what it was and deal with it accordingly.  */
+  goto restart_loop;
+}
+
+gomp_barrier_state_t
+gomp_team_flat_barrier_wait_start (gomp_barrier_t *bar, unsigned id)
+{
+  gomp_barrier_state_t state = gomp_flat_barrier_arrive (bar, id);
+  if (state & BAR_WAS_LAST)
+    {
+      gomp_assert (id == 0, "Secondary thread %u believes is last", id);
+      unsigned gstate = state & (BAR_BOTH_GENS_MASK | BAR_CANCELLED);
+      unsigned tstate = state & BAR_GEN_MASK;
+      unsigned long long spincount = spin_count ();
+      for (unsigned i = 1; i < bar->total; i++)
+	gomp_team_barrier_wait_on_two_flags (bar, i, spincount, gstate, tstate);
+      gomp_assert_seenflags (bar, false);
+    }
+  return state;
+}
+
+gomp_barrier_state_t
+gomp_team_tree_barrier_wait_start (gomp_barrier_t *bar, unsigned id)
+{
+  unsigned gstate = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
+  gstate &= (BAR_BOTH_GENS_MASK | BAR_CANCELLED);
+  unsigned tstate = gstate & BAR_GEN_MASK;
+  unsigned long long spincount = spin_count ();
+#define wait_on_child(child_tid)                                               \
+  gomp_team_barrier_wait_on_two_flags (bar, child_tid, spincount, gstate,      \
+				       tstate)
+  hyper_iteration (id, bar, gomp_assert_and_increment_flag (bar, id, gstate),
+		   wait_on_child);
+#undef wait_on_child
+if (id == 0)
+  {
+#if _LIBGOMP_CHECKING_
+    gomp_assert_and_increment_flag (bar, id, gstate);
+#endif
+    gstate |= BAR_WAS_LAST;
+    gomp_assert_seenflags (bar, false);
+  }
+
+  return gstate;
+}
+
+static inline bool
+gomp_team_barrier_primary_check_for_tasks (gomp_barrier_t *bar,
+					   unsigned increment,
+					   gomp_barrier_state_t state,
+					   enum memmodel m)
+{
+  struct gomp_thread *thr = gomp_thread ();
+  struct gomp_team *team = thr->ts.team;
+  team->work_share_cancelled = 0;
+  unsigned task_count = __atomic_load_n (&team->task_count, MEMMODEL_ACQUIRE);
+  if (__builtin_expect (task_count, 0))
+    {
+      gomp_barrier_handle_tasks (state, bar, increment);
+      return true;
+    }
+  else
+    {
+      unsigned gens = state & ~BAR_WAS_LAST;
+      gens = gomp_increment_gen (gens, increment);
+      bar->use_tree = gomp_barrier_choose_tree ();
+      __atomic_store_n (&bar->generation, gens, m);
+      futex_wake ((int *) &bar->generation, INT_MAX);
+      return false;
+    }
+}
+
+void
+gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state,
+			    unsigned id)
+{
+  unsigned generation, gen;
+  gomp_assert (!(state & BAR_CANCELLED),
+	       "Generation number includes BAR_CANCELLED in "
+	       "gomp_team_barrier_wait_end: %u , id: %u",
+	       state, id);
+
+  if (__builtin_expect (state & BAR_WAS_LAST, 0))
+    {
+      gomp_assert (id == 0, "Id %u believes it is last\n", id);
+      if (!gomp_team_barrier_primary_check_for_tasks (bar, BAR_INCR, state,
+						      MEMMODEL_RELEASE))
+	return;
+      state &= ~BAR_WAS_LAST;
+    }
+
+  generation = state;
+  do
+    {
+      do_wait ((int *) &bar->generation, generation);
+      gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
+      gomp_assert (!(gen & BAR_CANCELLED),
+		   "BAR_CANCELLED set on generation in team barrier"
+		   " gen: %u  generation: %u  id: %u",
+		   gen, generation, id);
+      if (__builtin_expect (gen & BAR_TASK_PENDING, 0))
+	{
+	  gomp_barrier_handle_tasks (state, bar, BAR_INCR);
+	  gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
+	}
+      /* These flags will not change until this barrier is completed.
+	 Going forward we don't want to be continually waking up checking for
+	 whether this barrier has completed yet.  */
+      generation |= gen & BAR_WAITING_FOR_TASK;
+      /* There should be no other flags set in bar->generation outside of the
+	 ones we explicitly handle here.  */
+    }
+  while ((gen & BAR_BOTH_GENS_MASK) != state + BAR_INCR);
+}
+
+void
+gomp_team_barrier_wait_for_tasks (gomp_barrier_t *bar,
+				  gomp_barrier_state_t state, unsigned id)
+{
+  unsigned generation, gen;
+  bool have_cleared __attribute__ ((unused)) = false;
+
+  if (__builtin_expect (state & BAR_WAS_LAST, 0))
+    {
+      gomp_assert (id == 0, "Id %u believes it is last\n", id);
+      /* Acquire-release ordering from primary thread to secondary threads
+	 is formed in `gomp_team_barrier_done_final` rather than this function.
+	 That is where all secondary threads are allowed to run.
+	 However, we do still need to perform an atomic store as-per C "data
+	 race" requirements because there are other threads waiting in
+	 `do_spin` that are repeatedly watching this variable.  */
+      if (!gomp_team_barrier_primary_check_for_tasks (bar,
+						      BAR_HOLDING_SECONDARIES,
+						      state, MEMMODEL_RELAXED))
+	return;
+      state &= ~BAR_WAS_LAST;
+    }
+
+  generation = state;
+  state &= ~BAR_CANCELLED;
+  do
+    {
+      do_wait ((int *) &bar->generation, generation);
+      gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
+      gomp_assert ((gen & BAR_CANCELLED) == (generation & BAR_CANCELLED)
+		     /* Cancellation of *next* barrier is fine -- we will
+			exit.  */
+		     || gomp_barrier_state_is_incremented (
+		       gen, generation, BAR_HOLDING_SECONDARIES)
+		     /* Removing cancellation flag once all threads reached
+			"final" barrier is also fine.  */
+		     || ((gen & BAR_HOLDING_SECONDARIES)
+			 && !(generation & BAR_HOLDING_SECONDARIES))
+		     /* Can cancel a barrier when already gotten into final
+			implicit barrier at the end of a parallel loop.
+			This happens in `cancel-parallel-3.c`.
+			In this case the above assertion does not hold because
+			We are waiting on the implicit barrier at the end of a
+			parallel region while some other thread is performing
+			work in that parallel region, hits a
+			`#pragma omp cancel parallel`, and sets said flag.  */
+		     || !(generation & BAR_CANCELLED),
+		   "Unnecessary looping due to BAR_CANCELLED diff"
+		   " gen: %u  state: %u  generation: %u  id: %u",
+		   gen, state, generation, id);
+      gomp_assert (!gomp_barrier_state_is_incremented (gen, state,
+						       BAR_HOLDING_SECONDARIES)
+		     || !gomp_barrier_state_is_incremented (
+		       gen, gomp_increment_gen (state, BAR_INCR),
+		       BAR_HOLDING_SECONDARIES),
+		   "Generation has gone ahead of us two times!"
+		   " gen = %u, state = %u, generation = %u",
+		   gen, state, generation);
+      if (__builtin_expect (gen & BAR_TASK_PENDING, 0))
+	{
+	  gomp_barrier_handle_tasks (state, bar, BAR_HOLDING_SECONDARIES);
+	  gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
+	}
+#if _LIBGOMP_CHECKING_
+      if (have_cleared)
+	gomp_assert ((!(gen & BAR_WAITING_FOR_TASK)
+		      && !(generation & BAR_WAITING_FOR_TASK)),
+		     "Invalid state in "
+		     "gomp_team_barrier_primary_wait_for_tasks");
+      else if ((generation & BAR_WAITING_FOR_TASK)
+	       && !(gen & BAR_WAITING_FOR_TASK))
+	have_cleared = true;
+#endif
+      generation
+	= (generation & ~BAR_WAITING_FOR_TASK) | (gen & BAR_WAITING_FOR_TASK);
+      generation = (generation & ~BAR_CANCELLED) | (gen & BAR_CANCELLED);
+      /* Situation where we know all tasks are finished, and we were called
+	 with instructions to wait for all tasks to finish before letting the
+	 primary thread exit and holding secondary threads ready to be woken
+	 up.  */
+      if (id == 0 && gen & BAR_HOLDING_SECONDARIES)
+	return;
+      generation |= gen & BAR_HOLDING_SECONDARIES;
+    }
+  while ((gen & BAR_BOTH_GENS_MASK) != state + BAR_INCR);
+}
+
+void
+gomp_team_barrier_done_final (gomp_barrier_t *bar, unsigned id)
+{
+  gomp_assert (id == 0, "called with ID = %u", id);
+  unsigned gen = __atomic_load_n (&bar->generation, MEMMODEL_RELAXED);
+  gomp_assert ((gen & BAR_FLAGS_MASK) == BAR_HOLDING_SECONDARIES,
+	       "gomp_team_barrier_done_final called with generation: %u", gen);
+  gen &= ~BAR_HOLDING_SECONDARIES;
+  gen += BAR_INCR;
+  bar->use_tree = gomp_barrier_choose_tree ();
+  __atomic_store_n (&bar->generation, gen, MEMMODEL_RELEASE);
+  futex_wake ((int *) &bar->generation, INT_MAX);
+}
+
+void
+gomp_assert_and_increment_cancel_flag (gomp_barrier_t *bar, unsigned id,
+				       unsigned gens)
+{
+  unsigned orig = bar->threadgens[id].gen;
+  unsigned new = BAR_INCREMENT_CANCEL (orig);
+  gomp_assert_and_store_flag (&bar->threadgens[id].gen, new, orig, gens, id);
+}
+
+static inline bool
+gomp_team_barrier_cancel_wait_on_two_flags (gomp_barrier_t *bar,
+					    unsigned child_tid,
+					    unsigned long long spincount,
+					    unsigned gstate, unsigned tstate)
+{
+  unsigned long long j;
+  unsigned threadgen_orig;
+  struct thread_lock_data *arr = bar->threadgens;
+restart_loop:
+  for (j = 0; j <= spincount; j++)
+    {
+      threadgen_orig = __atomic_load_n (&arr[child_tid].gen, MEMMODEL_ACQUIRE);
+      unsigned threadgen = threadgen_orig & BAR_CANCEL_GEN_MASK;
+      gomp_assert ((threadgen_orig & BAR_FLAGS_MASK) == 0,
+		   "Id %u: thread local generation has non-zero flags: "
+		   "%u\n",
+		   child_tid, threadgen_orig);
+      if (__builtin_expect (threadgen != tstate, 0))
+	{
+	  gomp_assert (threadgen == BAR_INCREMENT_CANCEL (tstate),
+		       "Thread-local state seen to be %u"
+		       " when global state was %u.\n",
+		       threadgen, tstate);
+	  return true;
+	}
+      unsigned gen = __atomic_load_n (&bar->generation, MEMMODEL_RELAXED);
+      if (__builtin_expect (gen != gstate, 0))
+	{
+	  gomp_assert (!(gen & BAR_WAITING_FOR_TASK),
+		       "BAR_WAITING_FOR_TASK set in non-primary thread "
+		       "gen=%d",
+		       gen);
+	  gomp_assert (!gomp_barrier_state_is_incremented (gen, gstate,
+							   BAR_CANCEL_INCR),
+		       "Global state incremented by non-primary thread "
+		       "gstate=%d gen=%d",
+		       gstate, gen);
+	  if (gen & BAR_CANCELLED)
+	    return false;
+
+	  if (gen & BAR_TASK_PENDING)
+	    gomp_barrier_handle_tasks (gstate, bar, false);
+	  goto restart_loop;
+	}
+    }
+  futex_waitv (&arr[child_tid].gen, threadgen_orig, &bar->generation, gstate);
+  goto restart_loop;
+}
+
+gomp_barrier_state_t
+gomp_team_flat_barrier_wait_cancel_start (gomp_barrier_t *bar, unsigned id)
+{
+  unsigned ret = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
+  ret &= BAR_BOTH_GENS_MASK | BAR_CANCELLED;
+  if (ret & BAR_CANCELLED)
+    /* Do not return BAR_WAS_LAST because this thread is not last.
+       The barrier is cancelled.  Some thread did not enter it.  */
+    return ret;
+  if (id == 0)
+    {
+      unsigned gstate = ret & BAR_BOTH_GENS_MASK;
+      unsigned tstate = ret & BAR_CANCEL_GEN_MASK;
+      unsigned long long spincount = spin_count ();
+      for (unsigned i = 1; i < bar->total; i++)
+	if (!gomp_team_barrier_cancel_wait_on_two_flags (bar, i, spincount,
+							 gstate, tstate))
+	  return (ret | BAR_CANCELLED);
+      ret |= BAR_WAS_LAST;
+#if _LIBGOMP_CHECKING_
+      gomp_assert_and_increment_cancel_flag (bar, id, ret);
+#endif
+      gomp_assert_seenflags (bar, true);
+    }
+  else
+    gomp_assert_and_increment_cancel_flag (bar, id, ret);
+  return ret;
+}
+
+gomp_barrier_state_t
+gomp_team_tree_barrier_wait_cancel_start (gomp_barrier_t *bar, unsigned id)
+{
+  unsigned gstate = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
+  gstate &= (BAR_BOTH_GENS_MASK | BAR_CANCELLED);
+  if (gstate & BAR_CANCELLED)
+    /* Do not return BAR_WAS_LAST because this thread is not last.
+       The barrier is cancelled.  Some thread did not enter it.  */
+    return gstate;
+  unsigned tstate = gstate & BAR_CANCEL_GEN_MASK;
+  unsigned long long spincount = spin_count ();
+#define wait_on_child(child_tid)                                               \
+  if (!gomp_team_barrier_cancel_wait_on_two_flags (bar, child_tid, spincount,  \
+						   gstate, tstate))            \
+    return (gstate | BAR_CANCELLED);
+  hyper_iteration (id, bar,
+		   gomp_assert_and_increment_cancel_flag (bar, id, gstate),
+		   wait_on_child);
+#undef wait_on_child
+if (id == 0)
+  {
+#if _LIBGOMP_CHECKING_
+    gomp_assert_and_increment_cancel_flag (bar, id, gstate);
+#endif
+    gstate |= BAR_WAS_LAST;
+    gomp_assert_seenflags (bar, true);
+  }
+  return gstate;
+}
+#undef hyper_iteration
+
+bool
+gomp_team_barrier_wait_cancel_end_impl (gomp_barrier_t *bar,
+					gomp_barrier_state_t state, unsigned id)
+{
+  unsigned generation, gen;
+  gomp_assert (!(state & BAR_CANCELLED),
+	       "gomp_team_barrier_wait_cancel_end_impl called when barrier "
+	       "cancelled state: %u",
+	       state);
+
+  if (__builtin_expect (state & BAR_WAS_LAST, 0))
+    {
+      gomp_assert (id == 0, "Id %u believes it is last\n", id);
+      /* If primary thread got here, barrier was not cancelled (barrier can not
+	 be cancelled from a task because not closely nested in parallel
+	 region).  Hence if below function increments generation due to no
+	 tasks remaining we don't have to worry about any other threads having
+	 exited barrier due to cancellation beforehand.  */
+      if (!gomp_team_barrier_primary_check_for_tasks (bar, BAR_CANCEL_INCR,
+						      state, MEMMODEL_RELEASE))
+	return false;
+      state &= ~BAR_WAS_LAST;
+    }
+
+  generation = state;
+  do
+    {
+      do_wait ((int *) &bar->generation, generation);
+      gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
+      if (__builtin_expect (gen & BAR_CANCELLED, 0))
+	{
+	  if (__builtin_expect ((gen & BAR_CANCEL_GEN_MASK)
+				  != (state & BAR_CANCEL_GEN_MASK),
+				0))
+	    {
+	      /* Have cancelled a barrier just after completing the current
+		 one.  We must not reset our local state.  */
+	      gomp_assert ((gen & BAR_CANCEL_GEN_MASK)
+			     == (BAR_INCREMENT_CANCEL (state)
+				 & BAR_CANCEL_GEN_MASK),
+			   "Incremented global generation (cancellable) more "
+			   "than one gen: %u  original state: %u",
+			   gen, state);
+	      /* Other threads could have continued on in between the time that
+		 the primary thread signalled all other threads should wake up
+		 and the time that we actually read `bar->generation` above.
+		 Any one of them could be performing tasks so we can't assert
+		 that the corresponding flag is clear.
+
+		 W.r.t. the task handling that could have been set (primary
+		 thread increments generation telling us to go, then it
+		 continues itself and finds an `omp task` directive and
+		 schedules a task) we need to move on with it (PR122314).  */
+	      gomp_assert (!(gen & BAR_WAITING_FOR_TASK),
+			   "Generation incremented while  main thread is "
+			   "still waiting for tasks: gen: %u  original state:"
+			   " %u",
+			   gen, state);
+	      /* *This* barrier wasn't cancelled -- the next barrier is
+		 cancelled.  Returning `false` gives that information back up
+		 to calling functions.  */
+	      return false;
+	    }
+	  /* Need to reset our thread-local generation.  Don't want state to be
+	     messed up the next time we hit a cancellable barrier.
+
+	     We are helped by the invariant that when this barrier is cancelled,
+	     the next barrier that will be entered is a non-cancellable barrier.
+	     That means we don't have to worry about the primary thread getting
+	     confused by this generation being incremented to say it's reached a
+	     cancellable barrier.  Since we are the only thread that will modify
+	     our own control word we don't need to worry about non-atomic RMW.
+	   */
+	  unsigned *cgen = &bar->threadgens[id].gen;
+	  unsigned orig = *cgen;
+	  unsigned new = ((orig - BAR_CANCEL_INCR) & BAR_CANCEL_GEN_MASK)
+			 | (orig & ~BAR_CANCEL_GEN_MASK);
+	  __atomic_store_n (cgen, new, MEMMODEL_RELAXED);
+#if _LIBGOMP_CHECKING_
+	  unsigned global_gen = gen & BAR_BOTH_GENS_MASK;
+	  gomp_assert (new == global_gen,
+		       "Thread-local generation %u unknown modification:"
+		       " expected %u seen %u",
+		       id, BAR_INCREMENT_CANCEL (global_gen), orig);
+#endif
+	  return true;
+	}
+      if (__builtin_expect (gen & BAR_TASK_PENDING, 0))
+	{
+	  gomp_barrier_handle_tasks (state, bar, BAR_CANCEL_INCR);
+	  gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
+	}
+      generation |= gen & BAR_WAITING_FOR_TASK;
+    }
+  while ((gen & BAR_CANCEL_GEN_MASK)
+	 != (BAR_INCREMENT_CANCEL (state) & BAR_CANCEL_GEN_MASK));
+
+  return false;
+}
+
+void
+gomp_team_barrier_wake (gomp_barrier_t *bar, int count)
+{
+  futex_wake ((int *) &bar->generation, count == 0 ? INT_MAX : count);
+}
+
+void
+gomp_team_barrier_cancel (struct gomp_team *team)
+{
+  /* Always set CANCEL on the barrier.  This needs to be atomic because
+     barrier wait paths use `generation` to communicate without holding
+     team->task_lock.  Don't need any memory-ordering here since the only thing
+     that BAR_CANCELLED indicates is that the barrier is cancelled (no thread
+     infers anything else about data being available based on this flag).  */
+  unsigned orig = __atomic_fetch_or (&team->barrier.generation, BAR_CANCELLED,
+				     MEMMODEL_RELAXED);
+  if (!(orig & BAR_CANCELLED))
+    futex_wake ((int *) &team->barrier.generation, INT_MAX);
+}
diff --git a/libgomp/config/linux/futex_waitv/bar.h b/libgomp/config/linux/futex_waitv/bar.h
new file mode 100644
index 00000000000..c7fbc0d363f
--- /dev/null
+++ b/libgomp/config/linux/futex_waitv/bar.h
@@ -0,0 +1,406 @@
+/* Copyright The GNU Toolchain Authors.
+
+   This file is part of the GNU Offloading and Multi Processing Library
+   (libgomp).
+
+   Libgomp is free software; you can redistribute it and/or modify it
+   under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3, or (at your option)
+   any later version.
+
+   Libgomp is distributed in the hope that it will be useful, but WITHOUT ANY
+   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+   FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+   more details.
+
+   Under Section 7 of GPL version 3, you are granted additional
+   permissions described in the GCC Runtime Library Exception, version
+   3.1, as published by the Free Software Foundation.
+
+   You should have received a copy of the GNU General Public License and
+   a copy of the GCC Runtime Library Exception along with this program;
+   see the files COPYING3 and COPYING.RUNTIME respectively.  If not, see
+   <http://www.gnu.org/licenses/>.  */
+
+/* This is a Linux > 5.16 specific implementation of a barrier synchronization
+   mechanism for libgomp.  This type is private to the library.  This
+   implementation uses atomic instructions, and the futex and futex_waitv
+   syscalls.  */
+
+#ifndef GOMP_BARRIER_H
+#define GOMP_BARRIER_H 1
+
+#include "mutex.h"
+
+struct __attribute__ ((aligned (64))) thread_lock_data
+{
+  unsigned gen;
+};
+
+typedef struct
+{
+  /* Make sure total/allocated and generation are in mostly read cachelines,
+     awaited is in a separate cacheline and each generation structure is in a
+     separate cache line.  */
+  unsigned total __attribute__ ((aligned (64)));
+  unsigned allocated;
+  unsigned generation;
+  /* Which barrier to use for the current generation.  Updated once per
+     generation by whichever thread publishes the generation increment to
+     others just before that is published -- update determines what will be
+     used for *next* generation.  Since only used once per generation know can
+     adjust non-atomically.  It's important that this is the same according to
+     every thread in a given generation so all threads perform the same barrier
+     algorithm for any given generation.
+     N.b. our ability to adjust non-atomically is somewhat fragile.  Most
+     notably we are relying on the fact that
+     `gomp_team_barrier_wait_cancel_end` and `gomp_team_barrier_wait_end` do
+     not read `bar->use_tree` to avoid C data races.   */
+  bool use_tree;
+  /* `awaited` only used for "simple" barrier.  */
+  unsigned awaited __attribute__ ((aligned (64)));
+  struct thread_lock_data *threadgens;
+} gomp_barrier_t;
+
+typedef unsigned int gomp_barrier_state_t;
+
+/* The generation field contains a counter in the high bits, with a few
+   low bits dedicated to flags.  Note that TASK_PENDING and WAS_LAST can
+   share space because WAS_LAST is never stored back to generation.  */
+#define BAR_TASK_PENDING 1
+/* In this particular target BAR_WAS_LAST indicates something more like
+   "chosen by design to be last", but I like having the macro the same name as
+   it is given in other targets.  */
+#define BAR_WAS_LAST 1
+#define BAR_WAITING_FOR_TASK 2
+#define BAR_CANCELLED 4
+/* Flag to indicate that primary should be released while others should be
+   left.  */
+#define BAR_HOLDING_SECONDARIES 8
+/* Using bits 4 -> 11 for the generation number of cancellable barriers and
+   remaining bits for the generation number of non-cancellable barriers.  */
+#define BAR_CANCEL_INCR 16
+#define BAR_INCR 4096
+#define BAR_FLAGS_MASK (~(-BAR_CANCEL_INCR))
+#define BAR_GEN_MASK (-BAR_INCR)
+#define BAR_BOTH_GENS_MASK (-BAR_CANCEL_INCR)
+#define BAR_CANCEL_GEN_MASK (-BAR_CANCEL_INCR & (~(-BAR_INCR)))
+/* Increment BAR_CANCEL_INCR, with wrapping arithmetic within the bits assigned
+   to this generation number.  I.e. Increment, then set bits above BAR_INCR to
+   what they were before.  */
+#define BAR_INCREMENT_CANCEL(X)                                                \
+  ({                                                                           \
+    __typeof__ (X) _X = (X);                                                   \
+    (((_X + BAR_CANCEL_INCR) & BAR_CANCEL_GEN_MASK)                            \
+     | (_X & ~BAR_CANCEL_GEN_MASK));                                           \
+  })
+
+static inline bool
+gomp_barrier_choose_tree ()
+{
+  return gomp_barrier_type == GOMP_FLAT_BARRIER	  ? false
+	 : gomp_barrier_type == GOMP_TREE_BARRIER ? true
+						  : spin_count () < 50000000ULL;
+}
+
+static inline void
+gomp_barrier_init (gomp_barrier_t *bar, unsigned count, void *ptr)
+{
+  gomp_assert ((((uintptr_t) ptr) & (__alignof (struct thread_lock_data) - 1))
+		 == 0,
+	       "Barrier extra allocation is misaligned: %p", ptr);
+  bar->threadgens = (struct thread_lock_data *) ptr;
+  bar->use_tree = gomp_barrier_choose_tree ();
+  for (unsigned i = 0; i < count; ++i)
+    bar->threadgens[i].gen = 0;
+  bar->total = count;
+  bar->allocated = count;
+  bar->awaited = count;
+  bar->generation = 0;
+}
+
+static inline struct gomp_barrier_extra_alloc_needs
+gomp_barrier_extra_alloc (unsigned count)
+{
+  return (struct gomp_barrier_extra_alloc_needs){
+      sizeof(struct thread_lock_data) * count,
+      __alignof(struct thread_lock_data)};
+}
+
+/* When re-initialising a barrier we know that all threads are serialised on
+   something else (because `gomp_barrier_can_hold` returns true.  However we
+   still want to have memory synchronisation between each thread so still use
+   atomic operations.  */
+static inline void
+gomp_centralized_barrier_reinit (gomp_barrier_t *bar, unsigned nthreads)
+{
+  __atomic_add_fetch (&bar->awaited, nthreads - bar->total, MEMMODEL_ACQ_REL);
+  bar->total = nthreads;
+}
+
+static inline void
+gomp_centralized_barrier_init (gomp_barrier_t *bar, unsigned count)
+{
+  bar->threadgens = NULL;
+  bar->total = count;
+  bar->allocated = 0;
+  bar->awaited = count;
+  bar->generation = 0;
+}
+
+static inline void
+gomp_barrier_destroy (gomp_barrier_t *bar)
+{}
+
+extern void gomp_centralized_barrier_wait (gomp_barrier_t *);
+extern void gomp_centralized_barrier_wait_last (gomp_barrier_t *);
+
+extern gomp_barrier_state_t gomp_flat_barrier_wait_start (gomp_barrier_t *,
+							  unsigned);
+extern gomp_barrier_state_t gomp_tree_barrier_wait_start (gomp_barrier_t *,
+							  unsigned);
+extern void gomp_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t,
+				   unsigned);
+
+extern gomp_barrier_state_t gomp_team_flat_barrier_wait_start (gomp_barrier_t *,
+							       unsigned);
+extern gomp_barrier_state_t gomp_team_tree_barrier_wait_start (gomp_barrier_t *,
+							       unsigned);
+extern void gomp_team_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t,
+					unsigned);
+extern void gomp_team_barrier_wait_for_tasks (gomp_barrier_t *,
+					      gomp_barrier_state_t, unsigned);
+extern void gomp_team_barrier_done_final (gomp_barrier_t *bar, unsigned id);
+
+extern gomp_barrier_state_t
+gomp_team_flat_barrier_wait_cancel_start (gomp_barrier_t *, unsigned);
+extern gomp_barrier_state_t
+gomp_team_tree_barrier_wait_cancel_start (gomp_barrier_t *, unsigned);
+
+extern bool gomp_team_barrier_wait_cancel_end_impl (gomp_barrier_t *,
+						    gomp_barrier_state_t,
+						    unsigned);
+
+void gomp_team_barrier_wake (gomp_barrier_t *bar, int count);
+struct gomp_team;
+extern void gomp_team_barrier_cancel (struct gomp_team *);
+
+static inline gomp_barrier_state_t
+gomp_barrier_wait_start (gomp_barrier_t *bar, unsigned id)
+{
+  if (bar->use_tree)
+    return gomp_tree_barrier_wait_start (bar, id);
+  else
+    return gomp_flat_barrier_wait_start (bar, id);
+}
+
+/* Semantics are supposed to be like gomp_barrier_wait, except that if the
+   encountering thread is not the last one to hit the barrier, it returns
+   immediately.  The intended usage is that a thread which intends to
+   gomp_barrier_destroy this barrier calls gomp_barrier_wait, while all other
+   threads call gomp_barrier_wait_last.  When gomp_barrier_wait returns, the
+   barrier can be safely destroyed.
+
+   This is pretty difficult to implement with the `tree` based barrier.
+   If this thread should be the co-ordinator of a sub-group of threads then it
+   can't immediately exit.  Instead we wait until all threads "above" it have
+   arrived.  This fulfills the requirements of knowing that when
+   `gomp_barrier_wait` returns the barrier can be safely destroyed.  */
+static inline void
+gomp_barrier_wait_last (gomp_barrier_t *bar, unsigned id)
+{
+  gomp_assert (id != 0, "Thread with id %u called gomp_barrier_wait_last", id);
+  /* Just using `gomp_barrier_wait_start` means that:
+     - For the "flat" barrier all secondary threads mark themselves as having
+       arrived and immediately exit.
+     - For the "tree" barrier secondary threads wait for whatever children they
+       are supposed to wait on and then exit.
+     For the intended use of this function we don't need to care about being
+     called by the primary thread (see assertion above), if we did need to care
+     about such things then it would require a different barrier structure.  */
+  gomp_barrier_wait_start (bar, id);
+}
+
+static inline void
+gomp_barrier_wait (gomp_barrier_t *bar, unsigned id)
+{
+  gomp_barrier_state_t state;
+  if (bar->use_tree)
+    state = gomp_tree_barrier_wait_start (bar, id);
+  else
+    state = gomp_flat_barrier_wait_start (bar, id);
+  gomp_barrier_wait_end (bar, state, id);
+}
+
+static inline void
+gomp_team_barrier_wait (gomp_barrier_t *bar, unsigned id)
+{
+  gomp_barrier_state_t state;
+  if (bar->use_tree)
+    state = gomp_team_tree_barrier_wait_start (bar, id);
+  else
+    state = gomp_team_flat_barrier_wait_start (bar, id);
+  gomp_team_barrier_wait_end (bar, state, id);
+}
+
+static inline void
+gomp_team_barrier_wait_final (gomp_barrier_t *bar, unsigned id)
+{
+  gomp_barrier_state_t state;
+  if (bar->use_tree)
+    state = gomp_team_tree_barrier_wait_start (bar, id);
+  else
+    state = gomp_team_flat_barrier_wait_start (bar, id);
+  gomp_team_barrier_wait_for_tasks (bar, state, id);
+}
+
+static inline gomp_barrier_state_t
+gomp_barrier_wait_cancel_start (gomp_barrier_t *bar, unsigned id)
+{
+  if (bar->use_tree)
+    return gomp_team_tree_barrier_wait_cancel_start (bar, id);
+  else
+    return gomp_team_flat_barrier_wait_cancel_start (bar, id);
+}
+
+static inline bool
+gomp_team_barrier_wait_cancel (gomp_barrier_t *bar, unsigned id)
+{
+  gomp_barrier_state_t state;
+  if (bar->use_tree)
+    state = gomp_team_tree_barrier_wait_cancel_start (bar, id);
+  else
+    state = gomp_team_flat_barrier_wait_cancel_start (bar, id);
+
+  if (__builtin_expect (state & BAR_CANCELLED, 0))
+    return true;
+
+  return gomp_team_barrier_wait_cancel_end_impl (bar, state, id);
+}
+
+static inline bool
+gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
+				   gomp_barrier_state_t state, unsigned id)
+{
+  if (state & BAR_CANCELLED)
+    return true;
+  return gomp_team_barrier_wait_cancel_end_impl (bar, state, id);
+}
+
+static inline bool
+gomp_barrier_last_thread (gomp_barrier_state_t state)
+{
+  return state & BAR_WAS_LAST;
+}
+
+static inline bool
+gomp_barrier_can_hold (gomp_barrier_t *bar)
+{
+  return true;
+}
+
+static inline void
+gomp_team_barrier_set_task_pending (gomp_barrier_t *bar)
+{
+  __atomic_fetch_or (&bar->generation, BAR_TASK_PENDING, MEMMODEL_RELAXED);
+}
+
+static inline void
+gomp_team_barrier_clear_task_pending (gomp_barrier_t *bar)
+{
+  __atomic_fetch_and (&bar->generation, ~BAR_TASK_PENDING, MEMMODEL_RELAXED);
+}
+
+static inline void
+gomp_team_barrier_set_waiting_for_tasks (gomp_barrier_t *bar)
+{
+  __atomic_fetch_or (&bar->generation, BAR_WAITING_FOR_TASK, MEMMODEL_RELAXED);
+}
+
+static inline bool
+gomp_team_barrier_waiting_for_tasks (gomp_barrier_t *bar)
+{
+  unsigned gen = __atomic_load_n (&bar->generation, MEMMODEL_RELAXED);
+  return (gen & BAR_WAITING_FOR_TASK) != 0;
+}
+
+static inline bool
+gomp_team_barrier_cancelled (gomp_barrier_t *bar)
+{
+  unsigned gen = __atomic_load_n (&bar->generation, MEMMODEL_RELAXED);
+  return __builtin_expect ((gen & BAR_CANCELLED) != 0, 0);
+}
+
+static inline unsigned
+gomp_increment_gen (gomp_barrier_state_t gen, unsigned increment)
+{
+  unsigned gens = (gen & BAR_BOTH_GENS_MASK);
+  switch (increment)
+    {
+    case BAR_CANCEL_INCR:
+      return BAR_INCREMENT_CANCEL (gens);
+    /* Increment of `false` used when increment of barrier not possible.
+       Used when the primary thread is waiting for all other threads to arrive.
+       Using `false` just as something to make it clear in the code that this
+       function can't increment.  This function still has to handle such an
+       argument.  */
+    case false:
+    case BAR_INCR:
+      return gens + BAR_INCR;
+    case BAR_HOLDING_SECONDARIES:
+      return gens | BAR_HOLDING_SECONDARIES;
+    default:
+      gomp_fatal ("Unknown increment in gomp_increment_gen: %u\n", increment);
+    }
+}
+
+static inline void
+gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state,
+			unsigned increment)
+{
+  /* Using MEMMODEL_RELEASE because this will "publish" the user data written
+     in tasks to other threads.  */
+  bar->use_tree = gomp_barrier_choose_tree ();
+  __atomic_store_n (&bar->generation, gomp_increment_gen (state, increment),
+		    MEMMODEL_RELEASE);
+}
+
+static inline bool
+gomp_barrier_state_is_incremented (gomp_barrier_state_t gen,
+				   gomp_barrier_state_t state,
+				   unsigned increment)
+{
+  /* Remove all flags from the "original state" field.
+     Logic of the function is that if/when we spill over the generation will be
+     less than the original generation.  However just checking that the current
+     generation is less fails if it's less because flags have been cleared.
+     Hence check against a `state` with cleared flags.
+     `gomp_increment_gen` already clears flags before using `state` so this
+     makes no difference there.  */
+  state &= BAR_BOTH_GENS_MASK;
+  /* In the case of BAR_HOLDING_SECONDARIES the actual point at which all
+     threads are released is an increment of BAR_INCR after the primary thread
+     has continued past that temporary hold.  */
+  if (increment == BAR_HOLDING_SECONDARIES)
+    increment = BAR_INCR;
+  unsigned next = gomp_increment_gen (state, increment);
+  return __builtin_expect (next > state, 1) ? gen >= next : gen < state;
+}
+
+static inline bool
+gomp_barrier_has_completed (gomp_barrier_state_t state, gomp_barrier_t *bar,
+			    unsigned increment)
+{
+  unsigned curgen = __atomic_load_n (&bar->generation, MEMMODEL_RELAXED);
+  if (curgen & BAR_HOLDING_SECONDARIES)
+    {
+      gomp_assert (increment == BAR_HOLDING_SECONDARIES,
+		   "Barrier generation %u which claims holding secondaries"
+		   " but increment was %u",
+		   curgen, increment);
+      return true;
+    }
+  return gomp_barrier_state_is_incremented (curgen, state, increment);
+}
+
+#endif /* GOMP_BARRIER_H */
diff --git a/libgomp/config/linux/futex_waitv/futex_waitv.h b/libgomp/config/linux/futex_waitv/futex_waitv.h
new file mode 100644
index 00000000000..ca47752e78e
--- /dev/null
+++ b/libgomp/config/linux/futex_waitv/futex_waitv.h
@@ -0,0 +1,67 @@
+/* Copyright The GNU Toolchain Authors.
+
+   This file is part of the GNU Offloading and Multi Processing Library
+   (libgomp).
+
+   Libgomp is free software; you can redistribute it and/or modify it
+   under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3, or (at your option)
+   any later version.
+
+   Libgomp is distributed in the hope that it will be useful, but WITHOUT ANY
+   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+   FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+   more details.
+
+   Under Section 7 of GPL version 3, you are granted additional
+   permissions described in the GCC Runtime Library Exception, version
+   3.1, as published by the Free Software Foundation.
+
+   You should have received a copy of the GNU General Public License and
+   a copy of the GCC Runtime Library Exception along with this program;
+   see the files COPYING3 and COPYING.RUNTIME respectively.  If not, see
+   <http://www.gnu.org/licenses/>.  */
+
+#define _GNU_SOURCE
+#include <sys/syscall.h>
+
+#ifndef SYS_futex_waitv
+#  error SYS_futex_waitv is required for the linux/futex_waitv libgomp target
+#endif
+
+#include <linux/futex.h>
+#include <stdint.h>
+#include <string.h>
+#include <errno.h>
+#include <time.h>
+
+#pragma GCC visibility push(default)
+#include <unistd.h>
+#pragma GCC visibility pop
+
+/* Only define the interface we need.  `futex_waitv` can wait on more
+   addresses, but the barrier code only needs two.
+
+   See https://docs.kernel.org/userspace-api/futex2.html for documentation on
+   futex_waitv syscall.  */
+static inline void
+futex_waitv (uint32_t *addr, uint32_t val, uint32_t *addr2, uint32_t val2)
+{
+  struct futex_waitv addrs[2];
+  addrs[0].val = val;
+  addrs[0].uaddr = (uint64_t) (uintptr_t) addr;
+  addrs[0].flags = FUTEX_PRIVATE_FLAG | FUTEX_32;
+  addrs[0].__reserved = 0;
+  addrs[1].val = val2;
+  addrs[1].uaddr = (uint64_t) (uintptr_t) addr2;
+  addrs[1].flags = FUTEX_PRIVATE_FLAG | FUTEX_32;
+  addrs[1].__reserved = 0;
+  int err __attribute__ ((unused))
+    = syscall (SYS_futex_waitv, addrs, 2, 0, NULL, CLOCK_MONOTONIC);
+  /* If a signal woke us then we simply leave and let the loop outside of us
+     handle it.  We never require knowledge about whether anything changed or
+     not.  */
+  gomp_assert (err >= 0 || errno == EAGAIN || errno == EINTR,
+	       "Failed with futex_waitv err = %d, message: %s", err,
+	       strerror (errno));
+}
diff --git a/libgomp/config/linux/futex_waitv/simple-bar.h b/libgomp/config/linux/futex_waitv/simple-bar.h
new file mode 100644
index 00000000000..99391cfe880
--- /dev/null
+++ b/libgomp/config/linux/futex_waitv/simple-bar.h
@@ -0,0 +1,66 @@
+/* Copyright The GNU Toolchain Authors.
+
+   This file is part of the GNU Offloading and Multi Processing Library
+   (libgomp).
+
+   Libgomp is free software; you can redistribute it and/or modify it
+   under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3, or (at your option)
+   any later version.
+
+   Libgomp is distributed in the hope that it will be useful, but WITHOUT ANY
+   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+   FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+   more details.
+
+   Under Section 7 of GPL version 3, you are granted additional
+   permissions described in the GCC Runtime Library Exception, version
+   3.1, as published by the Free Software Foundation.
+
+   You should have received a copy of the GNU General Public License and
+   a copy of the GCC Runtime Library Exception along with this program;
+   see the files COPYING3 and COPYING.RUNTIME respectively.  If not, see
+   <http://www.gnu.org/licenses/>.  */
+
+/* This is a simplified barrier that is suitable for thread pool
+   synchronization.  Only a subset of full barrier API (bar.h) is exposed.  */
+
+#ifndef GOMP_SIMPLE_BARRIER_H
+#define GOMP_SIMPLE_BARRIER_H 1
+
+#include "bar.h"
+
+typedef struct
+{
+  gomp_barrier_t bar;
+} gomp_simple_barrier_t;
+
+static inline void
+gomp_simple_barrier_init (gomp_simple_barrier_t *bar, unsigned count)
+{
+  gomp_centralized_barrier_init (&bar->bar, count);
+}
+
+static inline void
+gomp_simple_barrier_reinit (gomp_simple_barrier_t *bar, unsigned nthreads)
+{
+  gomp_centralized_barrier_reinit (&bar->bar, nthreads);
+}
+
+static inline void
+gomp_simple_barrier_destroy (gomp_simple_barrier_t *bar)
+{}
+
+static inline void
+gomp_simple_barrier_wait (gomp_simple_barrier_t *bar)
+{
+  gomp_centralized_barrier_wait (&bar->bar);
+}
+
+static inline void
+gomp_simple_barrier_wait_last (gomp_simple_barrier_t *bar)
+{
+  gomp_centralized_barrier_wait_last (&bar->bar);
+}
+
+#endif /* GOMP_SIMPLE_BARRIER_H */
diff --git a/libgomp/config/linux/wait.h b/libgomp/config/linux/wait.h
index f9141408c5b..a6b235f89fc 100644
--- a/libgomp/config/linux/wait.h
+++ b/libgomp/config/linux/wait.h
@@ -35,6 +35,7 @@
 
 #define FUTEX_WAIT	0
 #define FUTEX_WAKE	1
+
 #define FUTEX_PRIVATE_FLAG	128
 
 #ifdef HAVE_ATTRIBUTE_VISIBILITY
@@ -45,14 +46,10 @@ extern int gomp_futex_wait, gomp_futex_wake;
 
 #include <futex.h>
 
-static inline int do_spin (int *addr, int val)
+static inline int
+do_spin (int *addr, int val)
 {
-  unsigned long long i, count = gomp_spin_count_var;
-
-  if (__builtin_expect (__atomic_load_n (&gomp_managed_threads,
-                                         MEMMODEL_RELAXED)
-                        > gomp_available_cpus, 0))
-    count = gomp_throttled_spin_count_var;
+  unsigned long long i, count = spin_count ();
   for (i = 0; i < count; i++)
     if (__builtin_expect (__atomic_load_n (addr, MEMMODEL_RELAXED) != val, 0))
       return 0;
diff --git a/libgomp/config/nvptx/bar.c b/libgomp/config/nvptx/bar.c
index aabc44a19e7..dbe59b04368 100644
--- a/libgomp/config/nvptx/bar.c
+++ b/libgomp/config/nvptx/bar.c
@@ -31,7 +31,8 @@
 #include "libgomp.h"
 
 void
-gomp_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
+gomp_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state,
+		       unsigned id __attribute__ ((unused)))
 {
   if (__builtin_expect (state & BAR_WAS_LAST, 0))
     {
@@ -45,9 +46,9 @@ gomp_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
 }
 
 void
-gomp_barrier_wait (gomp_barrier_t *bar)
+gomp_barrier_wait (gomp_barrier_t *bar, unsigned id)
 {
-  gomp_barrier_wait_end (bar, gomp_barrier_wait_start (bar));
+  gomp_barrier_wait_end (bar, gomp_barrier_wait_start (bar, id), id);
 }
 
 /* Like gomp_barrier_wait, except that if the encountering thread
@@ -58,7 +59,8 @@ gomp_barrier_wait (gomp_barrier_t *bar)
    the barrier can be safely destroyed.  */
 
 void
-gomp_barrier_wait_last (gomp_barrier_t *bar)
+gomp_barrier_wait_last (gomp_barrier_t *bar,
+			unsigned id __attribute__ ((unused)))
 {
   /* The above described behavior matches 'bar.arrive' perfectly.  */
   if (bar->total > 1)
@@ -77,7 +79,8 @@ gomp_barrier_wait_last (gomp_barrier_t *bar)
    on GPUs).  */
 
 void
-gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
+gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state,
+			    unsigned id __attribute__ ((unused)))
 {
   struct gomp_thread *thr = gomp_thread ();
   struct gomp_team *team = thr->ts.team;
@@ -98,7 +101,7 @@ gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
     {
       while (__atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE)
 	     & BAR_TASK_PENDING)
-	gomp_barrier_handle_tasks (state);
+	gomp_barrier_handle_tasks (state, bar, 0);
 
       if (bar->total > 1)
 	asm volatile ("bar.sync 1, %0;" : : "r" (32 * bar->total));
@@ -106,25 +109,26 @@ gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
 }
 
 void
-gomp_team_barrier_wait (gomp_barrier_t *bar)
+gomp_team_barrier_wait (gomp_barrier_t *bar, unsigned id)
 {
-  gomp_team_barrier_wait_end (bar, gomp_barrier_wait_start (bar));
+  gomp_team_barrier_wait_end (bar, gomp_barrier_wait_start (bar, id), id);
 }
 
 void
-gomp_team_barrier_wait_final (gomp_barrier_t *bar)
+gomp_team_barrier_wait_final (gomp_barrier_t *bar, unsigned id)
 {
-  gomp_barrier_state_t state = gomp_barrier_wait_final_start (bar);
+  gomp_barrier_state_t state = gomp_barrier_wait_final_start (bar, id);
   if (__builtin_expect (state & BAR_WAS_LAST, 0))
     bar->awaited_final = bar->total;
-  gomp_team_barrier_wait_end (bar, state);
+  gomp_team_barrier_wait_end (bar, state, id);
 }
 
 /* See also comments for gomp_team_barrier_wait_end.  */
 
 bool
 gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
-				   gomp_barrier_state_t state)
+				   gomp_barrier_state_t state,
+				   unsigned id __attribute__ ((unused)))
 {
   struct gomp_thread *thr = gomp_thread ();
   struct gomp_team *team = thr->ts.team;
@@ -152,7 +156,7 @@ gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
     {
       while (__atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE)
 	     & BAR_TASK_PENDING)
-	gomp_barrier_handle_tasks (state);
+	gomp_barrier_handle_tasks (state, bar, 0);
 
       if (bar->total > 1)
 	asm volatile ("bar.sync 1, %0;" : : "r" (32 * bar->total));
@@ -162,9 +166,11 @@ gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
 }
 
 bool
-gomp_team_barrier_wait_cancel (gomp_barrier_t *bar)
+gomp_team_barrier_wait_cancel (gomp_barrier_t *bar, unsigned id)
 {
-  return gomp_team_barrier_wait_cancel_end (bar, gomp_barrier_wait_start (bar));
+  return gomp_team_barrier_wait_cancel_end (bar,
+					    gomp_barrier_wait_start (bar, id),
+					    id);
 }
 
 void
diff --git a/libgomp/config/nvptx/bar.h b/libgomp/config/nvptx/bar.h
index fd375696580..09d6af71225 100644
--- a/libgomp/config/nvptx/bar.h
+++ b/libgomp/config/nvptx/bar.h
@@ -51,7 +51,9 @@ typedef unsigned int gomp_barrier_state_t;
 #define BAR_CANCELLED		4
 #define BAR_INCR		8
 
-static inline void gomp_barrier_init (gomp_barrier_t *bar, unsigned count)
+static inline void
+gomp_barrier_init (gomp_barrier_t *bar, unsigned count,
+		   void *ptr __attribute__ ((unused)))
 {
   bar->total = count;
   bar->awaited = count;
@@ -69,16 +71,23 @@ static inline void gomp_barrier_destroy (gomp_barrier_t *bar)
 {
 }
 
-extern void gomp_barrier_wait (gomp_barrier_t *);
-extern void gomp_barrier_wait_last (gomp_barrier_t *);
-extern void gomp_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t);
-extern void gomp_team_barrier_wait (gomp_barrier_t *);
-extern void gomp_team_barrier_wait_final (gomp_barrier_t *);
-extern void gomp_team_barrier_wait_end (gomp_barrier_t *,
-					gomp_barrier_state_t);
-extern bool gomp_team_barrier_wait_cancel (gomp_barrier_t *);
+static inline struct gomp_barrier_extra_alloc_needs
+gomp_barrier_extra_alloc (unsigned count __attribute__ ((unused)))
+{
+  return (struct gomp_barrier_extra_alloc_needs){0, 1};
+}
+
+extern void gomp_barrier_wait (gomp_barrier_t *, unsigned);
+extern void gomp_barrier_wait_last (gomp_barrier_t *, unsigned);
+extern void gomp_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t,
+				   unsigned);
+extern void gomp_team_barrier_wait (gomp_barrier_t *, unsigned);
+extern void gomp_team_barrier_wait_final (gomp_barrier_t *, unsigned);
+extern void gomp_team_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t,
+					unsigned);
+extern bool gomp_team_barrier_wait_cancel (gomp_barrier_t *, unsigned);
 extern bool gomp_team_barrier_wait_cancel_end (gomp_barrier_t *,
-					       gomp_barrier_state_t);
+					       gomp_barrier_state_t, unsigned);
 struct gomp_team;
 extern void gomp_team_barrier_cancel (struct gomp_team *);
 
@@ -90,7 +99,8 @@ gomp_team_barrier_wake (gomp_barrier_t *bar, int count)
 }
 
 static inline gomp_barrier_state_t
-gomp_barrier_wait_start (gomp_barrier_t *bar)
+gomp_barrier_wait_start (gomp_barrier_t *bar,
+			 unsigned id __attribute__ ((unused)))
 {
   unsigned int ret = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
   ret &= -BAR_INCR | BAR_CANCELLED;
@@ -105,16 +115,17 @@ gomp_barrier_wait_start (gomp_barrier_t *bar)
 }
 
 static inline gomp_barrier_state_t
-gomp_barrier_wait_cancel_start (gomp_barrier_t *bar)
+gomp_barrier_wait_cancel_start (gomp_barrier_t *bar, unsigned id)
 {
-  return gomp_barrier_wait_start (bar);
+  return gomp_barrier_wait_start (bar, id);
 }
 
 /* This is like gomp_barrier_wait_start, except it decrements
    bar->awaited_final rather than bar->awaited and should be used
    for the gomp_team_end barrier only.  */
 static inline gomp_barrier_state_t
-gomp_barrier_wait_final_start (gomp_barrier_t *bar)
+gomp_barrier_wait_final_start (gomp_barrier_t *bar,
+			       unsigned id __attribute__ ((unused)))
 {
   unsigned int ret = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
   ret &= -BAR_INCR | BAR_CANCELLED;
@@ -164,16 +175,29 @@ gomp_team_barrier_cancelled (gomp_barrier_t *bar)
 }
 
 static inline void
-gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state)
+gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state,
+			unsigned increment __attribute__ ((unused)))
 {
   bar->generation = (state & -BAR_INCR) + BAR_INCR;
 }
 
 static inline bool
-gomp_barrier_has_completed (gomp_barrier_state_t state, gomp_barrier_t *bar)
+gomp_barrier_has_completed (gomp_barrier_state_t state, gomp_barrier_t *bar,
+			    unsigned increment __attribute__ ((unused)))
 {
   unsigned int gen = bar->generation;
   return (gen & -BAR_INCR) == (state & -BAR_INCR) + BAR_INCR;
 }
 
+static inline bool
+gomp_barrier_can_hold (gomp_barrier_t *bar)
+{
+  return false;
+}
+
+/* Functions dummied out for this implementation.  */
+static inline void
+gomp_team_barrier_done_final (gomp_barrier_t *bar, unsigned id)
+{}
+
 #endif /* GOMP_BARRIER_H */
diff --git a/libgomp/config/nvptx/team.c b/libgomp/config/nvptx/team.c
index 39c8e8598b4..8cc35d72a87 100644
--- a/libgomp/config/nvptx/team.c
+++ b/libgomp/config/nvptx/team.c
@@ -136,7 +136,7 @@ gomp_thread_start (struct gomp_thread_pool *pool)
       thr->fn = NULL;
 
       struct gomp_task *task = thr->task;
-      gomp_team_barrier_wait_final (&thr->ts.team->barrier);
+      gomp_team_barrier_wait_final (&thr->ts.team->barrier, thr->ts.team_id);
       gomp_finish_task (task);
     }
   /* Work around an NVIDIA driver bug: when generating sm_50 machine code,
diff --git a/libgomp/config/posix/bar.c b/libgomp/config/posix/bar.c
index 8dbf7be08ea..caf2933d659 100644
--- a/libgomp/config/posix/bar.c
+++ b/libgomp/config/posix/bar.c
@@ -30,9 +30,9 @@
 
 #include "libgomp.h"
 
-
 void
-gomp_barrier_init (gomp_barrier_t *bar, unsigned count)
+gomp_barrier_init (gomp_barrier_t *bar, unsigned count,
+		   void *ptr __attribute__ ((unused)))
 {
   gomp_mutex_init (&bar->mutex1);
 #ifndef HAVE_SYNC_BUILTINS
@@ -105,13 +105,14 @@ gomp_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
 }
 
 void
-gomp_barrier_wait (gomp_barrier_t *barrier)
+gomp_barrier_wait (gomp_barrier_t *barrier, unsigned id)
 {
-  gomp_barrier_wait_end (barrier, gomp_barrier_wait_start (barrier));
+  gomp_barrier_wait_end (barrier, gomp_barrier_wait_start (barrier, id));
 }
 
 void
-gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
+gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state,
+			    unsigned id __attribute__ ((unused)))
 {
   unsigned int n;
 
@@ -127,7 +128,7 @@ gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
 	= __atomic_load_n (&team->task_count, MEMMODEL_ACQUIRE);
       if (task_count)
 	{
-	  gomp_barrier_handle_tasks (state);
+	  gomp_barrier_handle_tasks (state, bar, BAR_INCR);
 	  if (n > 0)
 	    gomp_sem_wait (&bar->sem2);
 	  gomp_mutex_unlock (&bar->mutex1);
@@ -154,7 +155,7 @@ gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
 	  gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
 	  if (gen & BAR_TASK_PENDING)
 	    {
-	      gomp_barrier_handle_tasks (state);
+	      gomp_barrier_handle_tasks (state, bar, BAR_INCR);
 	      gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
 	    }
 	}
@@ -175,7 +176,8 @@ gomp_team_barrier_wait_end (gomp_barrier_t *bar, gomp_barrier_state_t state)
 
 bool
 gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
-				   gomp_barrier_state_t state)
+				   gomp_barrier_state_t state,
+				   unsigned id __attribute__ ((unused)))
 {
   unsigned int n;
 
@@ -191,7 +193,7 @@ gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
 	= __atomic_load_n (&team->task_count, MEMMODEL_ACQUIRE);
       if (task_count)
 	{
-	  gomp_barrier_handle_tasks (state);
+	  gomp_barrier_handle_tasks (state, bar, BAR_INCR);
 	  if (n > 0)
 	    gomp_sem_wait (&bar->sem2);
 	  gomp_mutex_unlock (&bar->mutex1);
@@ -226,7 +228,7 @@ gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
 	    break;
 	  if (gen & BAR_TASK_PENDING)
 	    {
-	      gomp_barrier_handle_tasks (state);
+	      gomp_barrier_handle_tasks (state, bar, BAR_INCR);
 	      gen = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
 	      if (gen & BAR_CANCELLED)
 		break;
@@ -251,9 +253,10 @@ gomp_team_barrier_wait_cancel_end (gomp_barrier_t *bar,
 }
 
 void
-gomp_team_barrier_wait (gomp_barrier_t *barrier)
+gomp_team_barrier_wait (gomp_barrier_t *barrier, unsigned id)
 {
-  gomp_team_barrier_wait_end (barrier, gomp_barrier_wait_start (barrier));
+  gomp_team_barrier_wait_end (barrier, gomp_barrier_wait_start (barrier, id),
+			      id);
 }
 
 void
@@ -266,10 +269,10 @@ gomp_team_barrier_wake (gomp_barrier_t *bar, int count)
 }
 
 bool
-gomp_team_barrier_wait_cancel (gomp_barrier_t *bar)
+gomp_team_barrier_wait_cancel (gomp_barrier_t *bar, unsigned id)
 {
-  gomp_barrier_state_t state = gomp_barrier_wait_cancel_start (bar);
-  return gomp_team_barrier_wait_cancel_end (bar, state);
+  gomp_barrier_state_t state = gomp_barrier_wait_cancel_start (bar, id);
+  return gomp_team_barrier_wait_cancel_end (bar, state, id);
 }
 
 void
diff --git a/libgomp/config/posix/bar.h b/libgomp/config/posix/bar.h
index 8aab3733cc7..7ca78862da6 100644
--- a/libgomp/config/posix/bar.h
+++ b/libgomp/config/posix/bar.h
@@ -58,24 +58,31 @@ typedef unsigned int gomp_barrier_state_t;
 #define BAR_CANCELLED		4
 #define BAR_INCR		8
 
-extern void gomp_barrier_init (gomp_barrier_t *, unsigned);
+extern void gomp_barrier_init (gomp_barrier_t *, unsigned, void *);
 extern void gomp_barrier_reinit (gomp_barrier_t *, unsigned);
 extern void gomp_barrier_destroy (gomp_barrier_t *);
 
-extern void gomp_barrier_wait (gomp_barrier_t *);
+static inline struct gomp_barrier_extra_alloc_needs
+gomp_barrier_extra_alloc (unsigned count __attribute__ ((unused)))
+{
+  return (struct gomp_barrier_extra_alloc_needs){0, 1};
+}
+
+extern void gomp_barrier_wait (gomp_barrier_t *, unsigned);
 extern void gomp_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t);
-extern void gomp_team_barrier_wait (gomp_barrier_t *);
-extern void gomp_team_barrier_wait_end (gomp_barrier_t *,
-					gomp_barrier_state_t);
-extern bool gomp_team_barrier_wait_cancel (gomp_barrier_t *);
+extern void gomp_team_barrier_wait (gomp_barrier_t *, unsigned);
+extern void gomp_team_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t,
+					unsigned);
+extern bool gomp_team_barrier_wait_cancel (gomp_barrier_t *, unsigned);
 extern bool gomp_team_barrier_wait_cancel_end (gomp_barrier_t *,
-					       gomp_barrier_state_t);
+					       gomp_barrier_state_t, unsigned);
 extern void gomp_team_barrier_wake (gomp_barrier_t *, int);
 struct gomp_team;
 extern void gomp_team_barrier_cancel (struct gomp_team *);
 
 static inline gomp_barrier_state_t
-gomp_barrier_wait_start (gomp_barrier_t *bar)
+gomp_barrier_wait_start (gomp_barrier_t *bar,
+			 unsigned id __attribute__ ((unused)))
 {
   unsigned int ret;
   gomp_mutex_lock (&bar->mutex1);
@@ -86,7 +93,8 @@ gomp_barrier_wait_start (gomp_barrier_t *bar)
 }
 
 static inline gomp_barrier_state_t
-gomp_barrier_wait_cancel_start (gomp_barrier_t *bar)
+gomp_barrier_wait_cancel_start (gomp_barrier_t *bar,
+				unsigned id __attribute__ ((unused)))
 {
   unsigned int ret;
   gomp_mutex_lock (&bar->mutex1);
@@ -99,9 +107,9 @@ gomp_barrier_wait_cancel_start (gomp_barrier_t *bar)
 }
 
 static inline void
-gomp_team_barrier_wait_final (gomp_barrier_t *bar)
+gomp_team_barrier_wait_final (gomp_barrier_t *bar, unsigned id)
 {
-  gomp_team_barrier_wait (bar);
+  gomp_team_barrier_wait (bar, id);
 }
 
 static inline bool
@@ -111,9 +119,9 @@ gomp_barrier_last_thread (gomp_barrier_state_t state)
 }
 
 static inline void
-gomp_barrier_wait_last (gomp_barrier_t *bar)
+gomp_barrier_wait_last (gomp_barrier_t *bar, unsigned id)
 {
-  gomp_barrier_wait (bar);
+  gomp_barrier_wait (bar, id);
 }
 
 /* All the inlines below must be called with team->task_lock
@@ -150,7 +158,8 @@ gomp_team_barrier_cancelled (gomp_barrier_t *bar)
 }
 
 static inline void
-gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state)
+gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state,
+			unsigned increment __attribute__ ((unused)))
 {
   /* Need the atomic store for acquire-release synchronisation with the
      load in `gomp_team_barrier_wait_{cancel_,}end`.  See PR112356  */
@@ -159,10 +168,24 @@ gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state)
 }
 
 static inline bool
-gomp_barrier_has_completed (gomp_barrier_state_t state, gomp_barrier_t *bar)
+gomp_barrier_has_completed (gomp_barrier_state_t state, gomp_barrier_t *bar,
+			    unsigned increment __attribute__ ((unused)))
 {
   unsigned int gen = bar->generation;
   return (gen & -BAR_INCR) == (state & -BAR_INCR) + BAR_INCR;
 }
 
+static inline bool
+gomp_barrier_can_hold (gomp_barrier_t *bar)
+{
+  return false;
+}
+
+/* Functions dummied out for this implementation.  */
+static inline void
+gomp_team_barrier_done_final (gomp_barrier_t *bar,
+			      unsigned id __attribute__ ((unused)))
+{}
+
+/* TODO Introduce `gomp_barrier_completed`.  */
 #endif /* GOMP_BARRIER_H */
diff --git a/libgomp/config/posix/pool.h b/libgomp/config/posix/pool.h
index 1149354a7f3..385f339fd1f 100644
--- a/libgomp/config/posix/pool.h
+++ b/libgomp/config/posix/pool.h
@@ -44,6 +44,7 @@ gomp_get_thread_pool (struct gomp_thread *thr, unsigned nthreads)
       pool->threads_size = 0;
       pool->threads_used = 0;
       pool->last_team = NULL;
+      pool->prev_barrier = NULL;
       pool->threads_busy = nthreads;
       thr->thread_pool = pool;
       pthread_setspecific (gomp_thread_destructor, thr);
diff --git a/libgomp/config/posix/simple-bar.h b/libgomp/config/posix/simple-bar.h
index 9452a2d5fdc..dd467908bcb 100644
--- a/libgomp/config/posix/simple-bar.h
+++ b/libgomp/config/posix/simple-bar.h
@@ -39,13 +39,13 @@ typedef struct
 static inline void
 gomp_simple_barrier_init (gomp_simple_barrier_t *bar, unsigned count)
 {
-  gomp_barrier_init (&bar->bar, count);
+  gomp_barrier_init (&bar->bar, count, NULL);
 }
 
 static inline void
-gomp_simple_barrier_reinit (gomp_simple_barrier_t *bar, unsigned count)
+gomp_simple_barrier_reinit (gomp_simple_barrier_t *sbar, unsigned nthreads)
 {
-  gomp_barrier_reinit (&bar->bar, count);
+  gomp_barrier_reinit (&sbar->bar, nthreads);
 }
 
 static inline void
@@ -57,13 +57,15 @@ gomp_simple_barrier_destroy (gomp_simple_barrier_t *bar)
 static inline void
 gomp_simple_barrier_wait (gomp_simple_barrier_t *bar)
 {
-  gomp_barrier_wait (&bar->bar);
+  /* In default implementation the barrier ID argument is unused.  */
+  gomp_barrier_wait (&bar->bar, 0);
 }
 
 static inline void
 gomp_simple_barrier_wait_last (gomp_simple_barrier_t *bar)
 {
-  gomp_barrier_wait_last (&bar->bar);
+  /* In default implementation the barrier ID argument is unused.  */
+  gomp_barrier_wait_last (&bar->bar, 0);
 }
 
 #endif /* GOMP_SIMPLE_BARRIER_H */
diff --git a/libgomp/config/rtems/bar.h b/libgomp/config/rtems/bar.h
index 47b961c2104..5ca725bc0c2 100644
--- a/libgomp/config/rtems/bar.h
+++ b/libgomp/config/rtems/bar.h
@@ -54,7 +54,9 @@ typedef unsigned int gomp_barrier_state_t;
 #define BAR_CANCELLED		4
 #define BAR_INCR		8
 
-static inline void gomp_barrier_init (gomp_barrier_t *bar, unsigned count)
+static inline void
+gomp_barrier_init (gomp_barrier_t *bar, unsigned count,
+		   void *ptr __attribute__ ((unused)))
 {
   bar->total = count;
   bar->awaited = count;
@@ -73,22 +75,31 @@ static inline void gomp_barrier_destroy (gomp_barrier_t *bar)
 {
 }
 
-extern void gomp_barrier_wait (gomp_barrier_t *);
-extern void gomp_barrier_wait_last (gomp_barrier_t *);
-extern void gomp_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t);
-extern void gomp_team_barrier_wait (gomp_barrier_t *);
-extern void gomp_team_barrier_wait_final (gomp_barrier_t *);
-extern void gomp_team_barrier_wait_end (gomp_barrier_t *,
-					gomp_barrier_state_t);
-extern bool gomp_team_barrier_wait_cancel (gomp_barrier_t *);
+static inline struct gomp_barrier_extra_alloc_needs
+gomp_barrier_extra_alloc (unsigned count __attribute__ ((unused)))
+{
+  return (struct gomp_barrier_extra_alloc_needs){0, 1};
+}
+
+extern void gomp_barrier_wait (gomp_barrier_t *, unsigned);
+extern void gomp_barrier_wait_last (gomp_barrier_t *, unsigned);
+extern void gomp_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t,
+				   unsigned);
+
+extern void gomp_team_barrier_wait (gomp_barrier_t *, unsigned);
+extern void gomp_team_barrier_wait_final (gomp_barrier_t *, unsigned);
+extern void gomp_team_barrier_wait_end (gomp_barrier_t *, gomp_barrier_state_t,
+					unsigned);
+extern bool gomp_team_barrier_wait_cancel (gomp_barrier_t *, unsigned);
 extern bool gomp_team_barrier_wait_cancel_end (gomp_barrier_t *,
-					       gomp_barrier_state_t);
+					       gomp_barrier_state_t, unsigned);
 extern void gomp_team_barrier_wake (gomp_barrier_t *, int);
 struct gomp_team;
 extern void gomp_team_barrier_cancel (struct gomp_team *);
 
 static inline gomp_barrier_state_t
-gomp_barrier_wait_start (gomp_barrier_t *bar)
+gomp_barrier_wait_start (gomp_barrier_t *bar,
+			 unsigned id __attribute__ ((unused)))
 {
   unsigned int ret = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
   ret &= -BAR_INCR | BAR_CANCELLED;
@@ -103,16 +114,17 @@ gomp_barrier_wait_start (gomp_barrier_t *bar)
 }
 
 static inline gomp_barrier_state_t
-gomp_barrier_wait_cancel_start (gomp_barrier_t *bar)
+gomp_barrier_wait_cancel_start (gomp_barrier_t *bar, unsigned id)
 {
-  return gomp_barrier_wait_start (bar);
+  return gomp_barrier_wait_start (bar, id);
 }
 
 /* This is like gomp_barrier_wait_start, except it decrements
    bar->awaited_final rather than bar->awaited and should be used
    for the gomp_team_end barrier only.  */
 static inline gomp_barrier_state_t
-gomp_barrier_wait_final_start (gomp_barrier_t *bar)
+gomp_barrier_wait_final_start (gomp_barrier_t *bar,
+			       unsigned id __attribute__ ((unused)))
 {
   unsigned int ret = __atomic_load_n (&bar->generation, MEMMODEL_ACQUIRE);
   ret &= -BAR_INCR | BAR_CANCELLED;
@@ -162,7 +174,8 @@ gomp_team_barrier_cancelled (gomp_barrier_t *bar)
 }
 
 static inline void
-gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state)
+gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state,
+			unsigned increment __attribute__ ((unused)))
 {
   /* Need the atomic store for acquire-release synchronisation with the
      load in `gomp_team_barrier_wait_{cancel_,}end`.  See PR112356  */
@@ -171,10 +184,23 @@ gomp_team_barrier_done (gomp_barrier_t *bar, gomp_barrier_state_t state)
 }
 
 static inline bool
-gomp_barrier_has_completed (gomp_barrier_state_t state, gomp_barrier_t *bar)
+gomp_barrier_has_completed (gomp_barrier_state_t state, gomp_barrier_t *bar,
+			    unsigned increment __attribute__ ((unused)))
 {
   unsigned int gen = bar->generation;
   return (gen & -BAR_INCR) == (state & -BAR_INCR) + BAR_INCR;
 }
 
+static inline bool
+gomp_barrier_can_hold (gomp_barrier_t *bar)
+{
+  return false;
+}
+
+/* Functions dummied out for this implementation.  */
+static inline void
+gomp_team_barrier_done_final (gomp_barrier_t *bar,
+			      unsigned id __attribute__ ((unused)))
+{}
+
 #endif /* GOMP_BARRIER_H */
diff --git a/libgomp/configure b/libgomp/configure
index 6d3f9922f06..57bde51e68d 100755
--- a/libgomp/configure
+++ b/libgomp/configure
@@ -828,6 +828,7 @@ enable_libtool_lock
 enable_darwin_at_rpath
 enable_maintainer_mode
 enable_linux_futex
+enable_linux_futex_waitv
 enable_tls
 enable_symvers
 enable_cet
@@ -1488,6 +1489,9 @@ Optional Features:
                           enable make rules and dependencies not useful (and
                           sometimes confusing) to the casual installer
   --enable-linux-futex    use the Linux futex system call [default=default]
+  --enable-linux-futex-waitv
+                          use the Linux futex_waitv system call
+                          [default=default]
   --enable-tls            Use thread-local storage [default=yes]
   --enable-symvers=STYLE  enables symbol versioning of the shared library
                           [default=yes]
@@ -11858,7 +11862,7 @@ else
   lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2
   lt_status=$lt_dlunknown
   cat > conftest.$ac_ext <<_LT_EOF
-#line 11861 "configure"
+#line 11866 "configure"
 #include "confdefs.h"
 
 #if HAVE_DLFCN_H
@@ -11964,7 +11968,7 @@ else
   lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2
   lt_status=$lt_dlunknown
   cat > conftest.$ac_ext <<_LT_EOF
-#line 11967 "configure"
+#line 11972 "configure"
 #include "confdefs.h"
 
 #if HAVE_DLFCN_H
@@ -15944,6 +15948,80 @@ if test x$enable_linux_futex = xyes; then
 fi
 
 
+ # Check whether --enable-linux-futex-waitv was given.
+if test "${enable_linux_futex_waitv+set}" = set; then :
+  enableval=$enable_linux_futex_waitv;
+      case "$enableval" in
+       yes|no|default) ;;
+       *) as_fn_error $? "Unknown argument to enable/disable linux-futex-waitv" "$LINENO" 5 ;;
+                          esac
+
+else
+  enable_linux_futex_waitv=default
+fi
+
+
+
+case "$target" in
+  *-linux* | *-uclinux*)
+    if test x$enable_linux_futex != xyes; then
+      if test x$enable_linux_futex_waitv = xyes; then
+	as_fn_error $? "--enable-linux-futex-waitv requires --enable-linux-futex" "$LINENO" 5
+      fi
+      enable_linux_futex_waitv=no
+    else
+      case "$enable_linux_futex_waitv" in
+	default)
+	  enable_linux_futex_waitv=no
+	  cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+#include <sys/syscall.h>
+	     #include <unistd.h>
+int
+main ()
+{
+syscall (SYS_futex_waitv, 0, 0, 0, 0, 0);
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+  enable_linux_futex_waitv=yes
+else
+  enable_linux_futex_waitv=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext conftest.$ac_ext
+	  ;;
+	yes)
+	  cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+#include <sys/syscall.h>
+	     #include <unistd.h>
+int
+main ()
+{
+syscall (SYS_futex_waitv, 0, 0, 0, 0, 0);
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+
+else
+  as_fn_error $? "SYS_futex_waitv required for --enable-linux-futex-waitv" "$LINENO" 5
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext conftest.$ac_ext
+	  ;;
+      esac
+    fi
+    ;;
+  *)
+    enable_linux_futex_waitv=no
+    ;;
+esac
+
 # Check for pthread_{,attr_}[sg]etaffinity_np.
 cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
diff --git a/libgomp/configure.ac b/libgomp/configure.ac
index a7938beb1ca..b8ffa040030 100644
--- a/libgomp/configure.ac
+++ b/libgomp/configure.ac
@@ -255,6 +255,45 @@ esac
 
 GCC_LINUX_FUTEX(:)
 
+LIBGOMP_ENABLE(linux-futex-waitv, default, ,
+  [use the Linux futex_waitv system call],
+  permit yes|no|default)
+
+case "$target" in
+  *-linux* | *-uclinux*)
+    if test x$enable_linux_futex != xyes; then
+      if test x$enable_linux_futex_waitv = xyes; then
+	AC_MSG_ERROR([--enable-linux-futex-waitv requires --enable-linux-futex])
+      fi
+      enable_linux_futex_waitv=no
+    else
+      case "$enable_linux_futex_waitv" in
+	default)
+	  enable_linux_futex_waitv=no
+	  AC_LINK_IFELSE(
+	   [AC_LANG_PROGRAM(
+	    [#include <sys/syscall.h>
+	     #include <unistd.h>],
+	    [syscall (SYS_futex_waitv, 0, 0, 0, 0, 0);])],
+	    [enable_linux_futex_waitv=yes],
+	    [enable_linux_futex_waitv=no])
+	  ;;
+	yes)
+	  AC_LINK_IFELSE(
+	   [AC_LANG_PROGRAM(
+	    [#include <sys/syscall.h>
+	     #include <unistd.h>],
+	    [syscall (SYS_futex_waitv, 0, 0, 0, 0, 0);])],[],
+	    [AC_MSG_ERROR([SYS_futex_waitv required for --enable-linux-futex-waitv])])
+	  ;;
+      esac
+    fi
+    ;;
+  *)
+    enable_linux_futex_waitv=no
+    ;;
+esac
+
 # Check for pthread_{,attr_}[sg]etaffinity_np.
 AC_LINK_IFELSE(
  [AC_LANG_PROGRAM(
diff --git a/libgomp/configure.tgt b/libgomp/configure.tgt
index 46af75f978f..d0b07f64bc2 100644
--- a/libgomp/configure.tgt
+++ b/libgomp/configure.tgt
@@ -36,57 +36,63 @@ tmake_file=
 # Since we require POSIX threads, assume a POSIX system by default.
 config_path="posix"
 
+if test x$enable_linux_futex_waitv = xyes; then
+  linux_path="linux/futex_waitv linux"
+else
+  linux_path="linux"
+fi
+
 # Check for futex enabled all at once.
 if test x$enable_linux_futex = xyes; then
   case "${target}" in
 
     aarch64*-*-linux*)
-	config_path="linux posix"
+	config_path="$linux_path posix"
 	;;
 
     alpha*-*-linux*)
-	config_path="linux/alpha linux posix"
+	config_path="linux/alpha $linux_path posix"
 	;;
 
     arm*-*-linux*)
-	config_path="linux posix"
+	config_path="$linux_path posix"
 	;;
 
     hppa*-*-linux*)
-	config_path="linux posix"
+	config_path="$linux_path posix"
 	;;
 
     ia64*-*-linux*)
-	config_path="linux/ia64 linux posix"
+	config_path="linux/ia64 $linux_path posix"
 	;;
 
     loongarch*-*-linux*)
-	config_path="linux posix"
+	config_path="$linux_path posix"
 	;;
 
     mips*-*-linux*)
-	config_path="linux/mips linux posix"
+	config_path="linux/mips $linux_path posix"
 	;;
 
     powerpc*-*-linux*)
-	config_path="linux/powerpc linux posix"
+	config_path="linux/powerpc $linux_path posix"
 	;;
 
     riscv64*-*-linux*)
-	config_path="linux posix"
+	config_path="$linux_path posix"
 	;;
 
     s390*-*-linux*)
-	config_path="linux/s390 linux posix"
+	config_path="linux/s390 $linux_path posix"
 	;;
 
     tile*-*-linux*)
-	config_path="linux/tile linux posix"
+	config_path="linux/tile $linux_path posix"
 	;;
 
     # Note that bare i386 is not included here.  We need cmpxchg.
     i[456]86-*-linux* | x86_64-*-linux*)
-	config_path="linux/x86 linux posix"
+	config_path="linux/x86 $linux_path posix"
 	cat > conftestx.c <<EOF
 #ifndef __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
 #error need -march=i486
@@ -108,7 +114,7 @@ EOF
     sparcv9-*-linux* | sparc64-*-linux*)
 	echo "int i;" > conftestx.c
 	if ${CC} ${CFLAGS} -c -o conftestx.o conftestx.c > /dev/null 2>&1; then
-	  config_path="linux/sparc linux posix"
+	  config_path="linux/sparc $linux_path posix"
 	  case "`/usr/bin/file conftestx.o`" in
 	    *32-bit*)
 	      case " ${CC} ${CFLAGS}" in
diff --git a/libgomp/env.c b/libgomp/env.c
index 8088085dd6c..821a708aa56 100644
--- a/libgomp/env.c
+++ b/libgomp/env.c
@@ -124,6 +124,7 @@ size_t gomp_affinity_format_len;
 char *goacc_device_type;
 int goacc_device_num;
 int goacc_default_dims[GOMP_DIM_MAX];
+enum gomp_barrier_t gomp_barrier_type = GOMP_BARRIER_BY_SPINCOUNT;
 
 #ifndef LIBGOMP_OFFLOADED_ONLY
 
@@ -1131,6 +1132,35 @@ parse_wait_policy (const char *env, const char *val, void *const params[])
   return false;
 }
 
+/* Parse the GOMP_BARRIER_TYPE environment variable.  */
+static void
+parse_barrier_choice (const char *name, enum gomp_barrier_t *val)
+{
+  char *env = getenv (name);
+  if (env == NULL)
+    return;
+  while (isspace ((unsigned char) *env))
+    ++env;
+  if (*env == '\0')
+    goto invalid;
+  if (strncasecmp (env, "tree", 4) == 0)
+    *val = GOMP_TREE_BARRIER;
+  else if (strncasecmp (env, "flat", 4) == 0)
+    *val = GOMP_FLAT_BARRIER;
+  else if (strncasecmp (env, "auto", 4) == 0)
+    *val = GOMP_BARRIER_BY_SPINCOUNT;
+  else
+    goto invalid;
+  env += 4;
+  while (isspace ((unsigned char) *env))
+    ++env;
+  if (*env != '\0')
+    goto invalid;
+  return;
+invalid:
+  gomp_error ("Invalid value for environment variable %s", name);
+}
+
 /* Parse the GOMP_CPU_AFFINITY environment varible.  Return true if one was
    present and it was successfully parsed.  */
 
@@ -1974,6 +2004,10 @@ omp_display_env (int verbose)
       fprintf (stderr, "  [host] GOMP_SPINCOUNT = '%lu'\n",
 	       (unsigned long) gomp_spin_count_var);
 #endif
+      fprintf (stderr, "  [host] GOMP_BARRIER_TYPE = '%s'\n",
+	       gomp_barrier_type == GOMP_FLAT_BARRIER	? "flat"
+	       : gomp_barrier_type == GOMP_TREE_BARRIER ? "tree"
+							: "auto");
     }
 
   fputs ("OPENMP DISPLAY ENVIRONMENT END\n", stderr);
@@ -2495,6 +2529,8 @@ initialize_env (void)
 	gomp_error ("Stack size change failed: %s", strerror (err));
     }
 
+  parse_barrier_choice ("GOMP_BARRIER_TYPE", &gomp_barrier_type);
+
   handle_omp_display_env ();
 
   /* OpenACC.  */
diff --git a/libgomp/libgomp.h b/libgomp/libgomp.h
index 42f32439295..dd33bce7cc7 100644
--- a/libgomp/libgomp.h
+++ b/libgomp/libgomp.h
@@ -33,7 +33,7 @@
    that are part of the external ABI, and the lower case prefix "gomp"
    is used group items that are completely private to the library.  */
 
-#ifndef LIBGOMP_H 
+#ifndef LIBGOMP_H
 #define LIBGOMP_H 1
 
 #ifndef _LIBGOMP_CHECKING_
@@ -59,6 +59,7 @@
 #include <stdbool.h>
 #include <stdlib.h>
 #include <stdarg.h>
+#include <limits.h>
 
 /* Needed for memset in priority_queue.c.  */
 #if _LIBGOMP_CHECKING_
@@ -200,10 +201,55 @@ extern void gomp_vfatal (const char *, va_list)
 extern void gomp_fatal (const char *, ...)
 	__attribute__ ((noreturn, format (printf, 1, 2)));
 
+#if _LIBGOMP_CHECKING_
+#define gomp_assert(EXPR, MSG, ...)                                            \
+  do                                                                           \
+    {                                                                          \
+      if (!(EXPR))                                                             \
+	gomp_fatal ("%s:%d " MSG, __FILE__, __LINE__, ##__VA_ARGS__);          \
+    }                                                                          \
+  while (0)
+#else
+#define gomp_assert(...)                                                       \
+  do                                                                           \
+    {                                                                          \
+    }                                                                          \
+  while (0)
+#endif
+
 struct gomp_task;
 struct gomp_taskgroup;
 struct htab;
 
+struct gomp_barrier_extra_alloc_needs
+{
+  size_t size;
+  size_t align;
+};
+
+enum gomp_barrier_t
+{
+  GOMP_TREE_BARRIER,
+  GOMP_FLAT_BARRIER,
+  GOMP_BARRIER_BY_SPINCOUNT
+};
+extern enum gomp_barrier_t gomp_barrier_type;
+
+extern unsigned long long gomp_spin_count_var, gomp_throttled_spin_count_var;
+extern unsigned long gomp_available_cpus, gomp_managed_threads;
+
+static inline unsigned long long
+spin_count ()
+{
+  unsigned long long count = gomp_spin_count_var;
+  if (__builtin_expect (__atomic_load_n (&gomp_managed_threads,
+					 MEMMODEL_RELAXED)
+			  > gomp_available_cpus,
+			0))
+    count = gomp_throttled_spin_count_var;
+  return count;
+}
+
 #include "priority_queue.h"
 #include "sem.h"
 #include "mutex.h"
@@ -405,7 +451,7 @@ extern char gomp_workshare_struct_check1
 extern char gomp_workshare_struct_check2
   [offsetof (struct gomp_work_share, lock) == 64 ? 1 : -1];
 
-/* This structure contains all of the thread-local data associated with 
+/* This structure contains all of the thread-local data associated with
    a thread team.  This is the data that must be saved when a thread
    encounters a nested PARALLEL construct.  */
 
@@ -415,7 +461,7 @@ struct gomp_team_state
   struct gomp_team *team;
 
   /* This is the work share construct which this thread is currently
-     processing.  Recall that with NOWAIT, not all threads may be 
+     processing.  Recall that with NOWAIT, not all threads may be
      processing the same construct.  */
   struct gomp_work_share *work_share;
 
@@ -494,7 +540,7 @@ enum gomp_device_num
    section 2.3.1.  Those described as having one copy per task are
    stored within the structure; those described as having one copy
    for the whole program are (naturally) global variables.  */
-   
+
 struct gomp_task_icv
 {
   unsigned long nthreads_var;
@@ -596,11 +642,10 @@ extern struct gomp_task_icv gomp_global_icv;
 #ifndef HAVE_SYNC_BUILTINS
 extern gomp_mutex_t gomp_managed_threads_lock;
 #endif
+/* Included here for the variables.  */
 extern bool gomp_cancel_var;
 extern enum gomp_target_offload_t gomp_target_offload_var;
 extern int gomp_max_task_priority_var;
-extern unsigned long long gomp_spin_count_var, gomp_throttled_spin_count_var;
-extern unsigned long gomp_available_cpus, gomp_managed_threads;
 extern unsigned long *gomp_nthreads_var_list, gomp_nthreads_var_list_len;
 extern char *gomp_bind_var_list;
 extern unsigned long gomp_bind_var_list_len;
@@ -926,6 +971,7 @@ struct gomp_thread_pool
 
   /* This barrier holds and releases threads waiting in thread pools.  */
   gomp_simple_barrier_t threads_dock;
+  gomp_barrier_t *prev_barrier;
 };
 
 enum gomp_cancel_kind
@@ -1099,7 +1145,8 @@ extern unsigned gomp_dynamic_max_threads (void);
 extern void gomp_init_task (struct gomp_task *, struct gomp_task *,
 			    struct gomp_task_icv *);
 extern void gomp_end_task (void);
-extern void gomp_barrier_handle_tasks (gomp_barrier_state_t);
+extern void gomp_barrier_handle_tasks (gomp_barrier_state_t, gomp_barrier_t *,
+				       unsigned);
 extern void gomp_task_maybe_wait_for_dependencies (void **);
 extern bool gomp_create_target_task (struct gomp_device_descr *,
 				     void (*) (void *), size_t, void **,
@@ -1344,7 +1391,7 @@ typedef struct acc_dispatch_t
   __typeof (GOMP_OFFLOAD_openacc_create_thread_data) *create_thread_data_func;
   __typeof (GOMP_OFFLOAD_openacc_destroy_thread_data)
     *destroy_thread_data_func;
-  
+
   struct {
     /* Once created and put into the "active" list, asyncqueues are then never
        destructed and removed from the "active" list, other than if the TODO
diff --git a/libgomp/single.c b/libgomp/single.c
index 65126000ea9..1c6afc67fdc 100644
--- a/libgomp/single.c
+++ b/libgomp/single.c
@@ -77,7 +77,7 @@ GOMP_single_copy_start (void)
     }
   else
     {
-      gomp_team_barrier_wait (&thr->ts.team->barrier);
+      gomp_team_barrier_wait (&thr->ts.team->barrier, thr->ts.team_id);
 
       ret = thr->ts.work_share->copyprivate;
       gomp_work_share_end_nowait ();
@@ -98,7 +98,7 @@ GOMP_single_copy_end (void *data)
   if (team != NULL)
     {
       thr->ts.work_share->copyprivate = data;
-      gomp_team_barrier_wait (&team->barrier);
+      gomp_team_barrier_wait (&team->barrier, thr->ts.team_id);
     }
 
   gomp_work_share_end_nowait ();
diff --git a/libgomp/task.c b/libgomp/task.c
index cbba28516e3..0374df44389 100644
--- a/libgomp/task.c
+++ b/libgomp/task.c
@@ -1549,10 +1549,39 @@ gomp_task_run_post_remove_taskgroup (struct gomp_task *child_task)
 }
 
 void
-gomp_barrier_handle_tasks (gomp_barrier_state_t state)
+gomp_barrier_handle_tasks (gomp_barrier_state_t state, gomp_barrier_t *bar,
+			   unsigned increment)
 {
   struct gomp_thread *thr = gomp_thread ();
-  struct gomp_team *team = thr->ts.team;
+  /* acquire-release between the store in `gomp_team_start` because otherwise
+     very small chance that the `team->barrier` we just read is seen as
+     uninitialised (despite happening before the assignment of `nthr->ts.team`
+     in the primary thread) and that uninitialised value happens to match
+     `bar`.  */
+  struct gomp_team *team = __atomic_load_n (&thr->ts.team, MEMMODEL_ACQUIRE);
+  /* When performing the barrier in between iterations of non-nested threads in
+     `gomp_thread_start`, there's the race condition where:
+     - Secondary thread calls this function.
+     - Some other thread finishes the last task.
+     - Barrier is marked as completed.
+     - Primary thread continues to starting a new parallel region.
+     - Primary thread creates a new team.
+     - Primary thread stores the team on this secondary threads TLS storage
+       (before we read the `thr->ts.team` value above.
+     We pass `bar` in from the barrier functions in order to identify this
+     case.  When this happens we know that all tasks on the barrier have
+     completed (otherwise the primary thread would not have continued past the
+     barrier).  Hence we can simply return.
+
+     This race condition was mostly harmless (attempt to run tasks on the
+     barrier of this new team, but new team would have no tasks on it anyway).
+     With the switch to only have one barrier per iteration in
+     `gomp_thread_start` we add the possibility that we could perform tasks on
+     "the next" parallel region (no longer a second synch point between the
+     above race condition and where new tasks could be scheduled).  This
+     outcome is something we want to avoid.  */
+  if (&team->barrier != bar)
+    return;
   struct gomp_task *task = thr->task;
   struct gomp_task *child_task = NULL;
   struct gomp_task *to_free = NULL;
@@ -1570,7 +1599,7 @@ gomp_barrier_handle_tasks (gomp_barrier_state_t state)
      When `task_count == 0` we're not going to perform tasks anyway, so the
      problem of PR122314 is naturally avoided.  */
   if (team->task_count != 0
-      && gomp_barrier_has_completed (state, &team->barrier))
+      && gomp_barrier_has_completed (state, &team->barrier, increment))
     {
       gomp_mutex_unlock (&team->task_lock);
       return;
@@ -1580,7 +1609,7 @@ gomp_barrier_handle_tasks (gomp_barrier_state_t state)
     {
       if (team->task_count == 0)
 	{
-	  gomp_team_barrier_done (&team->barrier, state);
+	  gomp_team_barrier_done (&team->barrier, state, increment);
 	  gomp_mutex_unlock (&team->task_lock);
 	  gomp_team_barrier_wake (&team->barrier, 0);
 	  return;
@@ -1617,7 +1646,7 @@ gomp_barrier_handle_tasks (gomp_barrier_state_t state)
       else if (team->task_count == 0
 	       && gomp_team_barrier_waiting_for_tasks (&team->barrier))
 	{
-	  gomp_team_barrier_done (&team->barrier, state);
+	  gomp_team_barrier_done (&team->barrier, state, increment);
 	  gomp_mutex_unlock (&team->task_lock);
 	  gomp_team_barrier_wake (&team->barrier, 0);
 	  if (to_free)
@@ -2243,7 +2272,7 @@ GOMP_taskgroup_end (void)
 	 is #pragma omp target nowait that creates an implicit
 	 team with a single thread.  In this case, we want to wait
 	 for all outstanding tasks in this team.  */
-      gomp_team_barrier_wait (&team->barrier);
+      gomp_team_barrier_wait (&team->barrier, thr->ts.team_id);
       return;
     }
 
@@ -2698,7 +2727,7 @@ GOMP_workshare_task_reduction_unregister (bool cancelled)
     htab_free ((struct htab *) data[5]);
 
   if (!cancelled)
-    gomp_team_barrier_wait (&team->barrier);
+    gomp_team_barrier_wait (&team->barrier, thr->ts.team_id);
 }
 
 int
diff --git a/libgomp/team.c b/libgomp/team.c
index 5282a3133ba..5b97383d2e6 100644
--- a/libgomp/team.c
+++ b/libgomp/team.c
@@ -31,6 +31,24 @@
 #include <stdlib.h>
 #include <string.h>
 
+static void
+gomp_release_held_threads (struct gomp_thread_pool *pool,
+			   struct gomp_team *team, unsigned team_id)
+{
+  gomp_assert (team_id == 0, "Releasing threads from non-primary thread %u",
+	       team_id);
+  if (pool->prev_barrier)
+    {
+      struct gomp_team *saved_team __attribute__ ((unused))
+      = pool->last_team ? pool->last_team : team;
+      gomp_assert (pool->prev_barrier == &saved_team->barrier,
+		   "prev_barrier not within cached team: %p != %p",
+		   pool->prev_barrier, &saved_team->barrier);
+      gomp_team_barrier_done_final (pool->prev_barrier, team_id);
+      pool->prev_barrier = NULL;
+    }
+}
+
 #ifdef LIBGOMP_USE_PTHREADS
 pthread_attr_t gomp_thread_attr;
 
@@ -62,7 +80,6 @@ struct gomp_thread_start_data
   pthread_t handle;
 };
 
-
 /* This function is a pthread_create entry point.  This contains the idle
    loop in which a thread waits to be called up to become part of a team.  */
 
@@ -109,12 +126,12 @@ gomp_thread_start (void *xdata)
       struct gomp_team *team = thr->ts.team;
       struct gomp_task *task = thr->task;
 
-      gomp_barrier_wait (&team->barrier);
+      gomp_barrier_wait (&team->barrier, thr->ts.team_id);
 
       local_fn (local_data);
-      gomp_team_barrier_wait_final (&team->barrier);
+      gomp_team_barrier_wait_final (&team->barrier, thr->ts.team_id);
       gomp_finish_task (task);
-      gomp_barrier_wait_last (&team->barrier);
+      gomp_barrier_wait_last (&team->barrier, thr->ts.team_id);
     }
   else
     {
@@ -127,10 +144,19 @@ gomp_thread_start (void *xdata)
 	  struct gomp_task *task = thr->task;
 
 	  local_fn (local_data);
-	  gomp_team_barrier_wait_final (&team->barrier);
+	  gomp_team_barrier_wait_final (&team->barrier, thr->ts.team_id);
 	  gomp_finish_task (task);
-
-	  gomp_simple_barrier_wait (&pool->threads_dock);
+	  /* Hold at the simple barrier if we're exiting.
+	     Do this in order to synchronise with the primary thread that could
+	     (after the next piece of work has been performed) free our team.
+	     Need to ensure that the barrier waited on above is not freed until
+	     we've completed it.  With any threads that will be used in the
+	     next iteration we know that it will synchronise with the primary
+	     thread before it frees this team because it will synchronise with
+	     the primary thread in the next iteration of this loop (in
+	     `gomp_team_barrier_wait_final`).  */
+	  if (!gomp_barrier_can_hold (&team->barrier) || thr->fn == NULL)
+	    gomp_simple_barrier_wait (&pool->threads_dock);
 
 	  local_fn = thr->fn;
 	  local_data = thr->data;
@@ -153,13 +179,20 @@ get_last_team (unsigned nthreads)
   struct gomp_thread *thr = gomp_thread ();
   if (thr->ts.team == NULL)
     {
+      gomp_assert (thr->ts.level == 0,
+		   "Looking for cached team in nested region %u thr->ts.level",
+		   thr->ts.level);
       struct gomp_thread_pool *pool = gomp_get_thread_pool (thr, nthreads);
       struct gomp_team *last_team = pool->last_team;
       if (last_team != NULL && last_team->nthreads == nthreads)
         {
-          pool->last_team = NULL;
-          return last_team;
-        }
+	  gomp_assert (!pool->prev_barrier
+			 || pool->prev_barrier == &pool->last_team->barrier,
+		       "prev_barrier not within cached team: %p != %p",
+		       pool->prev_barrier, &pool->last_team->barrier);
+	  pool->last_team = NULL;
+	  return last_team;
+	}
     }
   return NULL;
 }
@@ -175,19 +208,41 @@ gomp_new_team (unsigned nthreads)
   team = get_last_team (nthreads);
   if (team == NULL)
     {
+      void *barrier_extra;
+      struct gomp_barrier_extra_alloc_needs barrier_extra_alloc
+	= gomp_barrier_extra_alloc (nthreads);
       size_t extra = sizeof (team->ordered_release[0])
 		     + sizeof (team->implicit_task[0]);
+      size_t team_extra = nthreads * extra;
+      size_t barrier_padding = 0;
+      if (barrier_extra_alloc.size)
+	{
+	  size_t barrier_offset = sizeof (*team) + team_extra;
+	  gomp_assert (barrier_extra_alloc.align
+			 <= __alignof (struct gomp_team),
+		       "Barrier extra alignment too large: %lu > %lu",
+		       (unsigned long) barrier_extra_alloc.align,
+		       (unsigned long) __alignof (struct gomp_team));
+	  barrier_padding = ((barrier_offset + barrier_extra_alloc.align - 1)
+			     & ~(barrier_extra_alloc.align - 1))
+			    - barrier_offset;
+	}
+      size_t alloc_extra
+	= team_extra + barrier_padding + barrier_extra_alloc.size;
 #ifdef GOMP_USE_ALIGNED_WORK_SHARES
       team = gomp_aligned_alloc (__alignof (struct gomp_team),
-				 sizeof (*team) + nthreads * extra);
+				 sizeof (*team) + alloc_extra);
 #else
-      team = team_malloc (sizeof (*team) + nthreads * extra);
+      team = team_malloc (sizeof (*team) + alloc_extra);
 #endif
 
 #ifndef HAVE_SYNC_BUILTINS
       gomp_mutex_init (&team->work_share_list_free_lock);
 #endif
-      gomp_barrier_init (&team->barrier, nthreads);
+      barrier_extra = (char *) &team->implicit_task[nthreads]
+		      + nthreads * sizeof (team->ordered_release[0])
+		      + barrier_padding;
+      gomp_barrier_init (&team->barrier, nthreads, barrier_extra);
       gomp_mutex_init (&team->task_lock);
 
       team->nthreads = nthreads;
@@ -278,9 +333,11 @@ gomp_free_thread (void *arg __attribute__((unused)))
 	      nthr->data = pool;
 	    }
 	  /* This barrier undocks threads docked on pool->threads_dock.  */
-	  gomp_simple_barrier_wait (&pool->threads_dock);
-	  /* And this waits till all threads have called gomp_barrier_wait_last
-	     in gomp_free_pool_helper.  */
+	  gomp_release_held_threads (pool, NULL, thr->ts.team_id);
+	  if (!gomp_barrier_can_hold (pool->prev_barrier))
+	    gomp_simple_barrier_wait (&pool->threads_dock);
+	  /* And this waits till all threads have called
+	     gomp_simple_barrier_wait_last in gomp_free_pool_helper.  */
 	  gomp_simple_barrier_wait (&pool->threads_dock);
 	  /* Now it is safe to destroy the barrier and free the pool.  */
 	  gomp_simple_barrier_destroy (&pool->threads_dock);
@@ -315,6 +372,21 @@ gomp_free_thread (void *arg __attribute__((unused)))
 /* Launch a team.  */
 
 #ifdef LIBGOMP_USE_PTHREADS
+static unsigned
+gomp_barrier_calc_wait (unsigned num_new_threads, unsigned num_exiting_threads,
+			unsigned old_threads_used, unsigned nthreads,
+			unsigned affinity_count, bool can_hold)
+{
+  if (can_hold)
+    {
+      unsigned tmp = num_new_threads + num_exiting_threads;
+      return tmp ? tmp + 1 : 0;
+    }
+  if (!affinity_count)
+    return old_threads_used > nthreads ? old_threads_used : nthreads;
+  return nthreads + affinity_count;
+}
+
 void
 gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads,
 		 unsigned flags, struct gomp_team *team,
@@ -327,6 +399,10 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads,
   bool nested;
   struct gomp_thread_pool *pool;
   unsigned i, n, old_threads_used = 0;
+  unsigned simple_barrier_n = 0;
+  unsigned num_new_threads = 0;
+  unsigned spawned_new_threads __attribute__ ((unused)) = 0;
+  unsigned num_exiting_threads = 0;
   pthread_attr_t thread_attr, *attr;
   unsigned long nthreads_var;
   char bind, bind_var;
@@ -464,22 +540,61 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads,
      only the initial program thread will modify gomp_threads.  */
   if (!nested)
     {
+      gomp_assert (team->prev_ts.team_id == 0,
+		   "Starting a team from thread with id %u in previous team\n",
+		   team->prev_ts.team_id);
       old_threads_used = pool->threads_used;
 
-      if (nthreads <= old_threads_used)
-	n = nthreads;
+      if (nthreads == old_threads_used)
+	{
+	  n = nthreads;
+	  num_new_threads = 0;
+	  num_exiting_threads = 0;
+	  simple_barrier_n
+	    = gomp_barrier_calc_wait (num_new_threads, num_exiting_threads,
+				      old_threads_used, nthreads,
+				      affinity_count,
+				      gomp_barrier_can_hold (
+					pool->prev_barrier));
+	  if (gomp_barrier_can_hold (pool->prev_barrier))
+	    gomp_simple_barrier_reinit (&pool->threads_dock, simple_barrier_n);
+	}
+      else if (nthreads < old_threads_used)
+	{
+	  n = nthreads;
+	  num_new_threads = 0;
+	  num_exiting_threads = old_threads_used - nthreads;
+	  simple_barrier_n
+	    = gomp_barrier_calc_wait (num_new_threads, num_exiting_threads,
+				      old_threads_used, nthreads,
+				      affinity_count,
+				      gomp_barrier_can_hold (
+					pool->prev_barrier));
+	  if (gomp_barrier_can_hold (pool->prev_barrier))
+	    gomp_simple_barrier_reinit (&pool->threads_dock, simple_barrier_n);
+	}
       else if (old_threads_used == 0)
 	{
 	  n = 0;
+	  num_new_threads = nthreads - 1;
+	  num_exiting_threads = 0;
+	  simple_barrier_n = nthreads;
 	  gomp_simple_barrier_init (&pool->threads_dock, nthreads);
 	}
       else
 	{
 	  n = old_threads_used;
-
+	  num_new_threads = nthreads - old_threads_used;
+	  num_exiting_threads = 0;
+	  simple_barrier_n
+	    = gomp_barrier_calc_wait (num_new_threads, num_exiting_threads,
+				      old_threads_used, nthreads,
+				      affinity_count,
+				      gomp_barrier_can_hold (
+					pool->prev_barrier));
 	  /* Increase the barrier threshold to make sure all new
 	     threads arrive before the team is released.  */
-	  gomp_simple_barrier_reinit (&pool->threads_dock, nthreads);
+	  gomp_simple_barrier_reinit (&pool->threads_dock, simple_barrier_n);
 	}
 
       /* Not true yet, but soon will be.  We're going to release all
@@ -640,7 +755,7 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads,
 	    }
 	  else
 	    nthr = pool->threads[i];
-	  nthr->ts.team = team;
+	  __atomic_store_n (&nthr->ts.team, team, MEMMODEL_RELEASE);
 	  nthr->ts.work_share = &team->work_shares[0];
 	  nthr->ts.last_work_share = NULL;
 	  nthr->ts.team_id = i;
@@ -719,14 +834,68 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads,
 		 threads and all the threads we're going to let die
 		 arrive before the team is released.  */
 	      if (affinity_count)
-		gomp_simple_barrier_reinit (&pool->threads_dock,
-					    nthreads + affinity_count);
+		{
+		  /* Total number of threads running is:
+		     nthreads + affinity_count
+		     Equation to be satisfied is:
+		      (nthreads + affinity_count ==
+		       old_threads_used + num_new_threads).
+		     So num_new_threads
+			 == (nthreads + affinity_count - old_threads_used)
+		     Originally
+		      num_new_threads == nthreads > old_threads_used
+			  ? (nthreads - old_threads_used)
+			  : 0;
+
+		     Now:
+		       nthreads + num_exiting_threads
+			 == old_threads_used + num_new_threads
+		     So:
+		       num_exiting_threads
+			 == old_threads_used + num_new_threads - nthreads
+			 == affinity_count
+		     */
+		  gomp_assert (nthreads + affinity_count > old_threads_used,
+			       "Not spawning more threads but "
+			       "reinitialising barrier "
+			       "nthreads = %u"
+			       ", affinity_count = %u"
+			       ", old_threads_used = %u",
+			       nthreads, affinity_count, old_threads_used);
+		  num_new_threads
+		    = (nthreads + affinity_count) - old_threads_used;
+		  num_exiting_threads = affinity_count;
+		  simple_barrier_n
+		    = gomp_barrier_calc_wait (num_new_threads,
+					      num_exiting_threads,
+					      old_threads_used, nthreads,
+					      affinity_count,
+					      gomp_barrier_can_hold (
+						pool->prev_barrier));
+		  gomp_simple_barrier_reinit (&pool->threads_dock,
+					      simple_barrier_n);
+		}
 	    }
 	}
 
       if (i == nthreads)
-	goto do_release;
-
+	{
+	  gomp_assert (num_new_threads == 0,
+		       "Calculated need %u new threads but spawning none",
+		       num_new_threads);
+	  gomp_assert (gomp_barrier_can_hold (pool->prev_barrier)
+			 ? (simple_barrier_n
+			    == (num_exiting_threads ? num_exiting_threads + 1
+						    : 0))
+			 : simple_barrier_n == old_threads_used,
+		       "Have calculated need simple_barrier_n == %u"
+		       " starting 0 threads, affinity_count == %u, "
+		       "old_threads_used == %u, calculated "
+		       "num_exiting_threads == %u, nthreads == %u",
+		       simple_barrier_n, affinity_count, old_threads_used,
+		       num_exiting_threads, nthreads);
+	  goto do_release;
+	}
     }
 
   if (__builtin_expect (nthreads + affinity_count > old_threads_used, 0))
@@ -858,19 +1027,49 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads,
       attr = gomp_adjust_thread_attr (attr, &thread_attr);
       err = pthread_create (&start_data->handle, attr, gomp_thread_start,
 			    start_data);
+      spawned_new_threads += 1;
       start_data++;
       if (err != 0)
 	gomp_fatal ("Thread creation failed: %s", strerror (err));
     }
+  if (!nested)
+    {
+      gomp_assert (spawned_new_threads == num_new_threads,
+		   "Calculated num_new_threads != spawned_new_threads: "
+		   " %u != %u",
+		   num_new_threads, spawned_new_threads);
+      bool calc_check __attribute__ ((unused)) = false;
+      if (old_threads_used == 0)
+	calc_check = (num_new_threads == (simple_barrier_n - 1));
+      else if (gomp_barrier_can_hold (pool->prev_barrier))
+	calc_check
+	  = ((num_new_threads + num_exiting_threads) == (simple_barrier_n - 1));
+      else
+	calc_check = (num_new_threads == (simple_barrier_n - old_threads_used));
+      gomp_assert (calc_check,
+		   "simple barrier calculation incorrect for %s team: "
+		   "simple_barrier_n: %u"
+		   ", nthreads: %u"
+		   ", num_new_threads: %u"
+		   ", num_exiting_threads: %u"
+		   ", old_threads_used: %u",
+		   nested ? "nested" : "top-level", simple_barrier_n, nthreads,
+		   num_new_threads, num_exiting_threads, old_threads_used);
+    }
 
   if (__builtin_expect (attr == &thread_attr, 0))
     pthread_attr_destroy (&thread_attr);
 
  do_release:
   if (nested)
-    gomp_barrier_wait (&team->barrier);
+    gomp_barrier_wait (&team->barrier, thr->ts.team_id);
   else
-    gomp_simple_barrier_wait (&pool->threads_dock);
+    {
+      gomp_release_held_threads (pool, team, thr->ts.team_id);
+      if (num_new_threads || num_exiting_threads
+	  || !gomp_barrier_can_hold (pool->prev_barrier))
+	gomp_simple_barrier_wait (&pool->threads_dock);
+    }
 
   /* Decrease the barrier threshold to match the number of threads
      that should arrive back at the end of this team.  The extra
@@ -898,6 +1097,14 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads,
       gomp_mutex_unlock (&gomp_managed_threads_lock);
 #endif
     }
+  else if (!nested && gomp_barrier_can_hold (pool->prev_barrier))
+    {
+      /* Need to adjust the simple barrier threshold once it's been used in
+	 order to account for all threads that would need to be waited on in
+	 something like `gomp_free_thread` or `gomp_pause_thread`.  */
+      gomp_simple_barrier_reinit (&pool->threads_dock, nthreads);
+    }
+
   if (__builtin_expect (gomp_display_affinity_var, 0))
     {
       if (nested
@@ -949,12 +1156,15 @@ gomp_team_end (void)
 {
   struct gomp_thread *thr = gomp_thread ();
   struct gomp_team *team = thr->ts.team;
+  unsigned team_id = thr->ts.team_id;
 
   /* This barrier handles all pending explicit threads.
      As #pragma omp cancel parallel might get awaited count in
      team->barrier in a inconsistent state, we need to use a different
      counter here.  */
-  gomp_team_barrier_wait_final (&team->barrier);
+  gomp_team_barrier_wait_final (&team->barrier, team_id);
+  /* After this point all tasking has finished.  Just haven't told all
+     secondary threads that they're good to continue.  */
   if (__builtin_expect (team->team_cancelled, 0))
     {
       struct gomp_work_share *ws = team->work_shares_to_free;
@@ -974,8 +1184,11 @@ gomp_team_end (void)
   gomp_end_task ();
   thr->ts = team->prev_ts;
 
+  bool finished_threads __attribute__ ((unused)) = false;
   if (__builtin_expect (thr->ts.level != 0, 0))
     {
+      gomp_team_barrier_done_final (&team->barrier, team_id);
+      finished_threads = true;
 #ifdef HAVE_SYNC_BUILTINS
       __sync_fetch_and_add (&gomp_managed_threads, 1L - team->nthreads);
 #else
@@ -985,7 +1198,7 @@ gomp_team_end (void)
 #endif
       /* This barrier has gomp_barrier_wait_last counterparts
 	 and ensures the team can be safely destroyed.  */
-      gomp_barrier_wait (&team->barrier);
+      gomp_barrier_wait (&team->barrier, team_id);
     }
 
   if (__builtin_expect (team->work_shares[0].next_alloc != NULL, 0))
@@ -1001,15 +1214,33 @@ gomp_team_end (void)
     }
   gomp_sem_destroy (&team->master_release);
 
-  if (__builtin_expect (thr->ts.team != NULL, 0)
+  if (__builtin_expect (thr->ts.team != NULL
+			  && (thr->ts.team->nthreads != 1
+			      || thr->ts.level != 0),
+			0)
       || __builtin_expect (team->nthreads == 1, 0))
-    free_team (team);
+    {
+      gomp_assert (finished_threads || team->nthreads == 1,
+		   "Freeing team while threads may be waiting on it."
+		   " team = %p, level = %u",
+		   team, thr->ts.level + 1);
+      free_team (team);
+    }
   else
     {
       struct gomp_thread_pool *pool = thr->thread_pool;
       if (pool->last_team)
 	free_team (pool->last_team);
       pool->last_team = team;
+      gomp_assert (!finished_threads,
+		   "Have let threads go while still recording prev_barrier"
+		   " level = %u, team = %p, nthreads = %u",
+		   thr->ts.level + 1, team, team->nthreads);
+      gomp_assert (!pool->prev_barrier,
+		   "Previous barrier not released before it gets overridden"
+		   " level = %u, team = %p, nthreads = %u",
+		   thr->ts.level + 1, team, team->nthreads);
+      pool->prev_barrier = &team->barrier;
       gomp_release_thread_pool (pool);
     }
 }
@@ -1081,9 +1312,11 @@ gomp_pause_host (void)
 	      thrs[i] = gomp_thread_to_pthread_t (nthr);
 	    }
 	  /* This barrier undocks threads docked on pool->threads_dock.  */
-	  gomp_simple_barrier_wait (&pool->threads_dock);
-	  /* And this waits till all threads have called gomp_barrier_wait_last
-	     in gomp_pause_pool_helper.  */
+	  gomp_release_held_threads (pool, NULL, thr->ts.team_id);
+	  if (!gomp_barrier_can_hold (pool->prev_barrier))
+	    gomp_simple_barrier_wait (&pool->threads_dock);
+	  /* And this waits till all threads have called
+	     gomp_simple_barrier_wait_last in gomp_pause_pool_helper.  */
 	  gomp_simple_barrier_wait (&pool->threads_dock);
 	  /* Now it is safe to destroy the barrier and free the pool.  */
 	  gomp_simple_barrier_destroy (&pool->threads_dock);
diff --git a/libgomp/testsuite/lib/libgomp.exp b/libgomp/testsuite/lib/libgomp.exp
index 93e2a681cb4..4987400f2a0 100644
--- a/libgomp/testsuite/lib/libgomp.exp
+++ b/libgomp/testsuite/lib/libgomp.exp
@@ -389,6 +389,12 @@ proc check_effective_target_nonstandard_math_functions { } {
     } "-lm" ]
 }
 
+proc check_effective_target_linux_futex_waitv { } {
+    global libgomp_config_path
+    return [expr {
+	      [lsearch -exact $libgomp_config_path "linux/futex_waitv"] >= 0}]
+}
+
 # Return 1 if compiling for the specified offload target
 # Takes -foffload=... into account by checking OFFLOAD_TARGET_NAMES=
 # in the -v compiler output.
@@ -453,7 +459,7 @@ proc check_effective_target_offload_device_nonshared_as { } {
 	}
     } ]
 }
-  
+
 # Return 1 if offload device is available and it has shared address space.
 proc check_effective_target_offload_device_shared_as { } {
     return [check_runtime_nocache offload_device_shared_as {
diff --git a/libgomp/testsuite/libgomp-site-extra.exp.in b/libgomp/testsuite/libgomp-site-extra.exp.in
index 8de14f48976..ba576251c63 100644
--- a/libgomp/testsuite/libgomp-site-extra.exp.in
+++ b/libgomp/testsuite/libgomp-site-extra.exp.in
@@ -1,2 +1,3 @@
 set FLOCK {@FLOCK@}
 set SYSROOT_CFLAGS_FOR_TARGET {@SYSROOT_CFLAGS_FOR_TARGET@}
+set libgomp_config_path {@config_path@}
diff --git a/libgomp/testsuite/libgomp.c++/task-reduction-20.C b/libgomp/testsuite/libgomp.c++/task-reduction-20.C
new file mode 100644
index 00000000000..161b238578a
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c++/task-reduction-20.C
@@ -0,0 +1,136 @@
+extern "C" void abort ();
+
+struct S
+{
+  S ();
+  S (long long int, int);
+  ~S ();
+  static int cnt1, cnt2, cnt3;
+  long long int s;
+  int t;
+};
+
+int S::cnt1;
+int S::cnt2;
+int S::cnt3;
+
+S::S ()
+{
+#pragma omp atomic
+  cnt1++;
+}
+
+S::S (long long int x, int y) : s (x), t (y)
+{
+#pragma omp atomic update
+  ++cnt2;
+}
+
+S::~S ()
+{
+#pragma omp atomic
+  cnt3 = cnt3 + 1;
+  if (t < 3 || t > 9 || (t & 1) == 0)
+    abort ();
+}
+
+void
+bar (S *p, S *o)
+{
+  p->s = 1;
+  if (o->t != 5)
+    abort ();
+  p->t = 9;
+}
+
+static inline void
+baz (S *o, S *i)
+{
+  if (o->t != 5 || i->t != 9)
+    abort ();
+  o->s *= i->s;
+}
+
+#pragma omp declare reduction(+ : S : omp_out.s += omp_in.s)                   \
+  initializer(omp_priv(0, 3))
+#pragma omp declare reduction(* : S : baz(&omp_out, &omp_in))                  \
+  initializer(bar(&omp_priv, &omp_orig))
+
+S as = {0LL, 7};
+S &a = as;
+S bs (1LL, 5);
+S &b = bs;
+
+void
+foo (S &c, S &d)
+{
+  int i;
+  for (i = 0; i < 2; i++)
+#pragma omp task in_reduction(+ : c) in_reduction(* : b, d) in_reduction(+ : a)
+    {
+      a.s += 7;
+      b.s *= 2;
+      c.s += 9;
+      d.s *= 3;
+      if ((a.t != 7 && a.t != 3) || (b.t != 5 && b.t != 9)
+	  || (c.t != 7 && c.t != 3) || (d.t != 5 && d.t != 9))
+	abort ();
+    }
+}
+
+void
+test ()
+{
+  S cs = {0LL, 7};
+  S &c = cs;
+  S ds (1LL, 5);
+  S &d = ds;
+#pragma omp parallel
+  {
+    asm volatile ("" ::: "memory");
+  }
+#pragma omp parallel reduction(task, + : a, c) reduction(task, * : b, d)
+  {
+#pragma omp for
+    for (int i = 0; i < 4; i++)
+#pragma omp task in_reduction(* : b, d) in_reduction(+ : a, c)
+      {
+	int j;
+	a.s += 7;
+	b.s *= 2;
+	for (j = 0; j < 2; j++)
+#pragma omp task in_reduction(+ : a) in_reduction(* : b) in_reduction(+ : c)   \
+  in_reduction(* : d)
+	  {
+	    a.s += 7;
+	    b.s *= 2;
+	    c.s += 9;
+	    d.s *= 3;
+	    foo (c, d);
+	    if ((a.t != 7 && a.t != 3) || (b.t != 5 && b.t != 9)
+		|| (c.t != 7 && c.t != 3) || (d.t != 5 && d.t != 9))
+	      abort ();
+	  }
+	c.s += 9;
+	d.s *= 3;
+	if ((a.t != 7 && a.t != 3) || (b.t != 5 && b.t != 9)
+	    || (c.t != 7 && c.t != 3) || (d.t != 5 && d.t != 9))
+	  abort ();
+      }
+  }
+#define THREEP7 (3LL * 3LL * 3LL * 3LL * 3LL * 3LL * 3LL)
+  if (d.s != (THREEP7 * THREEP7 * THREEP7 * THREEP7) || d.t != 5)
+    abort ();
+  if (a.s != 28 * 7 || a.t != 7 || b.s != (1L << 28) || b.t != 5
+      || c.s != 28 * 9 || c.t != 7)
+    abort ();
+}
+
+int
+main ()
+{
+  int c1 = S::cnt1, c2 = S::cnt2, c3 = S::cnt3;
+  test ();
+  if (S::cnt1 + S::cnt2 - c1 - c2 != S::cnt3 - c3)
+    abort ();
+}
diff --git a/libgomp/testsuite/libgomp.c++/task-reduction-21.C b/libgomp/testsuite/libgomp.c++/task-reduction-21.C
new file mode 100644
index 00000000000..5a42add400a
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c++/task-reduction-21.C
@@ -0,0 +1,140 @@
+extern "C" void abort ();
+
+struct S
+{
+  S ();
+  S (long long int, int);
+  ~S ();
+  static int cnt1, cnt2, cnt3;
+  long long int s;
+  int t;
+};
+
+int S::cnt1;
+int S::cnt2;
+int S::cnt3;
+
+S::S ()
+{
+#pragma omp atomic
+  cnt1++;
+}
+
+S::S (long long int x, int y) : s (x), t (y)
+{
+#pragma omp atomic update
+  ++cnt2;
+}
+
+S::~S ()
+{
+#pragma omp atomic
+  cnt3 = cnt3 + 1;
+  if (t < 3 || t > 9 || (t & 1) == 0)
+    abort ();
+}
+
+void
+bar (S *p, S *o)
+{
+  p->s = 1;
+  if (o->t != 5)
+    abort ();
+  p->t = 9;
+}
+
+static inline void
+baz (S *o, S *i)
+{
+  if (o->t != 5 || i->t != 9)
+    abort ();
+  o->s *= i->s;
+}
+
+#pragma omp declare reduction(+ : S : omp_out.s += omp_in.s)                   \
+  initializer(omp_priv(0, 3))
+#pragma omp declare reduction(* : S : baz(&omp_out, &omp_in))                  \
+  initializer(bar(&omp_priv, &omp_orig))
+
+S as = {0LL, 7};
+S &a = as;
+S bs (1LL, 5);
+S &b = bs;
+
+void
+foo (S &c, S &d)
+{
+  int i;
+  for (i = 0; i < 2; i++)
+#pragma omp task in_reduction(+ : c) in_reduction(* : b, d) in_reduction(+ : a)
+    {
+      a.s += 7;
+      b.s *= 2;
+      c.s += 9;
+      d.s *= 3;
+      if ((a.t != 7 && a.t != 3) || (b.t != 5 && b.t != 9)
+	  || (c.t != 7 && c.t != 3) || (d.t != 5 && d.t != 9))
+	abort ();
+    }
+}
+
+void
+test ()
+{
+  S cs = {0LL, 7};
+  S &c = cs;
+  S ds (1LL, 5);
+  S &d = ds;
+#pragma omp parallel
+  {
+    for (int i = 0; i < 4; i++)
+#pragma omp task
+      {
+	asm volatile ("" ::: "memory");
+      }
+  }
+#pragma omp parallel reduction(task, + : a, c) reduction(task, * : b, d)
+  {
+#pragma omp for
+    for (int i = 0; i < 4; i++)
+#pragma omp task in_reduction(* : b, d) in_reduction(+ : a, c)
+      {
+	int j;
+	a.s += 7;
+	b.s *= 2;
+	for (j = 0; j < 2; j++)
+#pragma omp task in_reduction(+ : a) in_reduction(* : b) in_reduction(+ : c)   \
+  in_reduction(* : d)
+	  {
+	    a.s += 7;
+	    b.s *= 2;
+	    c.s += 9;
+	    d.s *= 3;
+	    foo (c, d);
+	    if ((a.t != 7 && a.t != 3) || (b.t != 5 && b.t != 9)
+		|| (c.t != 7 && c.t != 3) || (d.t != 5 && d.t != 9))
+	      abort ();
+	  }
+	c.s += 9;
+	d.s *= 3;
+	if ((a.t != 7 && a.t != 3) || (b.t != 5 && b.t != 9)
+	    || (c.t != 7 && c.t != 3) || (d.t != 5 && d.t != 9))
+	  abort ();
+      }
+  }
+#define THREEP7 (3LL * 3LL * 3LL * 3LL * 3LL * 3LL * 3LL)
+  if (d.s != (THREEP7 * THREEP7 * THREEP7 * THREEP7) || d.t != 5)
+    abort ();
+  if (a.s != 28 * 7 || a.t != 7 || b.s != (1L << 28) || b.t != 5
+      || c.s != 28 * 9 || c.t != 7)
+    abort ();
+}
+
+int
+main ()
+{
+  int c1 = S::cnt1, c2 = S::cnt2, c3 = S::cnt3;
+  test ();
+  if (S::cnt1 + S::cnt2 - c1 - c2 != S::cnt3 - c3)
+    abort ();
+}
diff --git a/libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env-auto.c b/libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env-auto.c
new file mode 100644
index 00000000000..fb68b3d9e06
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env-auto.c
@@ -0,0 +1,11 @@
+/* { dg-do run } */
+/* { dg-set-target-env-var OMP_DISPLAY_ENV "verbose" } */
+/* { dg-set-target-env-var GOMP_BARRIER_TYPE "auto" } */
+
+int
+main (void)
+{
+  return 0;
+}
+
+/* { dg-output ".*\\\[host] GOMP_BARRIER_TYPE = 'auto'.*" { target native } } */
diff --git a/libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env-flat.c b/libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env-flat.c
new file mode 100644
index 00000000000..106b34d75be
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env-flat.c
@@ -0,0 +1,11 @@
+/* { dg-do run } */
+/* { dg-set-target-env-var OMP_DISPLAY_ENV "verbose" } */
+/* { dg-set-target-env-var GOMP_BARRIER_TYPE "flat" } */
+
+int
+main (void)
+{
+  return 0;
+}
+
+/* { dg-output ".*\\\[host] GOMP_BARRIER_TYPE = 'flat'.*" { target native } } */
diff --git a/libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env-unset.c b/libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env-unset.c
new file mode 100644
index 00000000000..2a6f5b3be86
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env-unset.c
@@ -0,0 +1,10 @@
+/* { dg-do run } */
+/* { dg-set-target-env-var OMP_DISPLAY_ENV "verbose" } */
+
+int
+main (void)
+{
+  return 0;
+}
+
+/* { dg-output ".*\\\[host] GOMP_BARRIER_TYPE = 'auto'.*" { target native } } */
diff --git a/libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env.c b/libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env.c
new file mode 100644
index 00000000000..51c8c578d59
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c-c++-common/gomp-barrier-type-env.c
@@ -0,0 +1,11 @@
+/* { dg-do run } */
+/* { dg-set-target-env-var OMP_DISPLAY_ENV "verbose" } */
+/* { dg-set-target-env-var GOMP_BARRIER_TYPE "tree" } */
+
+int
+main (void)
+{
+  return 0;
+}
+
+/* { dg-output ".*\\\[host] GOMP_BARRIER_TYPE = 'tree'.*" { target native } } */
diff --git a/libgomp/testsuite/libgomp.c/barrier-switch-1.c b/libgomp/testsuite/libgomp.c/barrier-switch-1.c
new file mode 100644
index 00000000000..c894fde6a46
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/barrier-switch-1.c
@@ -0,0 +1,135 @@
+/* Exercise changing between flat and tree barriers for a live team.
+   N.b. there should be no observable difference between switching and not, but
+   having a testcase at least exercises the code path.  */
+/* { dg-do run { target *-*-linux* } } */
+/* { dg-set-target-env-var GOMP_BARRIER_TYPE "auto" } */
+/* { dg-set-target-env-var GOMP_SPINCOUNT "100000000" } */
+/* { dg-set-target-env-var OMP_DYNAMIC "false" } */
+/* { dg-set-target-env-var OMP_MAX_ACTIVE_LEVELS "2" } */
+
+#define _GNU_SOURCE 1
+#include <omp.h>
+#include <sched.h>
+#include <stdatomic.h>
+#include <stdbool.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+static _Atomic int start_helper;
+static _Atomic int helper_active;
+static _Atomic int stop_helper;
+static _Atomic int helper_finished;
+
+static void
+wait_for (const _Atomic int *flag)
+{
+  while (!atomic_load_explicit (flag, memory_order_acquire))
+    sched_yield ();
+}
+
+static void
+barrier (void)
+{
+#pragma omp barrier
+}
+
+/* Restrict the process before libgomp is reinitialised so that
+   gomp_available_cpus is three.  With the main nested team active there are
+   three managed threads.  Starting the helper nested team increases that to
+   four, making spin_count switch to its throttled value.  */
+static bool
+restrict_to_three_cpus_and_reexec (void)
+{
+  if (getenv ("GOMP_BARRIER_SWITCH_REEXEC"))
+    return true;
+
+  cpu_set_t available;
+  cpu_set_t selected;
+  if (sched_getaffinity (0, sizeof (available), &available) != 0)
+    return false;
+
+  CPU_ZERO (&selected);
+  int count = 0;
+  for (int cpu = 0; cpu < CPU_SETSIZE && count != 3; ++cpu)
+    if (CPU_ISSET (cpu, &available))
+      {
+	CPU_SET (cpu, &selected);
+	++count;
+      }
+  if (count != 3
+      || sched_setaffinity (0, sizeof (selected), &selected) != 0
+      || setenv ("GOMP_BARRIER_SWITCH_REEXEC", "1", 1) != 0)
+    return false;
+
+  execl ("/proc/self/exe", "barrier-switch-1", NULL);
+  abort ();
+}
+
+int
+main (void)
+{
+  if (!restrict_to_three_cpus_and_reexec ())
+    return 0;
+
+  omp_set_dynamic (0);
+  omp_set_max_active_levels (2);
+
+#pragma omp parallel num_threads(2)
+  {
+    if (omp_get_thread_num () == 0)
+      {
+	/* Keep this nested team alive while its sibling team changes the global
+	   number of managed threads.  */
+#pragma omp parallel num_threads(2)
+	{
+	  /* Initially three managed threads run on three available CPUs.  This
+	     barrier is flat and selects flat for the next generation.  */
+	  barrier ();
+
+#pragma omp masked
+	  {
+	    atomic_store_explicit (&start_helper, 1, memory_order_release);
+	    wait_for (&helper_active);
+	  }
+
+	  /* The current barrier is flat.  Four managed threads are now running
+	     on three available CPUs, so completion selects tree.  */
+	  barrier ();
+	  /* Exercise the selected tree barrier.  */
+	  barrier ();
+
+#pragma omp masked
+	  {
+	    atomic_store_explicit (&stop_helper, 1, memory_order_release);
+	    wait_for (&helper_finished);
+	  }
+
+	  /* The current barrier is tree.  The helper nested team has finished,
+	     returning to three managed threads, so completion selects flat.  */
+	  barrier ();
+	  /* Exercise the selected flat barrier.  */
+	  barrier ();
+	}
+      }
+    else
+      {
+	wait_for (&start_helper);
+
+	/* This sibling nested team adds one managed thread until both members
+	   leave the region.  */
+#pragma omp parallel num_threads(2)
+	{
+#pragma omp masked
+	  atomic_store_explicit (&helper_active, 1, memory_order_release);
+
+	  wait_for (&stop_helper);
+	}
+
+	/* Nested-team workers are removed from gomp_managed_threads before the
+	   nested parallel construct returns.  */
+	atomic_store_explicit (&helper_finished, 1, memory_order_release);
+      }
+  }
+
+  return 0;
+}
diff --git a/libgomp/testsuite/libgomp.c/barrier_generation_overflow.c b/libgomp/testsuite/libgomp.c/barrier_generation_overflow.c
new file mode 100644
index 00000000000..1dc4b00e4c3
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/barrier_generation_overflow.c
@@ -0,0 +1,88 @@
+/* We only run this for linux/futex_waitv targets because they have their
+   generation number wrap at a much lower value than other targets.  The much
+   higher value of other targets takes a very long time to reach -- so much
+   that the default DejaGNU timeout treats it as a timeout anyway.  */
+/* { dg-do run { target linux_futex_waitv } } */
+/* Including the numbers for both futex_waitv and non-futex_waitv targets even
+   though we only run on futex_waitv targets.  Just for anyone reading this
+   file -- does not make any difference to how the testcase is ran.  */
+/* { dg-additional-options "-DPREWRAP_BARRIERS=1048575" { target linux_futex_waitv } } */
+/* { dg-additional-options "-DPREWRAP_BARRIERS=536870911" { target { ! linux_futex_waitv } } } */
+#include <omp.h>
+#include <unistd.h>
+
+#define NTHREADS 2
+#define TASK_USLEEP 500000
+
+/* This is the plain-barrier analogue of cancel_incr_wraparound.c.
+   The structure deliberately mirrors the 2-thread cancellable-generation
+   wraparound testcase:
+
+   1. A single long-lived 2-thread team executes 1,048,575 plain barriers
+      without any tasks.  That advances the non-cancellable generation to the
+      highest value value it can hold before wraparound (0xfffff000).
+   2. Thread 1 creates one deferred task and then falls moves on to the final
+      barrier.
+   3. Thread 0 waits outside that final barrier until the deferred task has
+      actually started.  In this testcase there are no other task scheduling
+      points between task creation and the final barrier, so knowing the task
+      has started implies that thread 1 has entered the plain barrier wait
+      path.
+   4. Thread 0 then enters the 1,048,576th plain barrier use while the task is
+      still running, so the wrap-sensitive raw-BAR_WAS_LAST completion check is
+      exercised with task_count != 0.
+
+   On this Linux backend BAR_INCR = 4096, so the plain generation wraps after
+   2^32 / 4096 = 1,048,576 uses of the same plain barrier generation.  */
+
+static int secondary_task_started;
+
+static void
+hold_barrier_task (void)
+{
+#pragma omp atomic write
+  secondary_task_started = 1;
+  usleep (TASK_USLEEP);
+}
+
+int
+main (void)
+{
+  omp_set_dynamic (0);
+  secondary_task_started = 0;
+
+#pragma omp parallel num_threads(NTHREADS) shared(secondary_task_started)
+  {
+    int tid = omp_get_thread_num ();
+
+    for (int round = 0; round < PREWRAP_BARRIERS; ++round)
+      {
+#pragma omp barrier
+      }
+
+    if (tid == 1)
+      {
+#pragma omp task
+	hold_barrier_task ();
+      }
+
+    if (tid == 0)
+      {
+	int seen = 0;
+	while (!seen)
+	  {
+#pragma omp atomic read
+	    seen = secondary_task_started;
+	  }
+      }
+
+      /* Thread 1 has already entered the barrier and started the deferred task
+	 above before thread 0 enters this wrap-sensitive 1,048,576th plain
+	 barrier use.  This means that we can force the check in
+	 `gomp_barrier_has_completed` to be ran using a `state` which has
+	 `BAR_WAS_LAST` set on it.   */
+#pragma omp barrier
+  }
+
+  return 0;
+}
diff --git a/libgomp/testsuite/libgomp.c/cancel_in_implicit_barrier.c b/libgomp/testsuite/libgomp.c/cancel_in_implicit_barrier.c
new file mode 100644
index 00000000000..0d78382f4a4
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/cancel_in_implicit_barrier.c
@@ -0,0 +1,115 @@
+/*
+   Test functionality of cancelling a region with no cancellation point.
+   I.e. this tests the `_final` barrier being able to handle cancellation
+   mid-way.
+
+   N.b. this testcase was triggering an assertion during development of the
+   linux/futex_waitv target.  It's here as a regression test, but the failure
+   mode is only really visible when testing a libgomp built with assertions.
+   The code should still work either way so it's not problematic to run each
+   time, but catching the performance problem that would be a real issue is not
+   very feasible.
+   While the problem was only seen in on that target I run the testcase on all
+   targets just for the sake of exercising code paths.
+
+   For information's sake:  the performance problem was that threads could
+   busy-wait when we judge they should have entering the kernel.  They would do
+   this because the code assumed the `BAR_CANCELLED` flag would not change and
+   they checked the originally read generation against their most recently read
+   generation to see whether to continue spinning or sleep.
+ */
+/* { dg-do run } */
+/* { dg-set-target-env-var OMP_CANCELLATION "true" } */
+
+#include <omp.h>
+#include <stdlib.h>
+
+#define NUM_THREADS 4
+#define NUM_TASKS 500
+
+/* Each task does busy work to keep tasks pending at the barrier. */
+#define TASK_WORK 500000
+
+static volatile int tasks_executed;
+static volatile int tasks_total;
+
+/* Goal is to trigger `BAR_CANCELLED` getting set on the barrier generation
+   flag while many threads are already waiting on the `*_final` barrier at the
+   end of this parallel region.
+
+   We have tasks running in order to ensure that the setting of `BAR_CANCELLED`
+   happens while secondary threads are in the tasking loop of
+   `gomp_team_barrier_wait_for_tasks`.  */
+int
+main (void)
+{
+  if (!omp_get_cancellation ())
+    abort ();
+  omp_set_dynamic (0);
+  omp_set_num_threads (NUM_THREADS);
+
+  for (int trial = 0; trial < 20; trial++)
+    {
+#pragma omp atomic write
+      tasks_executed = 0;
+#pragma omp atomic write
+      tasks_total = 0;
+
+#pragma omp parallel num_threads(NUM_THREADS)
+      {
+	int tid = omp_get_thread_num ();
+
+#pragma omp single nowait
+	{
+	  for (int i = 0; i < NUM_TASKS; i++)
+	    {
+#pragma omp task
+	      {
+		/* Busy work to keep the task running for ~1ms.
+		   Gives the thread that hopes to cancel the barrier enough
+		   time to observe that some tasks have executed and cancel the
+		   barrier all tasks have been completed. */
+		volatile int y = 0;
+		for (int j = 0; j < TASK_WORK; j++)
+		  y++;
+#pragma omp atomic update
+		tasks_executed += 1;
+	      }
+	    }
+	}
+
+	if (tid == 0)
+	  {
+	    /* Wait for some tasks to get executed.  As soon as there is at
+	       least one task getting executed cancel the barrier.  This should
+	       hopefully cancel the barrier before all tasks have been
+	       completed (and observation shows this to be the case).  I.e.
+	       ensures that other threads are in the relevant task handling loop
+	       when the barrier is cancelled.  */
+	    int executed;
+#pragma omp atomic read
+	    executed = tasks_executed;
+	    while (executed == 0)
+	      {
+#pragma omp atomic read
+		executed = tasks_executed;
+	      };
+#pragma omp cancel parallel
+	  }
+      }
+
+      int n_exec;
+#pragma omp atomic read
+      n_exec = tasks_executed;
+      /* If you're seeing an abort here then it is not a sign that libgomp is
+	 failing.  Rather it is a sign that the timings in this testsuite do
+	 not match your machine.  I.e. your machine completed the tasks too
+	 fast for this testcase to be stressing anything.
+
+	 That means this testcase may need to be updated, and without this
+	 abort we would not notice.  */
+      if (n_exec == NUM_TASKS)
+	abort ();
+    }
+  return 0;
+}
diff --git a/libgomp/testsuite/libgomp.c/cancel_in_implicit_barrier_while_primary_waiting.c b/libgomp/testsuite/libgomp.c/cancel_in_implicit_barrier_while_primary_waiting.c
new file mode 100644
index 00000000000..ccfe3ab1055
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/cancel_in_implicit_barrier_while_primary_waiting.c
@@ -0,0 +1,70 @@
+/* Test functionality of cancelling a region with no cancellable barrier.
+   On the `futex_waitv` target in this case the implicit barrier at the end of
+   the parallel region sees the BAR_CANCELLED flag and needs to be able to
+   handle it.
+
+   While it needs to be able to handle it, it still needs to wait for all
+   threads, because this is not a cancellable barrier it is the implicit
+   non-cancellable barrier at the end of the parallel region.
+
+   Still run the testcase on all targets for breadth of testing.  */
+/* { dg-do run } */
+/* { dg-set-target-env-var OMP_CANCELLATION "true" } */
+#include <omp.h>
+#include <stdlib.h>
+
+int
+main (void)
+{
+  omp_set_dynamic (0);
+
+  for (int round = 0; round < 200; ++round)
+    {
+      int phase = 0;
+
+#pragma omp parallel num_threads(8)
+      {
+	int tid = omp_get_thread_num ();
+
+	if (tid == 0)
+	  {
+	    /* Primary goes straight through to barrier, and waits for other
+	       threads to arrive.  Publish that we are about to fall into the
+	       final barrier so the non-primary coordination can start.  */
+#pragma omp atomic write
+	    phase = 1;
+	  }
+	else if (tid == 1)
+	  {
+	    /* We need one non-primary thread to issue the cancel while the
+	       primary is still waiting for secondaries to arrive.  */
+	    while (1)
+	      {
+		int seen;
+#pragma omp atomic read
+		seen = phase;
+		if (seen >= 1)
+		  break;
+	      }
+#pragma omp atomic write
+	    phase = 2;
+#pragma omp cancel parallel if (1)
+	  }
+	else if (tid == 2)
+	  {
+	    /* Second non-primary thread holds the primary thread until the
+	       region has been cancelled.  */
+	    while (1)
+	      {
+		int seen;
+#pragma omp atomic read
+		seen = phase;
+		if (seen >= 2)
+		  break;
+	      }
+	  }
+      }
+    }
+
+  return 0;
+}
diff --git a/libgomp/testsuite/libgomp.c/cancel_incr_wraparound.c b/libgomp/testsuite/libgomp.c/cancel_incr_wraparound.c
new file mode 100644
index 00000000000..77950de0fdd
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/cancel_incr_wraparound.c
@@ -0,0 +1,94 @@
+/* { dg-do run { target linux_futex_waitv } } */
+/* { dg-set-target-env-var OMP_CANCELLATION "true" } */
+/* Regression test for bug seen during development.
+
+   This exercises the case where `gomp_barrier_handle_tasks` is called with a
+   `state` with non-zero flags and while the cancel generation bitfield is
+   about to wrap around.  By doing this it exercises an edge case for
+   `gomp_barrier_state_is_incremented` where `state > gen` but `next < gen`.
+
+   On the futex_waitv backend the cancellable generation uses BAR_CANCEL_INCR =
+   16 and 8 dedicated bits, so the wrap-sensitive state can only be exhibited
+   after 256 uses of the same cancel-capable barrier generation. This test
+   has some pre-wrap rounds to get to the wrap-sensitive state before running
+   the edge case.  */
+
+#include <omp.h>
+#include <unistd.h>
+
+#define NTHREADS 2
+#define PREWRAP_ROUNDS 255
+#define TASK_USLEEP 200000
+
+static volatile int inactive_cancel;
+static int secondary_task_started;
+
+static inline void
+cheap_work (int i)
+{
+  asm volatile ("" : : "r"(i) : "memory");
+}
+
+static void
+hold_barrier_task (void)
+{
+#pragma omp atomic write
+  secondary_task_started = 1;
+  usleep (TASK_USLEEP);
+}
+
+static void
+run_outer_round (void)
+{
+  inactive_cancel = 0;
+  secondary_task_started = 0;
+
+#pragma omp parallel num_threads(NTHREADS) shared(secondary_task_started)
+  {
+    int tid = omp_get_thread_num ();
+
+    /* These 255 loop-end barriers, plus the explicit barrier below, give the
+       same cancel-capable barrier 256 uses on this 2-thread team.  */
+    for (int round = 0; round < PREWRAP_ROUNDS; ++round)
+      {
+#pragma omp cancel parallel if (tid == 0 && inactive_cancel)
+#pragma omp for schedule(static, 1)
+	for (int i = 0; i < NTHREADS; ++i)
+	  cheap_work (i + round);
+      }
+
+    if (tid == 1)
+      {
+#pragma omp task
+	hold_barrier_task ();
+      }
+
+#pragma omp cancel parallel if (tid == 0 && inactive_cancel)
+    if (tid == 0)
+      {
+	int seen = 0;
+	while (!seen)
+	  {
+#pragma omp atomic read
+	    seen = secondary_task_started;
+	  }
+      }
+
+      /* If we did everything right, thread 1 has already entered the
+	 cancellable barrier wait path and started the deferred task above
+	 before thread 0 enters this wrap-sensitive barrier use.  That deferred
+	 task should be long-running enough that the primary thread can go
+	 through the initial part of the barrier and start attempting to perform
+	 tasks before all tasks have been completed.  */
+#pragma omp barrier
+  }
+}
+
+int
+main (void)
+{
+  omp_set_dynamic (0);
+  omp_set_schedule (omp_sched_static, 1);
+  run_outer_round ();
+  return 0;
+}
diff --git a/libgomp/testsuite/libgomp.c/futex_waitv_overflow.c b/libgomp/testsuite/libgomp.c/futex_waitv_overflow.c
new file mode 100644
index 00000000000..67d1f7ce9ea
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/futex_waitv_overflow.c
@@ -0,0 +1,36 @@
+/* { dg-do run { target linux_futex_waitv } } */
+/* { dg-set-target-env-var GOMP_SPINCOUNT "0" } */
+/* Regression test for bug seen during development.
+
+   This exercises the case where `futex_waitv` is passed values that set the
+   top bit of the futex word.  A futex word is defined to be 32 bits.  The
+   generation is an unsigned value.  We delay all secondary threads before
+   reaching the barrier on the 524288'th time.  This ensures the primary thread
+   enters the barrier, does not see any threads arriving, then sleeps in the
+   kernel on one of these threads arriving.  */
+#include <omp.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+int
+main ()
+{
+  int warmup = 524287;
+  int delayed = 1;
+  int delay_us = 100000;
+
+  omp_set_dynamic (0);
+
+#pragma omp parallel num_threads(4)
+  {
+    int tid = omp_get_thread_num ();
+    for (int round = 0; round < warmup + delayed; ++round)
+      {
+	if (round >= warmup && tid != 0)
+	  usleep (delay_us);
+#pragma omp barrier
+      }
+  }
+
+  return 0;
+}
diff --git a/libgomp/testsuite/libgomp.c/primary-thread-tasking.c b/libgomp/testsuite/libgomp.c/primary-thread-tasking.c
new file mode 100644
index 00000000000..e4d2efb857a
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/primary-thread-tasking.c
@@ -0,0 +1,80 @@
+/* Test to check our primary thread can execute some tasks while waiting for
+   other threads.  This to check an edge-case in a recent implementation of the
+   barrier tasking mechanism.
+   I don't believe there's any way to guarantee that a task will be run on a
+   given thread.  Hence I don't know anywhere I can put an `abort` and say we
+   failed.  However I can set things up so that we'll timeout if the primary
+   thread is not executing any tasks.  That timeout will at least count as a
+   fail.
+
+   Idea here being that we keep spawning tasks until one is handled by the
+   primary thread.  Meanwhile we give the secondary threads lots of
+   opportunities to sleep and let the primary thread take a task.  */
+/* { dg-do run { target *-*-linux* } } */
+
+#define _GNU_SOURCE
+#include <omp.h>
+#include <unistd.h>
+#include <sys/syscall.h>
+#include <linux/futex.h>
+#include <assert.h>
+#include <stdatomic.h>
+#include <limits.h>
+
+int wake_flag = 0;
+
+void
+continue_until_on_thread0 ()
+{
+  if (omp_get_thread_num () == 0)
+    {
+      __atomic_fetch_add (&wake_flag, 1, memory_order_relaxed);
+      syscall (SYS_futex, &wake_flag, FUTEX_WAKE | FUTEX_PRIVATE_FLAG, INT_MAX);
+    }
+  else
+    {
+      /* If the flag has been set try again.  Otherwise put another few tasks
+       * on the task queue.  */
+      if (__atomic_load_n (&wake_flag, memory_order_relaxed))
+	{
+	  return;
+	}
+#pragma omp task
+      continue_until_on_thread0 ();
+#pragma omp task
+      continue_until_on_thread0 ();
+#pragma omp task
+      continue_until_on_thread0 ();
+      syscall (SYS_futex, &wake_flag, FUTEX_WAIT | FUTEX_PRIVATE_FLAG, 0, NULL);
+    }
+}
+
+void
+foo ()
+{
+#pragma omp parallel
+  {
+    if (omp_get_thread_num () != 0)
+      {
+#pragma omp task
+	continue_until_on_thread0 ();
+#pragma omp task
+	continue_until_on_thread0 ();
+	/* Wait on the master thread to have executed one of the tasks.  */
+	int val = __atomic_load_n (&wake_flag, memory_order_acquire);
+	while (val == 0)
+	  {
+	    syscall (SYS_futex, &wake_flag, FUTEX_WAIT | FUTEX_PRIVATE_FLAG,
+		     val, NULL);
+	    val = __atomic_load_n (&wake_flag, memory_order_acquire);
+	  }
+      }
+  }
+}
+
+int
+main ()
+{
+  foo ();
+  return 0;
+}
diff --git a/libgomp/work.c b/libgomp/work.c
index 5f23232ce77..280fa242fd5 100644
--- a/libgomp/work.c
+++ b/libgomp/work.c
@@ -240,7 +240,7 @@ gomp_work_share_end (void)
       return;
     }
 
-  bstate = gomp_barrier_wait_start (&team->barrier);
+  bstate = gomp_barrier_wait_start (&team->barrier, thr->ts.team_id);
 
   if (gomp_barrier_last_thread (bstate))
     {
@@ -251,7 +251,7 @@ gomp_work_share_end (void)
 	}
     }
 
-  gomp_team_barrier_wait_end (&team->barrier, bstate);
+  gomp_team_barrier_wait_end (&team->barrier, bstate, thr->ts.team_id);
   thr->ts.last_work_share = NULL;
 }
 
@@ -266,7 +266,7 @@ gomp_work_share_end_cancel (void)
   gomp_barrier_state_t bstate;
 
   /* Cancellable work sharing constructs cannot be orphaned.  */
-  bstate = gomp_barrier_wait_cancel_start (&team->barrier);
+  bstate = gomp_barrier_wait_cancel_start (&team->barrier, thr->ts.team_id);
 
   if (gomp_barrier_last_thread (bstate))
     {
@@ -278,7 +278,8 @@ gomp_work_share_end_cancel (void)
     }
   thr->ts.last_work_share = NULL;
 
-  return gomp_team_barrier_wait_cancel_end (&team->barrier, bstate);
+  return gomp_team_barrier_wait_cancel_end (&team->barrier, bstate,
+					    thr->ts.team_id);
 }
 
 /* The current thread is done with its current work sharing construct.
-- 
2.43.0